Skip to contentSkip to conversation

ARM/AMR64: Few patches on kvm and mm code

PATCH New Santosh Shilimkar 5 participants 41 messages Nov 14, 2013 – Nov 15, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover ARM/AMR64: Few patches on kvm and mm code

While playing with KVM on Keystone, I came across few issues which needs to be addressed. Have discussed most of these with Marc and Christoffer off-list but we all thought of getting these patchset on list for wider review.

At least virt_addr_valid() update am not sure and would like to get discussion going on how to proceed with it. The last patch in the series is a hack and included in the series to get some tips.

Special thanks to Christoffer and Marc for their support in figuring out these issues.

Generated against tip of Linus's tree(4fbf888) which includes __virt_to_idmap() and related patchset.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>

Santosh Shilimkar (6):
arm64: mm: Add __virt_to_idmap() to keep kvm build happy
ARM: kvm: Use virt_to_idmap instead of virt_to_phys for idmap
mappings
ARM: mm: Drop the lowmem watermark check from virt_addr_valid()
arm64: mm: Drop the lowmem watermark check from virt_addr_valid()
ARM: kvm: Use phys_addr_t instead of unsigned long in mm code
ARM: kvm: TMP: Commit the hyp page tables to main memory

 arch/arm/include/asm/memory.h   |    3 +--
 arch/arm/kvm/mmu.c              |   12 ++++++------
 arch/arm64/include/asm/memory.h |   11 +++++++++--
 3 files changed, 16 insertions(+), 10 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +8 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
 /*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)
+
+/*
  * Convert a physical address to a Page Frame Number and back
  */
 #define	__phys_to_pfn(paddr)	((unsigned long)((paddr) >> PAGE_SHIFT))
Signature · 1 line
1.7.9.5
maintainer
#2 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 6 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Quoted 6 lines
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>

BTW, there is normally no space between the Cc and Signed-off-by lines.

Quoted 9 lines
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +0 -0
index 3776217..d9341ee 100644

arch/arm64/include/asm/memory.h line 83

Quoted 9 lines
arch/arm64/include/asm/memory.h +6 -0
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
 /*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)

There are consistency issues with the use of underscores before __virt_to_idmap. And I think you can simply say something like "Required by ARM KVM code". The explanation on why really belongs to the 32-bit arm code.

Signature · 1 line
Catalin
#3 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 7 lines · Catalin Marinas
Quoted 6 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Quoted 9 lines
Quoted 6 lines
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>

BTW, there is normally no space between the Cc and Signed-off-by lines.

OK.

Quoted 11 lines
Quoted 11 lines
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +0 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
Quoted 13 lines

arch/arm64/include/asm/memory.h line 83

Quoted 7 lines
arch/arm64/include/asm/memory.h +6 -0
 /*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)

There are consistency issues with the use of underscores before __virt_to_idmap. And I think you can simply say something like "Required by ARM KVM code". The explanation on why really belongs to the 32-bit arm code.

Will update the comment as you suggested. Do you want me to rename the macros as well. I need to do update some more code for that.

Let me know

Regards,
Santosh

maintainer
#4 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 10 lines · Santosh Shilimkar
Quoted 9 lines · Catalin Marinas
Quoted 7 lines · Santosh Shilimkar

/*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x) __virt_to_phys(x)

Quoted 7 lines
Quoted 5 lines

There are consistency issues with the use of underscores before __virt_to_idmap. And I think you can simply say something like "Required by ARM KVM code". The explanation on why really belongs to the 32-bit arm code.

Will update the comment as you suggested. Do you want me to rename the macros as well. I need to do update some more code for that.

The macro name should be whatever is in arch/arm/ but the comment and the macro disagree here.

Signature · 1 line
Catalin
#5 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 16 lines · Catalin Marinas
Quoted 15 lines · Santosh Shilimkar
Quoted 14 lines · Catalin Marinas
Quoted 7 lines · Santosh Shilimkar

/*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x) __virt_to_phys(x)

There are consistency issues with the use of underscores before __virt_to_idmap. And I think you can simply say something like "Required by ARM KVM code". The explanation on why really belongs to the 32-bit arm code.

Quoted 6 lines
Quoted 2 lines

Will update the comment as you suggested. Do you want me to rename the macros as well. I need to do update some more code for that.

The macro name should be whatever is in arch/arm/ but the comment and the macro disagree here.

Damn .... I should have read it carefully. The comment is using wrong name. Sorry.

regards,
Santosh

#6 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

In reply to Santosh Shilimkar
Quoted 26 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +1 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
 /*
+ * Added to keep arm64 kvm build working which shares code with

arch/arm64/include/asm/memory.h line 88

Quoted 10 lines
arch/arm64/include/asm/memory.h +7 -0
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)
+
+/*
  * Convert a physical address to a Page Frame Number and back
  */
 #define	__phys_to_pfn(paddr)	((unsigned long)((paddr) >> PAGE_SHIFT))

I'd rather have a kvm_virt_to_phys() in kvm_mmu.h. That's how we've dealt with that kind of difference so far.

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#7 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 24 lines · Marc Zyngier
Quoted 23 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +0 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
Quoted 17 lines

arch/arm64/include/asm/memory.h line 88

Quoted 13 lines
arch/arm64/include/asm/memory.h +8 -0
 /*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)
+
+/*
  * Convert a physical address to a Page Frame Number and back
  */
 #define	__phys_to_pfn(paddr)	((unsigned long)((paddr) >> PAGE_SHIFT))

I'd rather have a kvm_virt_to_phys() in kvm_mmu.h. That's how we've dealt with that kind of difference so far.

Are you suggesting something like below ?

---
 arch/arm/include/asm/kvm_mmu.h   |    1 +
 arch/arm/kvm/mmu.c               |    8 ++++----
 arch/arm64/include/asm/kvm_mmu.h |    1 +
 3 files changed, 6 insertions(+), 4 deletions(-)
arch/arm/include/asm/kvm_mmu.h +1 -0
index 9b28c41..fd90efa 100644
@@ -129,6 +129,7 @@ static inline void coherent_icache_guest_page(struct kvm *kvm, gfn_t gfn)
 }
 
 #define kvm_flush_dcache_to_poc(a,l)	__cpuc_flush_dcache_area((a), (l))
+#define kvm_virt_to_phys(x)		virt_to_idmap((unsigned long)(x))
 
 #endif	/* !__ASSEMBLY__ */
 
arch/arm/kvm/mmu.c +4 -4
index b0de86b..071e535 100644
@@ -747,9 +747,9 @@ int kvm_mmu_init(void)
 {
 	int err;
 
-	hyp_idmap_start = virt_to_phys(__hyp_idmap_text_start);
-	hyp_idmap_end = virt_to_phys(__hyp_idmap_text_end);
-	hyp_idmap_vector = virt_to_phys(__kvm_hyp_init);
+	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
+	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
+	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
 
 	if ((hyp_idmap_start ^ hyp_idmap_end) & PAGE_MASK) {
 		/*
@@ -776,7 +776,7 @@ int kvm_mmu_init(void)
 		 */
 		kvm_flush_dcache_to_poc(init_bounce_page, len);
 
-		phys_base = virt_to_phys(init_bounce_page);
+		phys_base = kvm_virt_to_phys(init_bounce_page);
 		hyp_idmap_vector += phys_base - hyp_idmap_start;
 		hyp_idmap_start = phys_base;
 		hyp_idmap_end = phys_base + len;
arch/arm64/include/asm/kvm_mmu.h +1 -0
index efe609c..9ce7c88 100644
@@ -130,6 +130,7 @@ static inline void coherent_icache_guest_page(struct kvm *kvm, gfn_t gfn)
 }
 
 #define kvm_flush_dcache_to_poc(a,l)	__flush_dcache_area((a), (l))
+#define kvm_virt_to_phys(x)		__virt_to_phys((unsigned long)(x))
 
 #endif /* __ASSEMBLY__ */
 #endif /* __ARM64_KVM_MMU_H__ */
Signature · 1 line
1.7.9.5
#8 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 42 lines · Santosh Shilimkar
Quoted 24 lines · Marc Zyngier
Quoted 23 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +0 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
Quoted 17 lines

arch/arm64/include/asm/memory.h line 88

Quoted 13 lines
arch/arm64/include/asm/memory.h +8 -0
 /*
+ * Added to keep arm64 kvm build working which shares code with
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)
+
+/*
  * Convert a physical address to a Page Frame Number and back
  */
 #define	__phys_to_pfn(paddr)	((unsigned long)((paddr) >> PAGE_SHIFT))

I'd rather have a kvm_virt_to_phys() in kvm_mmu.h. That's how we've dealt with that kind of difference so far.

Quoted 1 line

Are you suggesting something like below ?

Yes, I like it a lot more.

Catalin, what do you think?

	M.
Quoted 58 lines
---
 arch/arm/include/asm/kvm_mmu.h   |    1 +
 arch/arm/kvm/mmu.c               |    8 ++++----
 arch/arm64/include/asm/kvm_mmu.h |    1 +
 3 files changed, 6 insertions(+), 4 deletions(-)
arch/arm/include/asm/kvm_mmu.h +1 -0
index 9b28c41..fd90efa 100644
@@ -129,6 +129,7 @@ static inline void coherent_icache_guest_page(struct kvm *kvm, gfn_t gfn)
 }
 
 #define kvm_flush_dcache_to_poc(a,l)	__cpuc_flush_dcache_area((a), (l))
+#define kvm_virt_to_phys(x)		virt_to_idmap((unsigned long)(x))
 
 #endif	/* !__ASSEMBLY__ */
 
arch/arm/kvm/mmu.c +4 -4
index b0de86b..071e535 100644
@@ -747,9 +747,9 @@ int kvm_mmu_init(void)
 {
 	int err;
 
-	hyp_idmap_start = virt_to_phys(__hyp_idmap_text_start);
-	hyp_idmap_end = virt_to_phys(__hyp_idmap_text_end);
-	hyp_idmap_vector = virt_to_phys(__kvm_hyp_init);
+	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
+	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
+	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
 
 	if ((hyp_idmap_start ^ hyp_idmap_end) & PAGE_MASK) {
 		/*
@@ -776,7 +776,7 @@ int kvm_mmu_init(void)
 		 */
 		kvm_flush_dcache_to_poc(init_bounce_page, len);
 
-		phys_base = virt_to_phys(init_bounce_page);
+		phys_base = kvm_virt_to_phys(init_bounce_page);
 		hyp_idmap_vector += phys_base - hyp_idmap_start;
 		hyp_idmap_start = phys_base;
 		hyp_idmap_end = phys_base + len;
arch/arm64/include/asm/kvm_mmu.h +1 -0
index efe609c..9ce7c88 100644
@@ -130,6 +130,7 @@ static inline void coherent_icache_guest_page(struct kvm *kvm, gfn_t gfn)
 }
 
 #define kvm_flush_dcache_to_poc(a,l)	__flush_dcache_area((a), (l))
+#define kvm_virt_to_phys(x)		__virt_to_phys((unsigned long)(x))
 
 #endif /* __ASSEMBLY__ */
 #endif /* __ARM64_KVM_MMU_H__ */
Signature · 1 line
Jazz is not dead. It just smells funny...
maintainer
#9 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 43 lines · Marc Zyngier
Quoted 42 lines · Santosh Shilimkar
Quoted 41 lines · Marc Zyngier
Quoted 26 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +1 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
 /*
+ * Added to keep arm64 kvm build working which shares code with

arch/arm64/include/asm/memory.h line 88

Quoted 10 lines
arch/arm64/include/asm/memory.h +7 -0
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)
+
+/*
  * Convert a physical address to a Page Frame Number and back
  */
 #define	__phys_to_pfn(paddr)	((unsigned long)((paddr) >> PAGE_SHIFT))

I'd rather have a kvm_virt_to_phys() in kvm_mmu.h. That's how we've dealt with that kind of difference so far.

Quoted 5 lines
Quoted 1 line

Are you suggesting something like below ?

Yes, I like it a lot more.

Catalin, what do you think?

It looks better ;)

Signature · 1 line
Catalin
#10 lore raw

PATCH 1/6 arm64: mm: Add __virt_to_idmap() to keep kvm build happy

Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Marc Zyngier
Quoted 42 lines · Santosh Shilimkar
Quoted 41 lines · Marc Zyngier
Quoted 26 lines · Santosh Shilimkar

ARM kvm code will make use of __virt_to_idmap() on arm32 machines as hardware interconnect supported alias of physical memory for idmap purposes. The same code is shared with arm64 bit and hence will break the builds. So we add __virt_to_idmap() which is just __virt_to_phys() on arm64 bit to keep build happy.

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Christoffer Dall <[email protected]>
---
 arch/arm64/include/asm/memory.h |    8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/include/asm/memory.h +1 -0
index 3776217..d9341ee 100644
@@ -75,6 +75,14 @@
 #define __phys_to_virt(x)	((unsigned long)((x) - PHYS_OFFSET + PAGE_OFFSET))
 
 /*
+ * Added to keep arm64 kvm build working which shares code with

arch/arm64/include/asm/memory.h line 88

Quoted 10 lines
arch/arm64/include/asm/memory.h +7 -0
+ * 32bit port. ARM kvm code makes use of __virt_to_idmap() on arm32
+ * machines as hardware interconnect supported alias of physical
+ * memory for idmap purposes.
+ */
+#define virt_to_idmap(x)	__virt_to_phys(x)
+
+/*
  * Convert a physical address to a Page Frame Number and back
  */
 #define	__phys_to_pfn(paddr)	((unsigned long)((paddr) >> PAGE_SHIFT))

I'd rather have a kvm_virt_to_phys() in kvm_mmu.h. That's how we've dealt with that kind of difference so far.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Are you suggesting something like below ?

Yes, I like it a lot more.

Catalin, what do you think?

It looks better ;)

OK then. I will use this approach in my refreshed version. Still need to get around other lingering issue with guest userspace corruption.

Regards,
Santosh

#11 lore raw

PATCH 2/6 ARM: kvm: Use virt_to_idmap instead of virt_to_phys for idmap mappings

In reply to Santosh Shilimkar

KVM initialisation fails on architectures implementing virt_to_idmap() because virt_to_phys() on such architectures won't fetch you the correct idmap page.

So update the KVM ARM code to use the virt_to_idmap() to fix the issue.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 arch/arm/kvm/mmu.c |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
arch/arm/kvm/mmu.c +4 -5
index b0de86b..3bd652f 100644
@@ -747,9 +747,9 @@ int kvm_mmu_init(void)
 {
 	int err;
 
-	hyp_idmap_start = virt_to_phys(__hyp_idmap_text_start);
-	hyp_idmap_end = virt_to_phys(__hyp_idmap_text_end);
-	hyp_idmap_vector = virt_to_phys(__kvm_hyp_init);
+	hyp_idmap_start = virt_to_idmap(__hyp_idmap_text_start);
+	hyp_idmap_end = virt_to_idmap(__hyp_idmap_text_end);
+	hyp_idmap_vector = virt_to_idmap(__kvm_hyp_init);
 
 	if ((hyp_idmap_start ^ hyp_idmap_end) & PAGE_MASK) {
 		/*
@@ -775,8 +775,7 @@ int kvm_mmu_init(void)
 		 * caches off at that point.
 		 */
 		kvm_flush_dcache_to_poc(init_bounce_page, len);
-
-		phys_base = virt_to_phys(init_bounce_page);
+		phys_base = virt_to_idmap(init_bounce_page);
 		hyp_idmap_vector += phys_base - hyp_idmap_start;
 		hyp_idmap_start = phys_base;
 		hyp_idmap_end = phys_base + len;
Signature · 1 line
1.7.9.5
#12 lore raw

PATCH 2/6 ARM: kvm: Use virt_to_idmap instead of virt_to_phys for idmap mappings

Quoted 43 lines · Santosh Shilimkar

KVM initialisation fails on architectures implementing virt_to_idmap() because virt_to_phys() on such architectures won't fetch you the correct idmap page.

So update the KVM ARM code to use the virt_to_idmap() to fix the issue.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 arch/arm/kvm/mmu.c |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
arch/arm/kvm/mmu.c +4 -5
index b0de86b..3bd652f 100644
@@ -747,9 +747,9 @@ int kvm_mmu_init(void)
 {
 	int err;
 
-	hyp_idmap_start = virt_to_phys(__hyp_idmap_text_start);
-	hyp_idmap_end = virt_to_phys(__hyp_idmap_text_end);
-	hyp_idmap_vector = virt_to_phys(__kvm_hyp_init);
+	hyp_idmap_start = virt_to_idmap(__hyp_idmap_text_start);
+	hyp_idmap_end = virt_to_idmap(__hyp_idmap_text_end);
+	hyp_idmap_vector = virt_to_idmap(__kvm_hyp_init);
 
 	if ((hyp_idmap_start ^ hyp_idmap_end) & PAGE_MASK) {
 		/*
@@ -775,8 +775,7 @@ int kvm_mmu_init(void)
 		 * caches off at that point.
 		 */
 		kvm_flush_dcache_to_poc(init_bounce_page, len);
-
-		phys_base = virt_to_phys(init_bounce_page);
+		phys_base = virt_to_idmap(init_bounce_page);
 		hyp_idmap_vector += phys_base - hyp_idmap_start;
 		hyp_idmap_start = phys_base;
 		hyp_idmap_end = phys_base + len;
Signature · 1 line
1.7.9.5

Looks good to me.

#13 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

In reply to Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5
#14 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 38 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

We use the check in create_hyp_mappings to be sure that the physical address returned by virt_to_phys is valid and that if we're mapping more than one page that those pages are physically contiguous.

So if you want to get rid of this check, you need to change the mapping functionality to obtain the physical address by walking the page table mappings for each page that you are mapping instead. Or limit each call to a single page in size and take the physical address as input and use per_cpu_ptr_to_phys at the caller side instead.

Alternatively, we need to get rid of alloc_percpu and use regular kmallocs instead, unless anyone else knows of an even better way.

-Christoffer

#15 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 35 lines · Christoffer Dall
Quoted 34 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
Quoted 9 lines
Quoted 4 lines
arch/arm/include/asm/memory.h +0 -0
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

I also thought it might not be right fix and hence added a disclaimer in the commit message ;-)

Regards,
Santosh

#16 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 45 lines · Santosh Shilimkar
Quoted 35 lines · Christoffer Dall
Quoted 34 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
Quoted 9 lines
Quoted 4 lines
arch/arm/include/asm/memory.h +0 -0
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

Quoted 3 lines

I also thought it might not be right fix and hence added a disclaimer in the commit message ;-)

Yes I know, I'm not holding it against you ;)

Not sure what the nicest fix is though. I think we can get away with the single page restriction per call for Hyp mappings for now. IIRC the Hyp code is still limited to be within a single page.

Let's see what Marc has to say.

-Christoffer

#17 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 49 lines · Christoffer Dall
Quoted 45 lines · Santosh Shilimkar
Quoted 35 lines · Christoffer Dall
Quoted 34 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
Quoted 9 lines
Quoted 4 lines
arch/arm/include/asm/memory.h +0 -0
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

Quoted 3 lines

I also thought it might not be right fix and hence added a disclaimer in the commit message ;-)

Quoted 7 lines

Yes I know, I'm not holding it against you ;)

Not sure what the nicest fix is though. I think we can get away with the single page restriction per call for Hyp mappings for now. IIRC the Hyp code is still limited to be within a single page.

Let's see what Marc has to say.

So that may change quite quickly. I already have additional code for GICv3 in HYP, and I'd like to avoid putting restrictions that we'd have a hard time removing later.

I'd like to try the approach I just sent out in a separate email first, and if it can't be made to work, we can see how to either enforce a single-page restriction or move the percpu stuff to kmalloc instead.

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#18 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

In reply to Christoffer Dall
Quoted 48 lines · Christoffer Dall
Quoted 38 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

We use the check in create_hyp_mappings to be sure that the physical address returned by virt_to_phys is valid and that if we're mapping more than one page that those pages are physically contiguous.

Quoted 8 lines

So if you want to get rid of this check, you need to change the mapping functionality to obtain the physical address by walking the page table mappings for each page that you are mapping instead. Or limit each call to a single page in size and take the physical address as input and use per_cpu_ptr_to_phys at the caller side instead.

Alternatively, we need to get rid of alloc_percpu and use regular kmallocs instead, unless anyone else knows of an even better way.

alloc_percpu has nice properties (cache locality, mostly).

One way out of it would be to give percpu stuff a special treatment. Can you try the attach patch as a first approximation? It needs more refinements (allocations straddling two pages?), but I think that's the right sort of things.

Let me know how it works for you.

	M.
Signature · 8 lines
Jazz is not dead. It just smells funny...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-arm-arm64-KVM-introduce-new-mapping-API-for-percpu-m.patch
Type: text/x-diff
Size: 3340 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20131115/88381749/attachment.bin>
#19 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 57 lines · Marc Zyngier
Quoted 48 lines · Christoffer Dall
Quoted 48 lines · Santosh Shilimkar
Quoted 34 lines
Quoted 34 lines

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
Quoted 4 lines
Quoted 4 lines
arch/arm/include/asm/memory.h +0 -0
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

We use the check in create_hyp_mappings to be sure that the physical address returned by virt_to_phys is valid and that if we're mapping more than one page that those pages are physically contiguous.

Quoted 8 lines
Quoted 8 lines

So if you want to get rid of this check, you need to change the mapping functionality to obtain the physical address by walking the page table mappings for each page that you are mapping instead. Or limit each call to a single page in size and take the physical address as input and use per_cpu_ptr_to_phys at the caller side instead.

Alternatively, we need to get rid of alloc_percpu and use regular kmallocs instead, unless anyone else knows of an even better way.

Quoted 9 lines

alloc_percpu has nice properties (cache locality, mostly).

One way out of it would be to give percpu stuff a special treatment. Can you try the attach patch as a first approximation? It needs more refinements (allocations straddling two pages?), but I think that's the right sort of things.

Let me know how it works for you.

Host boots bug guest fails. Patch needs small update as mentioned with inline patch.

Quoted 56 lines

From 01bc1c8eaebdd70b1ea044050144b9bfb3375f82 Mon Sep 17 00:00:00 2001
From: Marc Zyngier <[email protected]>
Date: Fri, 15 Nov 2013 11:36:36 +0000
Subject: [PATCH] arm/arm64: KVM: introduce new mapping API for percpu mappings

Using virt_to_phys on percpu mappings is horribly wrong (my own bad). Thankfully, the kernel offers a way to obtain the phisical address of such a mapping.

Add a new "create_hyp_percpu_mappings" to deal with those.

*Fully untested, don't merge*

Not-Even-Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm/include/asm/kvm_mmu.h   |  1 +
 arch/arm/kvm/arm.c               |  2 +-
 arch/arm/kvm/mmu.c               | 20 ++++++++++++++++++++
 arch/arm64/include/asm/kvm_mmu.h |  1 +
 4 files changed, 23 insertions(+), 1 deletion(-)
arch/arm/include/asm/kvm_mmu.h +1 -0
index 9b28c41..6dcb9ff 100644
@@ -43,6 +43,7 @@
 #include <asm/pgalloc.h>
 
 int create_hyp_mappings(void *from, void *to);
+int create_hyp_percpu_mappings(void *from, void *to);
 int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
 void free_boot_hyp_pgd(void);
 void free_hyp_pgds(void);
arch/arm/kvm/arm.c +1 -1
index 9c697db..6191960 100644
@@ -911,7 +911,7 @@ static int init_hyp_mode(void)
 		kvm_cpu_context_t *cpu_ctxt;
 
 		cpu_ctxt = per_cpu_ptr(kvm_host_cpu_state, cpu);
-		err = create_hyp_mappings(cpu_ctxt, cpu_ctxt + 1);
+		err = create_hyp_percpu_mappings(cpu_ctxt, cpu_ctxt + 1);
 
 		if (err) {
 			kvm_err("Cannot map host CPU state: %d\n", err);
arch/arm/kvm/mmu.c +2 -0
index b0de86b..e509718 100644
@@ -331,6 +331,26 @@ int create_hyp_mappings(void *from, void *to)
 }
 
 /**
+ * create_hyp_percpu_mappings - duplicate a percpu kernel virtual address
+ *				range in Hyp mode

arch/arm/kvm/mmu.c int create_hyp_mappings(void *from, void *to) · line 345

Quoted 10 lines
arch/arm/kvm/mmu.c +10 -0
+ * @from:	The virtual kernel start address of the range
+ * @to:		The virtual kernel end address of the range (exclusive)
+ *
+ * The same virtual address as the kernel virtual address is also used
+ * in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
+ * physical pages. It *has* to be a percpu pointer.
+ */
+int create_hyp_percpu_mappings(void *from, void *to)
+{
+	unsigned long phys_addr = per_cpu_ptr_to_phys(from);

phys_addr_t phys_addr = per_cpu_ptr_to_phys(from);

With this change things work as expected without $subject patch. Thanks for the patch.

Regards,
Ssantosh

#20 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 123 lines · Santosh Shilimkar
Quoted 57 lines · Marc Zyngier
Quoted 48 lines · Christoffer Dall
Quoted 48 lines · Santosh Shilimkar
Quoted 34 lines
Quoted 34 lines

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
Quoted 4 lines
Quoted 4 lines
arch/arm/include/asm/memory.h +0 -0
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

We use the check in create_hyp_mappings to be sure that the physical address returned by virt_to_phys is valid and that if we're mapping more than one page that those pages are physically contiguous.

Quoted 8 lines
Quoted 8 lines

So if you want to get rid of this check, you need to change the mapping functionality to obtain the physical address by walking the page table mappings for each page that you are mapping instead. Or limit each call to a single page in size and take the physical address as input and use per_cpu_ptr_to_phys at the caller side instead.

Alternatively, we need to get rid of alloc_percpu and use regular kmallocs instead, unless anyone else knows of an even better way.

Quoted 9 lines

alloc_percpu has nice properties (cache locality, mostly).

One way out of it would be to give percpu stuff a special treatment. Can you try the attach patch as a first approximation? It needs more refinements (allocations straddling two pages?), but I think that's the right sort of things.

Let me know how it works for you.

Host boots bug guest fails. Patch needs small update as mentioned with inline patch.

Quoted 53 lines

From 01bc1c8eaebdd70b1ea044050144b9bfb3375f82 Mon Sep 17 00:00:00 2001
From: Marc Zyngier <[email protected]>
Date: Fri, 15 Nov 2013 11:36:36 +0000
Subject: [PATCH] arm/arm64: KVM: introduce new mapping API for percpu mappings

Using virt_to_phys on percpu mappings is horribly wrong (my own bad). Thankfully, the kernel offers a way to obtain the phisical address of such a mapping.

Add a new "create_hyp_percpu_mappings" to deal with those.

*Fully untested, don't merge*

Not-Even-Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm/include/asm/kvm_mmu.h   |  1 +
 arch/arm/kvm/arm.c               |  2 +-
 arch/arm/kvm/mmu.c               | 20 ++++++++++++++++++++
 arch/arm64/include/asm/kvm_mmu.h |  1 +
 4 files changed, 23 insertions(+), 1 deletion(-)
arch/arm/include/asm/kvm_mmu.h +1 -0
index 9b28c41..6dcb9ff 100644
@@ -43,6 +43,7 @@
 #include <asm/pgalloc.h>
 
 int create_hyp_mappings(void *from, void *to);
+int create_hyp_percpu_mappings(void *from, void *to);
 int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
 void free_boot_hyp_pgd(void);
 void free_hyp_pgds(void);
arch/arm/kvm/arm.c +1 -1
index 9c697db..6191960 100644
@@ -911,7 +911,7 @@ static int init_hyp_mode(void)
 		kvm_cpu_context_t *cpu_ctxt;
 
 		cpu_ctxt = per_cpu_ptr(kvm_host_cpu_state, cpu);
-		err = create_hyp_mappings(cpu_ctxt, cpu_ctxt + 1);
+		err = create_hyp_percpu_mappings(cpu_ctxt, cpu_ctxt + 1);
 
 		if (err) {
 			kvm_err("Cannot map host CPU state: %d\n", err);
arch/arm/kvm/mmu.c +0 -0
index b0de86b..e509718 100644
@@ -331,6 +331,26 @@ int create_hyp_mappings(void *from, void *to)
 }
 
Quoted 14 lines

arch/arm/kvm/mmu.c int create_hyp_mappings(void *from, void *to) · line 345

Quoted 13 lines
arch/arm/kvm/mmu.c +12 -0
 /**
+ * create_hyp_percpu_mappings - duplicate a percpu kernel virtual address
+ *				range in Hyp mode
+ * @from:	The virtual kernel start address of the range
+ * @to:		The virtual kernel end address of the range (exclusive)
+ *
+ * The same virtual address as the kernel virtual address is also used
+ * in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
+ * physical pages. It *has* to be a percpu pointer.
+ */
+int create_hyp_percpu_mappings(void *from, void *to)
+{
+	unsigned long phys_addr = per_cpu_ptr_to_phys(from);

phys_addr_t phys_addr = per_cpu_ptr_to_phys(from);

Yeah, of course... ;-)

Quoted 2 lines

With this change things work as expected without $subject patch. Thanks for the patch.

Good. I'll respin another version with support for allocations straddling multiple pages and post it ASAP.

Thanks for testing!

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#21 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 139 lines · Marc Zyngier
Quoted 123 lines · Santosh Shilimkar
Quoted 57 lines · Marc Zyngier
Quoted 48 lines · Christoffer Dall
Quoted 48 lines · Santosh Shilimkar
Quoted 34 lines
Quoted 34 lines

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/memory.h +1 -2
index 4dd2145..412da47 100644
@@ -343,8 +343,7 @@ static inline __deprecated void *bus_to_virt(unsigned long x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET && (unsigned long)(kaddr) < (unsigned long)high_memory)
-
+#define virt_addr_valid(kaddr)	((unsigned long)(kaddr) >= PAGE_OFFSET)
 #endif
 
Quoted 4 lines
Quoted 4 lines
arch/arm/include/asm/memory.h +0 -0
 #include <asm-generic/memory_model.h>
Signature · 1 line
1.7.9.5

This looks wrong to me. Check Documentation/arm/memory.txt, this would return true for the VMALLOC region, which would cause virt_to_phys to give you something invalid, which would be bad.

We use the check in create_hyp_mappings to be sure that the physical address returned by virt_to_phys is valid and that if we're mapping more than one page that those pages are physically contiguous.

Quoted 8 lines
Quoted 8 lines

So if you want to get rid of this check, you need to change the mapping functionality to obtain the physical address by walking the page table mappings for each page that you are mapping instead. Or limit each call to a single page in size and take the physical address as input and use per_cpu_ptr_to_phys at the caller side instead.

Alternatively, we need to get rid of alloc_percpu and use regular kmallocs instead, unless anyone else knows of an even better way.

Quoted 9 lines

alloc_percpu has nice properties (cache locality, mostly).

One way out of it would be to give percpu stuff a special treatment. Can you try the attach patch as a first approximation? It needs more refinements (allocations straddling two pages?), but I think that's the right sort of things.

Let me know how it works for you.

Host boots bug guest fails. Patch needs small update as mentioned with inline patch.

Quoted 53 lines

From 01bc1c8eaebdd70b1ea044050144b9bfb3375f82 Mon Sep 17 00:00:00 2001
From: Marc Zyngier <[email protected]>
Date: Fri, 15 Nov 2013 11:36:36 +0000
Subject: [PATCH] arm/arm64: KVM: introduce new mapping API for percpu mappings

Using virt_to_phys on percpu mappings is horribly wrong (my own bad). Thankfully, the kernel offers a way to obtain the phisical address of such a mapping.

Add a new "create_hyp_percpu_mappings" to deal with those.

*Fully untested, don't merge*

Not-Even-Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm/include/asm/kvm_mmu.h   |  1 +
 arch/arm/kvm/arm.c               |  2 +-
 arch/arm/kvm/mmu.c               | 20 ++++++++++++++++++++
 arch/arm64/include/asm/kvm_mmu.h |  1 +
 4 files changed, 23 insertions(+), 1 deletion(-)
arch/arm/include/asm/kvm_mmu.h +1 -0
index 9b28c41..6dcb9ff 100644
@@ -43,6 +43,7 @@
 #include <asm/pgalloc.h>
 
 int create_hyp_mappings(void *from, void *to);
+int create_hyp_percpu_mappings(void *from, void *to);
 int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
 void free_boot_hyp_pgd(void);
 void free_hyp_pgds(void);
arch/arm/kvm/arm.c +1 -1
index 9c697db..6191960 100644
@@ -911,7 +911,7 @@ static int init_hyp_mode(void)
 		kvm_cpu_context_t *cpu_ctxt;
 
 		cpu_ctxt = per_cpu_ptr(kvm_host_cpu_state, cpu);
-		err = create_hyp_mappings(cpu_ctxt, cpu_ctxt + 1);
+		err = create_hyp_percpu_mappings(cpu_ctxt, cpu_ctxt + 1);
 
 		if (err) {
 			kvm_err("Cannot map host CPU state: %d\n", err);
arch/arm/kvm/mmu.c +0 -0
index b0de86b..e509718 100644
@@ -331,6 +331,26 @@ int create_hyp_mappings(void *from, void *to)
 }
 
Quoted 14 lines

arch/arm/kvm/mmu.c int create_hyp_mappings(void *from, void *to) · line 345

Quoted 13 lines
arch/arm/kvm/mmu.c +12 -0
 /**
+ * create_hyp_percpu_mappings - duplicate a percpu kernel virtual address
+ *				range in Hyp mode
+ * @from:	The virtual kernel start address of the range
+ * @to:		The virtual kernel end address of the range (exclusive)
+ *
+ * The same virtual address as the kernel virtual address is also used
+ * in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
+ * physical pages. It *has* to be a percpu pointer.
+ */
+int create_hyp_percpu_mappings(void *from, void *to)
+{
+	unsigned long phys_addr = per_cpu_ptr_to_phys(from);

phys_addr_t phys_addr = per_cpu_ptr_to_phys(from);

Quoted 8 lines

Yeah, of course... ;-)

Quoted 2 lines

With this change things work as expected without $subject patch. Thanks for the patch.

Good. I'll respin another version with support for allocations straddling multiple pages and post it ASAP.

Marc, hold on, can't we just make the create_hyp_mappings more generic?

I think it would be much cleaner to, either:

1) use the existing function, but take a physical address and let the caller figure that part out, and limit mappings to a single page

2) make create_hyp_mappings handle the full thing, check if the addr can be translated with virt_to_phys and otherwise do page_to_phys(vmalloc_tp_page(addr)), and handle cross-page mappings. Basically it's what the tail end of per_cpu_ptr_to_phys does, only more generically for any allocation.

-Christoffer

#22 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

In reply to Santosh Shilimkar
Quoted 2 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

This is definitely going to cause problems.

Quoted 6 lines

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

virt_addr_valid() gets used in some places to check whether the virtual address is part of the lowmem mapping and explicitly not part of vmalloc() or DMA coherent space:

drivers/mtd/nand/gpmi-nand/gpmi-nand.c
drivers/net/wireless/ath/ath6kl/sdio.c

It opens up the checks in include/linux/scatterlist.h to start accepting non-streaming DMA capable buffers as well.

It also bypasses a check in the slab code to ensure that it's a potentially valid pointer that was handed out by slab.

.. etc ..

#23 lore raw

PATCH 3/6 ARM: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 17 lines · Russell King - ARM Linux
Quoted 2 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

This is definitely going to cause problems.

Quoted 6 lines

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

virt_addr_valid() gets used in some places to check whether the virtual address is part of the lowmem mapping and explicitly not part of vmalloc() or DMA coherent space:

Quoted 9 lines

drivers/mtd/nand/gpmi-nand/gpmi-nand.c
drivers/net/wireless/ath/ath6kl/sdio.c

It opens up the checks in include/linux/scatterlist.h to start accepting non-streaming DMA capable buffers as well.

It also bypasses a check in the slab code to ensure that it's a potentially valid pointer that was handed out by slab.

Thanks for the additional information Russell. We are going with an alternate approach from Marc so this patch will be dropped any ways.

Regards,
Santosh

#24 lore raw

PATCH 4/6 arm64: mm: Drop the lowmem watermark check from virt_addr_valid()

In reply to Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/memory.h |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm64/include/asm/memory.h +1 -2
index d9341ee..d04d047 100644
@@ -154,8 +154,7 @@ static inline void *phys_to_virt(phys_addr_t x)
 #define ARCH_PFN_OFFSET		PHYS_PFN_OFFSET
 
 #define virt_to_page(kaddr)	pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define	virt_addr_valid(kaddr)	(((void *)(kaddr) >= (void *)PAGE_OFFSET) && \
-				 ((void *)(kaddr) < (void *)high_memory))
+#define	virt_addr_valid(kaddr)	((void *)(kaddr) >= (void *)PAGE_OFFSET)
 
 #endif
 
Signature · 1 line
1.7.9.5
maintainer
#25 lore raw

PATCH 4/6 arm64: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 9 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Do you have a problem on arm64? There is no lowmem watermark here.

Signature · 1 line
Catalin
#26 lore raw

PATCH 4/6 arm64: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 4 lines · Catalin Marinas
Quoted 3 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

Quoted 8 lines
Quoted 6 lines

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Do you have a problem on arm64? There is no lowmem watermark here.

I don't think that change is relevant for arm64. Actually, the more I look at it, I don't think virt_addr_valid() should be touched at all.

The real issue is the use of virt_to_phys()/virt_addr_valid() on a percpu pointer, which is wrong and only worked by chance so far.

I'm working on a patch to solve this.

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#27 lore raw

PATCH 4/6 arm64: mm: Drop the lowmem watermark check from virt_addr_valid()

Quoted 14 lines · Marc Zyngier
Quoted 4 lines · Catalin Marinas
Quoted 3 lines · Santosh Shilimkar

Slab allocator can allocate memory beyond the lowmem watermark which can lead to false failure of virt_addr_valid().

Quoted 8 lines
Quoted 6 lines

So drop the check. The issue was seen with percpu_alloc() in KVM code which was allocating memory beyond lowmem watermark.

Am not completly sure whether this is the right fix and if it could impact any other user of virt_addr_valid(). Without this fix as pointed out the KVM init was failing in my testing.

Do you have a problem on arm64? There is no lowmem watermark here.

Quoted 8 lines

I don't think that change is relevant for arm64. Actually, the more I look at it, I don't think virt_addr_valid() should be touched at all.

The real issue is the use of virt_to_phys()/virt_addr_valid() on a percpu pointer, which is wrong and only worked by chance so far.

I'm working on a patch to solve this.

Yep. I tried Marc's other approach and that does solve the issue. The $subject patch was any way more for getting attention to figure out right solution ;-) which I guess we have now.

Regards,
Santosh

#28 lore raw

PATCH 5/6 ARM: kvm: Use phys_addr_t instead of unsigned long in mm code

In reply to Santosh Shilimkar

The unsigned long datatype is not sufficient for mapping physical addresses greater than 4GB.

So fix the KVM mm code accordingly. Special thanks to Christopher for debug help to figure out the bug.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 arch/arm/kvm/mmu.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kvm/mmu.c +1 -1
index 3bd652f..657f15e 100644
@@ -318,7 +318,7 @@ out:
  */
 int create_hyp_mappings(void *from, void *to)
 {
-	unsigned long phys_addr = virt_to_phys(from);
+	phys_addr_t phys_addr = virt_to_phys(from);
 	unsigned long start = KERN_TO_HYP((unsigned long)from);
 	unsigned long end = KERN_TO_HYP((unsigned long)to);
 
Signature · 1 line
1.7.9.5
#29 lore raw

PATCH 5/6 ARM: kvm: Use phys_addr_t instead of unsigned long in mm code

Quoted 30 lines · Santosh Shilimkar

The unsigned long datatype is not sufficient for mapping physical addresses greater than 4GB.

So fix the KVM mm code accordingly. Special thanks to Christopher for debug help to figure out the bug.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 arch/arm/kvm/mmu.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kvm/mmu.c +1 -1
index 3bd652f..657f15e 100644
@@ -318,7 +318,7 @@ out:
  */
 int create_hyp_mappings(void *from, void *to)
 {
-	unsigned long phys_addr = virt_to_phys(from);
+	phys_addr_t phys_addr = virt_to_phys(from);
 	unsigned long start = KERN_TO_HYP((unsigned long)from);
 	unsigned long end = KERN_TO_HYP((unsigned long)to);
 
Signature · 1 line
1.7.9.5

I can just apply this to kvm-arm-next immediately, or we can wait and pull it in with your series if you prefer. Let me know.

-Christoffer

#30 lore raw

PATCH 5/6 ARM: kvm: Use phys_addr_t instead of unsigned long in mm code

Quoted 28 lines · Christoffer Dall
Quoted 27 lines · Santosh Shilimkar

The unsigned long datatype is not sufficient for mapping physical addresses greater than 4GB.

So fix the KVM mm code accordingly. Special thanks to Christopher for debug help to figure out the bug.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 arch/arm/kvm/mmu.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kvm/mmu.c +1 -1
index 3bd652f..657f15e 100644
@@ -318,7 +318,7 @@ out:
  */
 int create_hyp_mappings(void *from, void *to)
 {
-	unsigned long phys_addr = virt_to_phys(from);
+	phys_addr_t phys_addr = virt_to_phys(from);
 	unsigned long start = KERN_TO_HYP((unsigned long)from);
 	unsigned long end = KERN_TO_HYP((unsigned long)to);
 
Quoted 6 lines
Quoted 3 lines
Signature · 1 line
1.7.9.5

Thanks

Quoted 3 lines

I can just apply this to kvm-arm-next immediately, or we can wait and pull it in with your series if you prefer. Let me know.

Feel free to pick this up since this one can go as bug fix as well.

Regards,
Santosh

#31 lore raw

PATCH 5/6 ARM: kvm: Use phys_addr_t instead of unsigned long in mm code

In reply to Santosh Shilimkar
Quoted 10 lines · Santosh Shilimkar

The unsigned long datatype is not sufficient for mapping physical addresses greater than 4GB.

So fix the KVM mm code accordingly. Special thanks to Christopher for debug help to figure out the bug.

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>

Nice catch.

	M.
Quoted 18 lines
---
 arch/arm/kvm/mmu.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kvm/mmu.c +1 -1
index 3bd652f..657f15e 100644
@@ -318,7 +318,7 @@ out:
  */
 int create_hyp_mappings(void *from, void *to)
 {
-	unsigned long phys_addr = virt_to_phys(from);
+	phys_addr_t phys_addr = virt_to_phys(from);
 	unsigned long start = KERN_TO_HYP((unsigned long)from);
 	unsigned long end = KERN_TO_HYP((unsigned long)to);
 
Signature · 1 line
Jazz is not dead. It just smells funny...
#32 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

In reply to Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

I still don't know why this is needed but it helps to get around the issue and hence including the patch in the series for the discussion

Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/kvm/mmu.c |    1 +
 1 file changed, 1 insertion(+)
arch/arm/kvm/mmu.c +1 -0
index 657f15e..5f6f460 100644
@@ -826,6 +826,7 @@ int kvm_mmu_init(void)
 		goto out;
 	}
 
+	flush_cache_all();
 	return 0;
 out:
 	free_hyp_pgds();
Signature · 1 line
1.7.9.5
#33 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 8 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

I still don't know why this is needed but it helps to get around the issue and hence including the patch in the series for the discussion

Quoted 8 lines
Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>

---

Ignore this patch. I thought it helps but in more testing I found that issue still comes up even with the below change.

Quoted 16 lines
 arch/arm/kvm/mmu.c |    1 +
 1 file changed, 1 insertion(+)
arch/arm/kvm/mmu.c +1 -0
index 657f15e..5f6f460 100644
@@ -826,6 +826,7 @@ int kvm_mmu_init(void)
 		goto out;
 	}
 
+	flush_cache_all();
 	return 0;
 out:
 	free_hyp_pgds();
#34 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

In reply to Santosh Shilimkar
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

what crashes? The guest? Where, how?

Quoted 2 lines

I still don't know why this is needed but it helps to get around the issue and hence including the patch in the series for the discussion

It may not be needed but is just hiding the issue. I'm afraid you're going to have to dig a little more into this.

-Christoffer

Quoted 27 lines
Cc Christoffer Dall <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Russell King <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/kvm/mmu.c |    1 +
 1 file changed, 1 insertion(+)
arch/arm/kvm/mmu.c +1 -0
index 657f15e..5f6f460 100644
@@ -826,6 +826,7 @@ int kvm_mmu_init(void)
 		goto out;
 	}
 
+	flush_cache_all();
 	return 0;
 out:
 	free_hyp_pgds();
Signature · 1 line
1.7.9.5
#35 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 9 lines · Christoffer Dall
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

what crashes? The guest? Where, how?

When guest userspace starts. The crashes are random but always after the guest init process have started.

Quoted 7 lines
Quoted 3 lines

I still don't know why this is needed but it helps to get around the issue and hence including the patch in the series for the discussion

It may not be needed but is just hiding the issue. I'm afraid you're going to have to dig a little more into this.

I replied already on this. Further testing, I found the issue even with this patch applied. I need to dig bit more as you said.

Regards,
Santosh

#36 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 8 lines · Santosh Shilimkar
Quoted 7 lines · Christoffer Dall
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

Quoted 5 lines
Quoted 2 lines

what crashes? The guest? Where, how?

When guest userspace starts. The crashes are random but always after the guest init process have started.

So you get a guest kernel crash when guest userspace starts?

Are the crashes completely random or is it always some pointer dereference that goes wrong, is it init crashing and causing the kernel to crash (from killed init), or is it always the same kernel thread, or anything coherent at all?

It could be anything, really. You could try a really brute force debugging option of adding a complete cache flush at the end of user_mem_abort in arch/arm/kvm/mmu.c to see if this is cache related at all...

Are you running with huge pages enabled?

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

I still don't know why this is needed but it helps to get around the issue and hence including the patch in the series for the discussion

It may not be needed but is just hiding the issue. I'm afraid you're going to have to dig a little more into this.

I replied already on this. Further testing, I found the issue even with this patch applied. I need to dig bit more as you said.

Yeah, sorry I missed your reply before replying to the patch.

-Christoffer

#37 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 15 lines · Christoffer Dall
Quoted 8 lines · Santosh Shilimkar
Quoted 7 lines · Christoffer Dall
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

Quoted 5 lines
Quoted 2 lines

what crashes? The guest? Where, how?

When guest userspace starts. The crashes are random but always after the guest init process have started.

Quoted 7 lines

So you get a guest kernel crash when guest userspace starts?

Are the crashes completely random or is it always some pointer dereference that goes wrong, is it init crashing and causing the kernel to crash (from killed init), or is it always the same kernel thread, or anything coherent at all?

Completely random. I have seen almost all of the above possible crashes like pointer derefence, init process skipping some steps, console going for toss, the log in prompt just won't let me log in etc

Quoted 5 lines

It could be anything, really. You could try a really brute force debugging option of adding a complete cache flush at the end of user_mem_abort in arch/arm/kvm/mmu.c to see if this is cache related at all...

I will try that. I strongly suspect this has to do with bad page tables. remember I see this issue with when using memory which starts beyond 4GB.

Quoted 2 lines

Are you running with huge pages enabled?

Nope.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I still don't know why this is needed but it helps to get around the issue and hence including the patch in the series for the discussion

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

It may not be needed but is just hiding the issue. I'm afraid you're going to have to dig a little more into this.

I replied already on this. Further testing, I found the issue even with this patch applied. I need to dig bit more as you said.

Yeah, sorry I missed your reply before replying to the patch.

np. Thanks

regards,
Santosh

#38 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 28 lines · Santosh Shilimkar
Quoted 15 lines · Christoffer Dall
Quoted 8 lines · Santosh Shilimkar
Quoted 7 lines · Christoffer Dall
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

Quoted 5 lines
Quoted 2 lines

what crashes? The guest? Where, how?

When guest userspace starts. The crashes are random but always after the guest init process have started.

Quoted 7 lines

So you get a guest kernel crash when guest userspace starts?

Are the crashes completely random or is it always some pointer dereference that goes wrong, is it init crashing and causing the kernel to crash (from killed init), or is it always the same kernel thread, or anything coherent at all?

Completely random. I have seen almost all of the above possible crashes like pointer derefence, init process skipping some steps, console going for toss, the log in prompt just won't let me log in etc

Quoted 9 lines
Quoted 5 lines

It could be anything, really. You could try a really brute force debugging option of adding a complete cache flush at the end of user_mem_abort in arch/arm/kvm/mmu.c to see if this is cache related at all...

I will try that. I strongly suspect this has to do with bad page tables. remember I see this issue with when using memory which starts beyond 4GB.

But once it crashes, if you kill the VM process and start a new one, then the new one runs flawlessly? Did you stress test the second VM (hackbench or something) so we're sure the second one is indeed stable?

What happens if you start a guest, kill it immediately, and then start another guest?

-Christoffer

#39 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 29 lines · Christoffer Dall
Quoted 28 lines · Santosh Shilimkar
Quoted 15 lines · Christoffer Dall
Quoted 8 lines · Santosh Shilimkar
Quoted 7 lines · Christoffer Dall
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

Quoted 5 lines
Quoted 2 lines

what crashes? The guest? Where, how?

When guest userspace starts. The crashes are random but always after the guest init process have started.

Quoted 7 lines

So you get a guest kernel crash when guest userspace starts?

Are the crashes completely random or is it always some pointer dereference that goes wrong, is it init crashing and causing the kernel to crash (from killed init), or is it always the same kernel thread, or anything coherent at all?

Completely random. I have seen almost all of the above possible crashes like pointer derefence, init process skipping some steps, console going for toss, the log in prompt just won't let me log in etc

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

It could be anything, really. You could try a really brute force debugging option of adding a complete cache flush at the end of user_mem_abort in arch/arm/kvm/mmu.c to see if this is cache related at all...

I will try that. I strongly suspect this has to do with bad page tables. remember I see this issue with when using memory which starts beyond 4GB.

full cache full at end of user_mem_abort() doesn't help. So it might not be cache related then.

Quoted 8 lines

But once it crashes, if you kill the VM process and start a new one, then the new one runs flawlessly? Did you stress test the second VM (hackbench or something) so we're sure the second one is indeed stable?

What happens if you start a guest, kill it immediately, and then start another guest?

And the observation about subsequent VM's being stable also doesn't hold true. Additional symptom what I saw was segmentation fault as well as hitting kvm load/store trap. This also possibly indicates instructions corruption.

Regards,
Santosh

#40 lore raw

PATCH 6/6 ARM: kvm: TMP: Commit the hyp page tables to main memory

Quoted 46 lines · Santosh Shilimkar
Quoted 29 lines · Christoffer Dall
Quoted 28 lines · Santosh Shilimkar
Quoted 15 lines · Christoffer Dall
Quoted 8 lines · Santosh Shilimkar
Quoted 7 lines · Christoffer Dall
Quoted 5 lines · Santosh Shilimkar

This is a temporary hack which I have to use to avoid a weired crash while starting the guest OS on Keystsone. They are random crashesh while the guest os userspace starts. Additional data point is, it seen only with first guest OS lanch. Subsequest guest OS starts normal.

Quoted 5 lines
Quoted 2 lines

what crashes? The guest? Where, how?

When guest userspace starts. The crashes are random but always after the guest init process have started.

Quoted 7 lines

So you get a guest kernel crash when guest userspace starts?

Are the crashes completely random or is it always some pointer dereference that goes wrong, is it init crashing and causing the kernel to crash (from killed init), or is it always the same kernel thread, or anything coherent at all?

Completely random. I have seen almost all of the above possible crashes like pointer derefence, init process skipping some steps, console going for toss, the log in prompt just won't let me log in etc

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

It could be anything, really. You could try a really brute force debugging option of adding a complete cache flush at the end of user_mem_abort in arch/arm/kvm/mmu.c to see if this is cache related at all...

I will try that. I strongly suspect this has to do with bad page tables. remember I see this issue with when using memory which starts beyond 4GB.

full cache full at end of user_mem_abort() doesn't help. So it might not be cache related then.

Quoted 5 lines

But once it crashes, if you kill the VM process and start a new one, then the new one runs flawlessly? Did you stress test the second VM (hackbench or something) so we're sure the second one is indeed stable?

Quoted 8 lines
Quoted 3 lines

What happens if you start a guest, kill it immediately, and then start another guest?

And the observation about subsequent VM's being stable also doesn't hold true. Additional symptom what I saw was segmentation fault as well as hitting kvm load/store trap. This also possibly indicates instructions corruption.

Cool, so we only know it breaks when the physical address is >4GB. Awesome.

It may be helpful to cherry-pick this commit:
https://git.linaro.org/gitweb?p=people/cdall/linux-kvm-arm.git;a=commitdiff;h=df6dc9f43f2a37547d4ce034706ef0cfc4235129

Then capture a full trace of the VM when executing until the guest crashes and look at the trace to see if we're mapping and faulting on the pages we think we are or if it looks like something is being truncated.

Feel free to send me one of those logs and I'll be happy to take a look.

-Christoffer