KVM: s390: Add huge page VSIE support
Subsystems (5)
Janosch Frank
maintainercover KVM: s390: Add huge page VSIE support
As we finally want to get rid of the nested and hpage s390 KVM module parameters let's try again to integrate huge page VSIE support.
The following patches have been rebased on 5.11-rc3 and enable us to start huge page and normal page VSIE guest 3s in a huge page guest 2.
Branch:
https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git/log/?h=hlp_vsie
Problems that need to be solved:
* The VSIE guests crash on migration...
* I have lost most of my knowledge about this topic and I'm currently paging back in
* Lots of testing
Janosch Frank (14):
s390/mm: Code cleanups
s390/mm: Improve locking for huge page backings s390/mm: Take locking out of gmap_protect_pte s390/mm: split huge pages in GMAP when protecting s390/mm: Split huge pages when migrating
s390/mm: Provide vmaddr to pmd notification s390/mm: factor out idte global flush into gmap_idte_global s390/mm: Make gmap_read_table EDAT1 compatible s390/mm: Make gmap_protect_rmap EDAT1 compatible s390/mm: Add simple ptep shadow function
s390/mm: Add gmap shadowing for large pmds s390/mm: Add gmap lock classes
s390/mm: Pull pmd invalid check in gmap_pmd_op_walk
KVM: s390: Allow the VSIE to be used with huge pages
Documentation/virt/kvm/api.rst | 9 +- arch/s390/include/asm/gmap.h | 31 +- arch/s390/include/asm/pgtable.h | 5 + arch/s390/kvm/gaccess.c | 52 +- arch/s390/kvm/kvm-s390.c | 14 +- arch/s390/mm/gmap.c | 917 ++++++++++++++++++++++++-------- arch/s390/mm/pgtable.c | 61 ++- 7 files changed, 819 insertions(+), 270 deletions(-)
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 1/14 s390/mm: Code cleanups
Let's clean up leftovers before introducing new code.
--- arch/s390/mm/gmap.c | 8 ++++---- arch/s390/mm/pgtable.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/s390/mm/gmap.c +4 -4
@@ -5,7 +5,7 @@* Copyright IBM Corp. 2007, 2020 * Author(s): Martin Schwidefsky <[email protected]> * David Hildenbrand <[email protected]> -* Janosch Frank <[email protected]> +* Janosch Frank <[email protected]> */ #include <linux/kernel.h>
@@ -2290,10 +2290,10 @@ static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *pmdp, pmd_t new,pmdp_notify_gmap (gmap, pmdp, gaddr);pmd_val (new) &= ~_SEGMENT_ENTRY_GMAP_IN;if (MACHINE_HAS_TLB_GUEST ) -__pmdp_idte (gaddr, (pmd_t *)pmdp,IDTE_GUEST_ASCE , gmap->asce, +__pmdp_idte (gaddr, pmdp,IDTE_GUEST_ASCE , gmap->asce,IDTE_GLOBAL );else if (MACHINE_HAS_IDTE ) -__pmdp_idte (gaddr, (pmd_t *)pmdp,0 ,0 ,IDTE_GLOBAL ); +__pmdp_idte (gaddr, pmdp,0 ,0 ,IDTE_GLOBAL );else __pmdp_csp (pmdp); *pmdp = new;
@@ -2523,7 +2523,7 @@ static inline void thp_split_mm(struct mm_struct *mm)* - This must be called after THP was enabled */ static int __zap_zero_pages (pmd_t *pmd,unsigned long start, -unsigned long end,struct mm_walk *walk) +unsigned long end,struct mm_walk *walk) {unsigned long addr;
arch/s390/mm/pgtable.c +1 -1
@@ -743,7 +743,7 @@ void ptep_zap_key(struct mm_struct *mm, unsigned long addr, pte_t *ptep)* Test and reset if a guest page is dirty */ bool ptep_test_and_clear_uc (struct mm_struct *mm,unsigned long addr, -pte_t *ptep) +pte_t *ptep) {pgste_t pgste;pte_t pte;
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 2/14 s390/mm: Improve locking for huge page backings
In reply to Janosch FrankThe gmap guest_table_lock is used to protect changes to the guest's DAT tables from region 1 to segments. Therefore it also protects the host to guest radix tree where each new segment mapping by gmap_link() is tracked. Changes to ptes are synchronized through the pte lock, which is easily retrievable, because the gmap shares the page tables with userspace.
With huge pages the story changes. PMD tables are not shared and we're left with the pmd lock on userspace side and the guest_table_lock on the gmap side. Having two locks for an object is a guarantee for locking problems.
Therefore the guest_table_lock will only be used for population of the gmap tables and hence protecting the host_to_guest tree. While the pmd lock will be used for all changes to the pmd from both userspace and the gmap.
This means we need to retrieve the vmaddr to retrieve a gmap pmd, which takes a bit longer than before. But we can now operate on multiple pmds which are in disjoint segment tables instead of having a global lock.
--- arch/s390/include/asm/pgtable.h | 1 + arch/s390/mm/gmap.c | 70 ++++++++++++++++++++------------- arch/s390/mm/pgtable.c | 2 +- 3 files changed, 45 insertions(+), 28 deletions(-)
arch/s390/include/asm/pgtable.h +1 -0
@@ -1519,6 +1519,7 @@ static __always_inline void __pudp_idte(unsigned long addr, pud_t *pudp, } } +pmd_t *pmd_alloc_map (struct mm_struct *mm,unsigned long addr);pmd_t pmdp_xchg_direct (struct mm_struct *,unsigned long ,pmd_t *,pmd_t );pmd_t pmdp_xchg_lazy (struct mm_struct *,unsigned long ,pmd_t *,pmd_t );pud_t pudp_xchg_direct (struct mm_struct *,unsigned long ,pud_t *,pud_t );
arch/s390/mm/gmap.c +43 -27
126 lines not on this page · raw message
arch/s390/mm/pgtable.c +1 -1
@@ -429,7 +429,7 @@ static inline pmd_t pmdp_flush_lazy(struct mm_struct *mm, }#ifdef CONFIG_PGSTE -static pmd_t *pmd_alloc_map (struct mm_struct *mm,unsigned long addr) +pmd_t *pmd_alloc_map (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;p4d_t *p4d;
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 3/14 s390/mm: Take locking out of gmap_protect_pte
In reply to Janosch FrankLocking outside of the function gives us the freedom of ordering locks, which will be important to not get locking issues for gmap_protect_rmap.
--- arch/s390/mm/gmap.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
arch/s390/mm/gmap.c +14 -16
@@ -1017,25 +1017,15 @@ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr,* Expected to be called with sg->mm->mmap_lock in read */ static int gmap_protect_pte (struct gmap *gmap,unsigned long gaddr, -pmd_t *pmdp,int prot,unsigned long bits) +pte_t *ptep,int prot,unsigned long bits) {int rc; -pte_t *ptep; -spinlock_t *ptl =NULL ;unsigned long pbits =0 ; -if (pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID) -return -EAGAIN ; - - ptep =pte_alloc_map_lock (gmap->mm, pmdp, gaddr, &ptl); -if (!ptep) -return -ENOMEM ; - pbits |= (bits &GMAP_NOTIFY_MPROT ) ?PGSTE_IN_BIT :0 ; pbits |= (bits &GMAP_NOTIFY_SHADOW ) ?PGSTE_VSIE_BIT :0 ;/* Protect and unlock. */ rc =ptep_force_prot (gmap->mm, gaddr, ptep, prot, pbits); -gmap_pte_op_end (ptl);return rc; }
@@ -1056,18 +1046,26 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, unsigned long len, int prot, unsigned long bits) { unsigned long vmaddr, dist; -spinlock_t *ptl =NULL ; +spinlock_t *ptl_pmd =NULL , *ptl_pte =NULL ;pmd_t *pmdp; +pte_t *ptep;int rc;BUG_ON (gmap_is_shadow (gmap));while (len) { rc = -EAGAIN ; - pmdp =gmap_pmd_op_walk (gmap, gaddr, &ptl); + pmdp =gmap_pmd_op_walk (gmap, gaddr, &ptl_pmd);if (pmdp) {if (!pmd_large (*pmdp)) { - rc =gmap_protect_pte (gmap, gaddr, pmdp, prot, - bits); + ptl_pte =NULL ; + ptep =pte_alloc_map_lock (gmap->mm, pmdp, gaddr, + &ptl_pte); +if (ptep) + rc =gmap_protect_pte (gmap, gaddr, + ptep, prot, bits); +else + rc = -ENOMEM ; +gmap_pte_op_end (ptl_pte);if (!rc) { len -=PAGE_SIZE ; gaddr +=PAGE_SIZE ;
@@ -1081,7 +1079,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, gaddr = (gaddr &HPAGE_MASK ) +HPAGE_SIZE ; } } -gmap_pmd_op_end (ptl); +gmap_pmd_op_end (ptl_pmd); }if (rc) {if (rc == -EINVAL )
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 4/14 s390/mm: split huge pages in GMAP when protecting
In reply to Janosch FrankDirty tracking, vsie protection and lowcore invalidation notification are best done on the smallest page size available to avoid unnecessary flushing and table management operations.
Hence we now split huge pages and introduce a page table if a notification bit is set or memory is protected via gmap_protect_range or gmap_protect_rmap.
--- arch/s390/include/asm/gmap.h | 18 +++ arch/s390/include/asm/pgtable.h | 3 + arch/s390/mm/gmap.c | 247 +++++++++++++++++++++++++------- arch/s390/mm/pgtable.c | 33 +++++ 4 files changed, 251 insertions(+), 50 deletions(-)
arch/s390/include/asm/gmap.h +18 -0
@@ -19,6 +19,11 @@/* Status bits only for huge segment entries */ #define _SEGMENT_ENTRY_GMAP_IN 0x8000/* invalidation notify bit */ #define _SEGMENT_ENTRY_GMAP_UC 0x4000/* dirty (migration) */ +/* Status bits in the gmap segment entry. */ +#define _SEGMENT_ENTRY_GMAP_SPLIT 0x0001/* split huge pmd */ + +#define GMAP_SEGMENT_STATUS_BITS (_SEGMENT_ENTRY_GMAP_UC | _SEGMENT_ENTRY_GMAP_SPLIT) +#define GMAP_SEGMENT_NOTIFY_BITS _SEGMENT_ENTRY_GMAP_IN/** * struct gmap_struct - guest address space
@@ -62,6 +67,8 @@ struct gmap {struct radix_tree_root host_to_rmap;struct list_head children;struct list_head pt_list; +struct list_head split_list; +spinlock_t split_list_lock;spinlock_t shadow_lock;struct gmap *parent;unsigned long orig_asce;
@@ -102,6 +109,17 @@ static inline int gmap_is_shadow(struct gmap *gmap)return !!gmap->parent; } +/** +* gmap_pmd_is_split - Returns if a huge gmap pmd has been split. +* @pmdp: pointer to the pmd +* +* Returns true if the passed huge gmap pmd has been split. +*/ +static inline bool gmap_pmd_is_split (pmd_t *pmdp) +{ +return !!(pmd_val (*pmdp) & _SEGMENT_ENTRY_GMAP_SPLIT); +} +struct gmap *gmap_create (struct mm_struct *mm,unsigned long limit);void gmap_remove (struct gmap *gmap);struct gmap *gmap_get (struct gmap *gmap);
arch/s390/include/asm/pgtable.h +3 -0
@@ -1147,6 +1147,9 @@ int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *sptep, pte_t *tptep, pte_t pte);void ptep_unshadow_pte (struct mm_struct *mm,unsigned long saddr,pte_t *ptep); +unsigned long ptep_get_and_clear_notification_bits (pte_t *ptep); +void ptep_remove_protection_split (struct mm_struct *mm,pte_t *ptep, +unsigned long gaddr);bool ptep_test_and_clear_uc (struct mm_struct *mm,unsigned long address,pte_t *ptep); int set_guest_storage_key(struct mm_struct *mm, unsigned long addr,
arch/s390/mm/gmap.c +197 -50
439 lines not on this page · raw message
arch/s390/mm/pgtable.c +33 -0
@@ -739,6 +739,39 @@ void ptep_zap_key(struct mm_struct *mm, unsigned long addr, pte_t *ptep)preempt_enable (); } +unsigned long ptep_get_and_clear_notification_bits (pte_t *ptep) +{ +pgste_t pgste; +unsigned long bits; + + pgste =pgste_get_lock (ptep); + bits =pgste_val (pgste) & (PGSTE_IN_BIT |PGSTE_VSIE_BIT ); +pgste_val (pgste) ^= bits; +pgste_set_unlock (ptep, pgste); + +return bits; +} +EXPORT_SYMBOL_GPL (ptep_get_and_clear_notification_bits); + +void ptep_remove_protection_split (struct mm_struct *mm,pte_t *ptep, +unsigned long gaddr) +{ +pte_t pte; +pgste_t pgste; + + pgste =pgste_get_lock (ptep); +pgste_val (pgste) |=PGSTE_UC_BIT ; + pte = *ptep; +pte_val (pte) &= ~_PAGE_PROTECT; + + pgste =pgste_pte_notify (mm, gaddr, ptep, pgste); +ptep_ipte_global (mm, gaddr, ptep,0 ); + + *ptep = pte; +pgste_set_unlock (ptep, pgste); +} +EXPORT_SYMBOL_GPL (ptep_remove_protection_split); +/* * Test and reset if a guest page is dirty */
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 5/14 s390/mm: Split huge pages when migrating
In reply to Janosch FrankRight now we mark the huge page that is being written to as dirty although only a single byte may have changed. This means we have to migrate 1MB although only a very limited amount of memory in that range might be dirty.
To speed up migration this patch splits up write protected huge pages into normal pages. The protection for the normal pages is only removed for the page that caused the fault.
--- arch/s390/mm/gmap.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-)
arch/s390/mm/gmap.c +28 -6
@@ -532,6 +532,9 @@ void gmap_unlink(struct mm_struct *mm, unsigned long *table,static void gmap_pmdp_xchg (struct gmap *gmap,pmd_t *old,pmd_t new,unsigned long gaddr); +static void gmap_pmd_split (struct gmap *gmap,unsigned long gaddr, +pmd_t *pmdp,struct page *page); +/** * gmap_link - set up shadow page tables to connect a host to a guest address * @gmap: pointer to guest mapping meta data structure
@@ -547,12 +550,12 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) {struct mm_struct *mm;unsigned long *table; +struct page *page =NULL ;spinlock_t *ptl;pgd_t *pgd;p4d_t *p4d;pud_t *pud;pmd_t *pmd; -u64 unprot;pte_t *ptep;int rc;
@@ -600,6 +603,7 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)/* Are we allowed to use huge pages? */ if (pmd_large (*pmd) && !gmap->mm->context.allow_gmap_hpage_1m)return -EFAULT ; +retry_split:/* Link gmap segment table entry location to page table. */ rc =radix_tree_preload (GFP_KERNEL_ACCOUNT );if (rc)
@@ -627,10 +631,25 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)spin_unlock (&gmap->guest_table_lock); }else if (*table & _SEGMENT_ENTRY_PROTECT && !(pmd_val (*pmd) & _SEGMENT_ENTRY_PROTECT)) { - unprot = (u64)*table; - unprot &= ~_SEGMENT_ENTRY_PROTECT; - unprot |= _SEGMENT_ENTRY_GMAP_UC; -gmap_pmdp_xchg (gmap, (pmd_t *)table,__pmd (unprot), gaddr); +if (page) { +gmap_pmd_split (gmap, gaddr, (pmd_t *)table, page); + page =NULL ; + }else { +spin_unlock (ptl); + ptl =NULL ; +radix_tree_preload_end (); + page =page_table_alloc_pgste (mm); +if (!page) + rc = -ENOMEM ; +else + goto retry_split; + } +/* +* The split moves over the protection, so we still +* need to unprotect. +*/ + ptep =pte_offset_map ((pmd_t *)table, vmaddr); +ptep_remove_protection_split (mm, ptep, vmaddr); }else if (gmap_pmd_is_split ((pmd_t *)table)) {/* * Split pmds are somewhere in-between a normal and a
@@ -642,7 +661,10 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)if (pte_val (*ptep) & _PAGE_PROTECT)ptep_remove_protection_split (mm, ptep, vmaddr); } -spin_unlock (ptl); +if (page) +page_table_free_pgste (page); +if (ptl) +spin_unlock (ptl);radix_tree_preload_end ();return rc; }
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 6/14 s390/mm: Provide vmaddr to pmd notification
In reply to Janosch FrankIt will be needed for shadow tables.
--- arch/s390/mm/gmap.c | 52 +++++++++++++++++++++++---------------------- 1 file changed, 27 insertions(+), 25 deletions(-)
arch/s390/mm/gmap.c +27 -25
@@ -530,10 +530,10 @@ void gmap_unlink(struct mm_struct *mm, unsigned long *table, }static void gmap_pmdp_xchg (struct gmap *gmap,pmd_t *old,pmd_t new, -unsigned long gaddr); +unsigned long gaddr,unsigned long vmaddr);static void gmap_pmd_split (struct gmap *gmap,unsigned long gaddr, -pmd_t *pmdp,struct page *page); +unsigned long vmaddr,pmd_t *pmdp,struct page *page);/** * gmap_link - set up shadow page tables to connect a host to a guest address
@@ -632,7 +632,8 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) }else if (*table & _SEGMENT_ENTRY_PROTECT && !(pmd_val (*pmd) & _SEGMENT_ENTRY_PROTECT)) {if (page) { -gmap_pmd_split (gmap, gaddr, (pmd_t *)table, page); +gmap_pmd_split (gmap, gaddr, vmaddr, + (pmd_t *)table, page); page =NULL ; }else {spin_unlock (ptl);
@@ -952,19 +953,15 @@ static void gmap_pte_op_end(spinlock_t *ptl)* Returns a pointer to the pmd for a guest address, or NULL */ static inline pmd_t *gmap_pmd_op_walk (struct gmap *gmap,unsigned long gaddr, -spinlock_t **ptl) +unsigned long vmaddr,spinlock_t **ptl) {pmd_t *pmdp, *hpmdp; -unsigned long vmaddr;BUG_ON (gmap_is_shadow (gmap)); *ptl =NULL ;if (gmap->mm->context.allow_gmap_hpage_1m) { - vmaddr =__gmap_translate (gmap, gaddr); -if (IS_ERR_VALUE (vmaddr)) -return NULL ; hpmdp =pmd_alloc_map (gmap->mm, vmaddr);if (!hpmdp)return NULL ;
@@ -1047,7 +1044,7 @@ static inline void gmap_pmd_split_free(struct gmap *gmap, pmd_t *pmdp)* aren't tracked anywhere else. */ static void gmap_pmd_split (struct gmap *gmap,unsigned long gaddr, -pmd_t *pmdp,struct page *page) +unsigned long vmaddr,pmd_t *pmdp,struct page *page) {unsigned long *ptable = (unsigned long *)page_to_phys (page);pmd_t new;
@@ -1069,7 +1066,7 @@ static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr,spin_lock (&gmap->split_list_lock);list_add (&page->lru, &gmap->split_list);spin_unlock (&gmap->split_list_lock); -gmap_pmdp_xchg (gmap, pmdp, new, gaddr); +gmap_pmdp_xchg (gmap, pmdp, new, gaddr, vmaddr); }/*
@@ -1087,7 +1084,8 @@ static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr,* guest_table_lock held. */ static int gmap_protect_pmd (struct gmap *gmap,unsigned long gaddr, -pmd_t *pmdp,int prot,unsigned long bits) +unsigned long vmaddr,pmd_t *pmdp,int prot, +unsigned long bits) {int pmd_i =pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID;int pmd_p =pmd_val (*pmdp) & _SEGMENT_ENTRY_PROTECT;
@@ -1099,13 +1097,13 @@ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr,if (prot ==PROT_NONE && !pmd_i) {pmd_val (new) |= _SEGMENT_ENTRY_INVALID; -gmap_pmdp_xchg (gmap, pmdp, new, gaddr); +gmap_pmdp_xchg (gmap, pmdp, new, gaddr, vmaddr); }if (prot ==PROT_READ && !pmd_p) {pmd_val (new) &= ~_SEGMENT_ENTRY_INVALID;pmd_val (new) |= _SEGMENT_ENTRY_PROTECT; -gmap_pmdp_xchg (gmap, pmdp, new, gaddr); +gmap_pmdp_xchg (gmap, pmdp, new, gaddr, vmaddr); }if (bits &GMAP_NOTIFY_MPROT )
@@ -1168,10 +1166,14 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr,int rc;BUG_ON (gmap_is_shadow (gmap)); +while (len) { rc = -EAGAIN ; - - pmdp =gmap_pmd_op_walk (gmap, gaddr, &ptl_pmd); + vmaddr =__gmap_translate (gmap, gaddr); +if (IS_ERR_VALUE (vmaddr)) +return vmaddr; + vmaddr |= gaddr & ~PMD_MASK ; + pmdp =gmap_pmd_op_walk (gmap, gaddr, vmaddr, &ptl_pmd);if (pmdp && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)) {if (!pmd_large (*pmdp)) { ptep = gmap_pte_from_pmd(gmap, pmdp, gaddr,
@@ -1196,7 +1198,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr,return -ENOMEM ;continue ; }else { -gmap_pmd_split (gmap, gaddr, +gmap_pmd_split (gmap, gaddr, vmaddr, pmdp, page); page =NULL ;gmap_pmd_op_end (ptl_pmd);
@@ -1214,9 +1216,6 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr,return rc;/* -EAGAIN, fixup of userspace mm and gmap */ - vmaddr =__gmap_translate (gmap, gaddr); -if (IS_ERR_VALUE (vmaddr)) -return vmaddr; rc =gmap_fixup (gmap, gaddr, vmaddr, prot);if (rc)return rc;
@@ -2441,6 +2440,7 @@ static inline void pmdp_notify_split(struct gmap *gmap, pmd_t *pmdp,static void pmdp_notify_gmap (struct gmap *gmap,pmd_t *pmdp,unsigned long gaddr,unsigned long vmaddr) { +BUG_ON ((gaddr & ~HPAGE_MASK ) || (vmaddr & ~HPAGE_MASK ));if (gmap_pmd_is_split (pmdp))return pmdp_notify_split (gmap, pmdp, gaddr, vmaddr);
@@ -2461,10 +2461,11 @@ static void pmdp_notify_gmap(struct gmap *gmap, pmd_t *pmdp,* held. */ static void gmap_pmdp_xchg (struct gmap *gmap,pmd_t *pmdp,pmd_t new, -unsigned long gaddr) +unsigned long gaddr,unsigned long vmaddr) { gaddr &=HPAGE_MASK ; -pmdp_notify_gmap (gmap, pmdp, gaddr,0 ); + vmaddr &=HPAGE_MASK ; +pmdp_notify_gmap (gmap, pmdp, gaddr, vmaddr);if (pmd_large (new))pmd_val (new) &= ~GMAP_SEGMENT_NOTIFY_BITS ;if (MACHINE_HAS_TLB_GUEST )
@@ -2612,7 +2613,8 @@ EXPORT_SYMBOL_GPL(gmap_pmdp_idte_global);* held. */ static bool gmap_test_and_clear_dirty_pmd (struct gmap *gmap,pmd_t *pmdp, -unsigned long gaddr) +unsigned long gaddr, +unsigned long vmaddr) {if (pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)return false;
@@ -2624,7 +2626,7 @@ static bool gmap_test_and_clear_dirty_pmd(struct gmap *gmap, pmd_t *pmdp,/* Clear UC indication and reset protection */ pmd_val (*pmdp) &= ~_SEGMENT_ENTRY_GMAP_UC; -gmap_protect_pmd (gmap, gaddr, pmdp,PROT_READ ,0 ); +gmap_protect_pmd (gmap, gaddr, vmaddr, pmdp,PROT_READ ,0 );return true; }
@@ -2647,12 +2649,12 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4],spinlock_t *ptl_pmd =NULL ;spinlock_t *ptl_pte =NULL ; - pmdp =gmap_pmd_op_walk (gmap, gaddr, &ptl_pmd); + pmdp =gmap_pmd_op_walk (gmap, gaddr, vmaddr, &ptl_pmd);if (!pmdp)return ;if (pmd_large (*pmdp)) { -if (gmap_test_and_clear_dirty_pmd (gmap, pmdp, gaddr)) +if (gmap_test_and_clear_dirty_pmd (gmap, pmdp, gaddr, vmaddr))bitmap_fill (bitmap, _PAGE_ENTRIES); }else {for (i =0 ; i < _PAGE_ENTRIES; i++, vmaddr +=PAGE_SIZE ) {
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 7/14 s390/mm: factor out idte global flush into gmap_idte_global
In reply to Janosch FrankIntroduce a function to do a idte global flush on a gmap pmd and remove some code duplication.
--- arch/s390/mm/gmap.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
arch/s390/mm/gmap.c +14 -14
@@ -1009,6 +1009,18 @@ static pte_t *gmap_pte_from_pmd(struct gmap *gmap, pmd_t *pmdp,return pte_offset_map (pmdp, addr); } +static inline void gmap_idte_global (unsigned long asce,pmd_t *pmdp, +unsigned long gaddr) +{ +if (MACHINE_HAS_TLB_GUEST ) +__pmdp_idte (gaddr, pmdp,IDTE_GUEST_ASCE , asce, +IDTE_GLOBAL ); +else if (MACHINE_HAS_IDTE ) +__pmdp_idte (gaddr, pmdp,0 ,0 ,IDTE_GLOBAL ); +else +__pmdp_csp (pmdp); +} +/** * gmap_pmd_split_free - Free a split pmd's page table * @pmdp The split pmd that we free of its page table
@@ -2468,13 +2480,7 @@ static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *pmdp, pmd_t new,pmdp_notify_gmap (gmap, pmdp, gaddr, vmaddr);if (pmd_large (new))pmd_val (new) &= ~GMAP_SEGMENT_NOTIFY_BITS ; -if (MACHINE_HAS_TLB_GUEST ) -__pmdp_idte (gaddr, pmdp,IDTE_GUEST_ASCE , gmap->asce, -IDTE_GLOBAL ); -else if (MACHINE_HAS_IDTE ) -__pmdp_idte (gaddr, pmdp,0 ,0 ,IDTE_GLOBAL ); -else -__pmdp_csp (pmdp); +gmap_idte_global (gmap->asce, pmdp, gaddr); *pmdp = new; }
@@ -2587,13 +2593,7 @@ void gmap_pmdp_idte_global(struct mm_struct *mm, unsigned long vmaddr)pmdp_notify_gmap (gmap, pmdp, gaddr, vmaddr);if (pmd_large (*pmdp))WARN_ON (*entry &GMAP_SEGMENT_NOTIFY_BITS ); -if (MACHINE_HAS_TLB_GUEST ) -__pmdp_idte (gaddr, pmdp,IDTE_GUEST_ASCE , - gmap->asce,IDTE_GLOBAL ); -else if (MACHINE_HAS_IDTE ) -__pmdp_idte (gaddr, pmdp,0 ,0 ,IDTE_GLOBAL ); -else -__pmdp_csp (pmdp); +gmap_idte_global (gmap->asce, pmdp, gaddr);gmap_pmd_split_free (gmap, pmdp); *entry = _SEGMENT_ENTRY_EMPTY; }
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 8/14 s390/mm: Make gmap_read_table EDAT1 compatible
In reply to Janosch FrankFor the upcoming support of VSIE guests on huge page backed hosts, we need to be able to read from large segments.
--- arch/s390/mm/gmap.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-)
arch/s390/mm/gmap.c +26 -17
@@ -1282,35 +1282,44 @@ EXPORT_SYMBOL_GPL(gmap_mprotect_notify);int gmap_read_table (struct gmap *gmap,unsigned long gaddr,unsigned long *val) {unsigned long address, vmaddr; -spinlock_t *ptl; +spinlock_t *ptl_pmd =NULL , *ptl_pte =NULL ; +pmd_t *pmdp;pte_t *ptep, pte;int rc; -if (gmap_is_shadow (gmap)) -return -EINVAL ; +BUG_ON (gmap_is_shadow (gmap));while (1 ) { rc = -EAGAIN ; - ptep =gmap_pte_op_walk (gmap, gaddr, &ptl); -if (ptep) { - pte = *ptep; -if (pte_present (pte) && (pte_val (pte) & _PAGE_READ)) { - address =pte_val (pte) &PAGE_MASK ; - address += gaddr & ~PAGE_MASK ; + vmaddr =__gmap_translate (gmap, gaddr); +if (IS_ERR_VALUE (vmaddr)) +return vmaddr; + pmdp =gmap_pmd_op_walk (gmap, gaddr, vmaddr, &ptl_pmd); +if (pmdp && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)) { +if (!pmd_large (*pmdp)) { + ptep =gmap_pte_from_pmd (gmap, pmdp, vmaddr, &ptl_pte); +if (ptep) { + pte = *ptep; +if (pte_present (pte) && (pte_val (pte) & _PAGE_READ)) { + address =pte_val (pte) &PAGE_MASK ; + address += gaddr & ~PAGE_MASK ; + *val = *(unsigned long *) address; +pte_val (*ptep) |= _PAGE_YOUNG; +/* Do *NOT* clear the _PAGE_INVALID bit! */ + rc =0 ; + } + } +gmap_pte_op_end (ptl_pte); + }else { + address =pmd_val (*pmdp) &HPAGE_MASK ; + address += gaddr & ~HPAGE_MASK ; *val = *(unsigned long *) address; -pte_val (*ptep) |= _PAGE_YOUNG; -/* Do *NOT* clear the _PAGE_INVALID bit! */ rc =0 ; } -gmap_pte_op_end (ptl); +gmap_pmd_op_end (ptl_pmd); }if (!rc)break ; - vmaddr =__gmap_translate (gmap, gaddr); -if (IS_ERR_VALUE (vmaddr)) { - rc = vmaddr; -break ; - } rc =gmap_fixup (gmap, gaddr, vmaddr,PROT_READ );if (rc)break ;
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 9/14 s390/mm: Make gmap_protect_rmap EDAT1 compatible
In reply to Janosch FrankFor the upcoming large page shadowing support, let's add the possibility to split a huge page and protect it with gmap_protect_rmap() for shadowing purposes.
--- arch/s390/mm/gmap.c | 93 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 20 deletions(-)
arch/s390/mm/gmap.c +73 -20
@@ -1142,7 +1142,8 @@ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr,* Expected to be called with sg->mm->mmap_lock in read */ static int gmap_protect_pte (struct gmap *gmap,unsigned long gaddr, -pte_t *ptep,int prot,unsigned long bits) +unsigned long vmaddr,pte_t *ptep, +int prot,unsigned long bits) {int rc;unsigned long pbits =0 ;
@@ -1191,7 +1192,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, ptep =gmap_pte_from_pmd (gmap, pmdp, gaddr, &ptl_pte);if (ptep) - rc =gmap_protect_pte (gmap, gaddr, + rc =gmap_protect_pte (gmap, gaddr, vmaddr, ptep, prot, bits);else rc = -ENOMEM ;
@@ -1354,6 +1355,21 @@ static inline void gmap_insert_rmap(struct gmap *sg, unsigned long vmaddr, } } +static int gmap_protect_rmap_pte (struct gmap *sg,struct gmap_rmap *rmap, +unsigned long paddr,unsigned long vmaddr, +pte_t *ptep,int prot) +{ +int rc =0 ; + +spin_lock (&sg->guest_table_lock); + rc =gmap_protect_pte (sg->parent, paddr, vmaddr, ptep, + prot,GMAP_NOTIFY_SHADOW ); +if (!rc) +gmap_insert_rmap (sg, vmaddr, rmap); +spin_unlock (&sg->guest_table_lock); +return rc; +} +/** * gmap_protect_rmap - restrict access rights to memory (RO) and create an rmap * @sg: pointer to the shadow guest address space structure
@@ -1370,16 +1386,15 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr,struct gmap *parent;struct gmap_rmap *rmap;unsigned long vmaddr; -spinlock_t *ptl; +pmd_t *pmdp;pte_t *ptep; +spinlock_t *ptl_pmd =NULL , *ptl_pte =NULL ; +struct page *page =NULL ;int rc;BUG_ON (!gmap_is_shadow (sg)); parent = sg->parent;while (len) { - vmaddr =__gmap_translate (parent, paddr); -if (IS_ERR_VALUE (vmaddr)) -return vmaddr; rmap =kzalloc (sizeof (*rmap),GFP_KERNEL_ACCOUNT );if (!rmap)return -ENOMEM ;
@@ -1390,26 +1405,64 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr,return rc; } rc = -EAGAIN ; - ptep =gmap_pte_op_walk (parent, paddr, &ptl); -if (ptep) { -spin_lock (&sg->guest_table_lock); - rc =ptep_force_prot (parent->mm, paddr, ptep,PROT_READ , -PGSTE_VSIE_BIT ); -if (!rc) -gmap_insert_rmap (sg, vmaddr, rmap); -spin_unlock (&sg->guest_table_lock); -gmap_pte_op_end (ptl); + vmaddr =__gmap_translate (parent, paddr); +if (IS_ERR_VALUE (vmaddr)) +return vmaddr; + vmaddr |= paddr & ~PMD_MASK ; + pmdp =gmap_pmd_op_walk (parent, paddr, vmaddr, &ptl_pmd); +if (pmdp && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)) { +if (!pmd_large (*pmdp)) { + ptl_pte =NULL ; + ptep =gmap_pte_from_pmd (parent, pmdp, paddr, + &ptl_pte); +if (ptep) + rc =gmap_protect_rmap_pte (sg, rmap, paddr, + vmaddr, ptep, +PROT_READ ); +else + rc = -ENOMEM ; +gmap_pte_op_end (ptl_pte); +gmap_pmd_op_end (ptl_pmd); +if (!rc) { + paddr +=PAGE_SIZE ; + len -=PAGE_SIZE ; +radix_tree_preload_end (); +continue ; + } + }else { +if (!page) { +/* Drop locks for allocation. */ +gmap_pmd_op_end (ptl_pmd); + ptl_pmd =NULL ; +radix_tree_preload_end (); +kfree (rmap); + page =page_table_alloc_pgste (parent->mm); +if (!page) +return -ENOMEM ; +continue ; + }else { +gmap_pmd_split (parent, paddr, vmaddr, + pmdp, page); +gmap_pmd_op_end (ptl_pmd); +radix_tree_preload_end (); +kfree (rmap); + page =NULL ; +continue ; + } + + } + } +if (page) { +page_table_free_pgste (page); + page =NULL ; }radix_tree_preload_end (); -if (rc) { -kfree (rmap); +kfree (rmap); +if (rc == -EAGAIN ) { rc =gmap_fixup (parent, paddr, vmaddr,PROT_READ );if (rc)return rc; -continue ; } - paddr +=PAGE_SIZE ; - len -=PAGE_SIZE ; }return 0 ; }
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 10/14 s390/mm: Add simple ptep shadow function
In reply to Janosch FrankLet's factor out setting the shadow pte, so we can reuse that function for later huge to 4k shadows where we don't have a spte or spgste.
--- arch/s390/include/asm/pgtable.h | 1 + arch/s390/mm/pgtable.c | 24 ++++++++++++++++-------- 2 files changed, 17 insertions(+), 8 deletions(-)
arch/s390/include/asm/pgtable.h +1 -0
@@ -1145,6 +1145,7 @@ void ptep_zap_unused(struct mm_struct *mm, unsigned long addr,void ptep_zap_key (struct mm_struct *mm,unsigned long addr,pte_t *ptep);int ptep_shadow_pte (struct mm_struct *mm,unsigned long saddr,pte_t *sptep,pte_t *tptep,pte_t pte); +void ptep_shadow_set (pte_t spte,pte_t *tptep,pte_t pte);void ptep_unshadow_pte (struct mm_struct *mm,unsigned long saddr,pte_t *ptep);unsigned long ptep_get_and_clear_notification_bits (pte_t *ptep);
arch/s390/mm/pgtable.c +16 -8
@@ -645,11 +645,24 @@ int ptep_force_prot(struct mm_struct *mm, unsigned long addr,return 0 ; } +void ptep_shadow_set (pte_t spte,pte_t *tptep,pte_t pte) +{ +pte_t tpte; +pgste_t tpgste; + + tpgste =pgste_get_lock (tptep); +pte_val (tpte) = (pte_val (spte) &PAGE_MASK ) | + (pte_val (pte) & _PAGE_PROTECT); +/* don't touch the storage key - it belongs to parent pgste */ + tpgste =pgste_set_pte (tptep, tpgste, tpte); +pgste_set_unlock (tptep, tpgste); +} +int ptep_shadow_pte (struct mm_struct *mm,unsigned long saddr,pte_t *sptep,pte_t *tptep,pte_t pte) { -pgste_t spgste, tpgste; -pte_t spte, tpte; +pgste_t spgste; +pte_t spte;int rc = -EAGAIN ;if (!(pte_val (*tptep) & _PAGE_INVALID))
@@ -660,12 +673,7 @@ int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr, !((pte_val(spte) & _PAGE_PROTECT) && !(pte_val(pte) & _PAGE_PROTECT))) {pgste_val (spgste) |=PGSTE_VSIE_BIT ; - tpgste =pgste_get_lock (tptep); -pte_val (tpte) = (pte_val (spte) &PAGE_MASK ) | - (pte_val (pte) & _PAGE_PROTECT); -/* don't touch the storage key - it belongs to parent pgste */ - tpgste =pgste_set_pte (tptep, tpgste, tpte); -pgste_set_unlock (tptep, tpgste); +ptep_shadow_set (spte, tptep, pte); rc =1 ; }pgste_set_unlock (sptep, spgste);
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 11/14 s390/mm: Add gmap shadowing for large pmds
In reply to Janosch FrankUp to now we could only shadow large pmds when the parent's mapping was done with normal sized pmds. This is done by introducing fake page tables and effectively running the level 3 guest with a standard memory backing instead of the large one.
With this patch we add shadowing when the host is large page backed. This allows us to run normal and large backed VMs inside a large backed host.
--- arch/s390/include/asm/gmap.h | 9 +- arch/s390/kvm/gaccess.c | 52 +++++- arch/s390/mm/gmap.c | 328 ++++++++++++++++++++++++++++------- 3 files changed, 317 insertions(+), 72 deletions(-)
arch/s390/include/asm/gmap.h +6 -3
@@ -19,11 +19,12 @@/* Status bits only for huge segment entries */ #define _SEGMENT_ENTRY_GMAP_IN 0x8000/* invalidation notify bit */ #define _SEGMENT_ENTRY_GMAP_UC 0x4000/* dirty (migration) */ +#define _SEGMENT_ENTRY_GMAP_VSIE 0x2000/* vsie bit */ /* Status bits in the gmap segment entry. */ #define _SEGMENT_ENTRY_GMAP_SPLIT 0x0001/* split huge pmd */ #define GMAP_SEGMENT_STATUS_BITS (_SEGMENT_ENTRY_GMAP_UC | _SEGMENT_ENTRY_GMAP_SPLIT) -#define GMAP_SEGMENT_NOTIFY_BITS _SEGMENT_ENTRY_GMAP_IN +#define GMAP_SEGMENT_NOTIFY_BITS (_SEGMENT_ENTRY_GMAP_IN | _SEGMENT_ENTRY_GMAP_VSIE)/** * struct gmap_struct - guest address space
@@ -152,9 +153,11 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt, int fake);int gmap_shadow_pgt (struct gmap *sg,unsigned long saddr,unsigned long pgt,int fake); -int gmap_shadow_pgt_lookup (struct gmap *sg,unsigned long saddr, -unsigned long *pgt,int *dat_protection,int *fake); +int gmap_shadow_sgt_lookup (struct gmap *sg,unsigned long saddr, +unsigned long *pgt,int *dat_protection, +int *fake,int *lvl);int gmap_shadow_page (struct gmap *sg,unsigned long saddr,pte_t pte); +int gmap_shadow_segment (struct gmap *sg,unsigned long saddr,pmd_t pmd);void gmap_register_pte_notifier (struct gmap_notifier *);void gmap_unregister_pte_notifier (struct gmap_notifier *);
arch/s390/kvm/gaccess.c +44 -8
91 lines not on this page · raw message
arch/s390/mm/gmap.c +267 -61
416 lines not on this page · raw message
Signature · 1 line
2.27.0
kernel test robot
build botRe: [PATCH 11/14] s390/mm: Add gmap shadowing for large pmds
I love your patch! Perhaps something to improve:
Hi Janosch,
I love your patch! Perhaps something to improve:
[auto build test WARNING on kvms390/next]
[also build test WARNING on s390/features kvm/linux-next v5.11-rc4 next-20210119] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Janosch-Frank/KVM-s390-Add-huge-page-VSIE-support/20210113-174708 base: https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git next config: s390-randconfig-r003-20210120 (attached as .config) compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project 22b68440e1647e16b5ee24b924986207173c02d1)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install s390 cross compiling tool for clang build
# apt-get install binutils-s390x-linux-gnu
# https://github.com/0day-ci/linux/commit/b3a8ab3492fef4fa7bf29a81261fd1f593cd50d0
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Janosch-Frank/KVM-s390-Add-huge-page-VSIE-support/20210113-174708 git checkout b3a8ab3492fef4fa7bf29a81261fd1f593cd50d0
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=s390
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:119:21: note: expanded from macro '__swab32'
___constant_swab32(x) : \
^
include/uapi/linux/swab.h:20:12: note: expanded from macro '___constant_swab32'
(((__u32)(x) & (__u32)0x0000ff00UL) << 8) | \
^
In file included from arch/s390/mm/gmap.c:13:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:26:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:80:
include/asm-generic/io.h:490:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:34:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:119:21: note: expanded from macro '__swab32'
___constant_swab32(x) : \
^
include/uapi/linux/swab.h:21:12: note: expanded from macro '___constant_swab32'
(((__u32)(x) & (__u32)0x00ff0000UL) >> 8) | \
^
In file included from arch/s390/mm/gmap.c:13:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:26:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:80:
include/asm-generic/io.h:490:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:34:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:119:21: note: expanded from macro '__swab32'
___constant_swab32(x) : \
^
include/uapi/linux/swab.h:22:12: note: expanded from macro '___constant_swab32'
(((__u32)(x) & (__u32)0xff000000UL) >> 24)))
^
In file included from arch/s390/mm/gmap.c:13:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:26:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:80:
include/asm-generic/io.h:490:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:34:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:120:12: note: expanded from macro '__swab32'
__fswab32(x))
^
In file included from arch/s390/mm/gmap.c:13:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:26:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:80:
include/asm-generic/io.h:501:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writeb(value, PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:511:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:521:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:609:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:617:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:625:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:634:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:643:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:652:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
>> arch/s390/mm/gmap.c:2183:6: warning: no previous prototype for function 'gmap_shadow_pgt_lookup' [-Wmissing-prototypes]
void gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long *sge,
^ arch/s390/mm/gmap.c:2183:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long *sge,
^
static
21 warnings generated.
vim +/gmap_shadow_pgt_lookup +2183 arch/s390/mm/gmap.c
2169
2170 /**
2171 * gmap_shadow_lookup_pgtable - find a shadow page table
2172 * @sg: pointer to the shadow guest address space structure
2173 * @saddr: the address in the shadow guest address space
2174 * @pgt: parent gmap address of the page table to get shadowed
2175 * @dat_protection: if the pgtable is marked as protected by dat
2176 * @fake: pgt references contiguous guest memory block, not a pgtable
2177 *
2178 * Returns 0 if the shadow page table was found and -EAGAIN if the page
2179 * table was not found.
2180 *
2181 * Called with sg->mm->mmap_lock in read.
2182 */
> 2183 void gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long *sge,
2184 unsigned long saddr, unsigned long *pgt,
2185 int *dat_protection, int *fake)
2186 {
2187 struct page *page;
2188
2189 /* Shadow page tables are full pages (pte+pgste) */
2190 page = pfn_to_page(*sge >> PAGE_SHIFT);
2191 *pgt = page->index & ~GMAP_SHADOW_FAKE_TABLE;
2192 *dat_protection = !!(*sge & _SEGMENT_ENTRY_PROTECT);
2193 *fake = !!(page->index & GMAP_SHADOW_FAKE_TABLE);
2194 }
2195 EXPORT_SYMBOL_GPL(gmap_shadow_pgt_lookup);
2196
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Janosch Frank
maintainerPATCH 12/14 s390/mm: Add gmap lock classes
In reply to Janosch FrankA shadow gmap and its parent are locked right after each other when doing VSIE management. Lockdep can't differentiate between the two classes without some help.
--- arch/s390/include/asm/gmap.h | 6 ++++++ arch/s390/mm/gmap.c | 40 +++++++++++++++++++----------------- 2 files changed, 27 insertions(+), 19 deletions(-)
arch/s390/include/asm/gmap.h +6 -0
@@ -26,6 +26,12 @@#define GMAP_SEGMENT_STATUS_BITS (_SEGMENT_ENTRY_GMAP_UC | _SEGMENT_ENTRY_GMAP_SPLIT)#define GMAP_SEGMENT_NOTIFY_BITS (_SEGMENT_ENTRY_GMAP_IN | _SEGMENT_ENTRY_GMAP_VSIE) + +enum gmap_lock_class { +GMAP_LOCK_PARENT , +GMAP_LOCK_SHADOW +}; +/** * struct gmap_struct - guest address space * @list: list head for the mm->context gmap list
arch/s390/mm/gmap.c +21 -19
@@ -1339,7 +1339,7 @@ static int gmap_protect_rmap_pte(struct gmap *sg, struct gmap_rmap *rmap, {int rc =0 ; -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); rc =gmap_protect_pte (sg->parent, paddr, vmaddr, ptep, prot,GMAP_NOTIFY_SHADOW );if (!rc)
@@ -1874,7 +1874,7 @@ struct gmap *gmap_shadow(struct gmap *parent, unsigned long asce,/* only allow one real-space gmap shadow */ list_for_each_entry (sg, &parent->children, list) {if (sg->orig_asce & _ASCE_REAL_SPACE) { -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW );gmap_unshadow (sg);spin_unlock (&sg->guest_table_lock);list_del (&sg->list);
@@ -1946,7 +1946,7 @@ int gmap_shadow_r2t(struct gmap *sg, unsigned long saddr, unsigned long r2t, page->index |=GMAP_SHADOW_FAKE_TABLE ; s_r2t = (unsigned long *)page_to_phys (page);/* Install shadow region second table */ -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); table =gmap_table_walk (sg, saddr,4 );/* get region-1 pointer */ if (!table) { rc = -EAGAIN ;/* Race with unshadow */
@@ -1979,7 +1979,7 @@ int gmap_shadow_r2t(struct gmap *sg, unsigned long saddr, unsigned long r2t, offset = ((r2t & _REGION_ENTRY_OFFSET) >>6 ) *PAGE_SIZE ; len = ((r2t & _REGION_ENTRY_LENGTH) +1 ) *PAGE_SIZE - offset; rc =gmap_protect_rmap (sg, raddr, origin + offset, len); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); if (!rc) { table = gmap_table_walk(sg, saddr, 4); if (!table || (*table & _REGION_ENTRY_ORIGIN) !=
@@ -2030,7 +2030,7 @@ int gmap_shadow_r3t(struct gmap *sg, unsigned long saddr, unsigned long r3t, page->index |=GMAP_SHADOW_FAKE_TABLE ; s_r3t = (unsigned long *)page_to_phys (page);/* Install shadow region second table */ -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); table =gmap_table_walk (sg, saddr,3 );/* get region-2 pointer */ if (!table) { rc = -EAGAIN ;/* Race with unshadow */
@@ -2063,7 +2063,7 @@ int gmap_shadow_r3t(struct gmap *sg, unsigned long saddr, unsigned long r3t, offset = ((r3t & _REGION_ENTRY_OFFSET) >>6 ) *PAGE_SIZE ; len = ((r3t & _REGION_ENTRY_LENGTH) +1 ) *PAGE_SIZE - offset; rc =gmap_protect_rmap (sg, raddr, origin + offset, len); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); if (!rc) { table = gmap_table_walk(sg, saddr, 3); if (!table || (*table & _REGION_ENTRY_ORIGIN) !=
@@ -2114,7 +2114,7 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt, page->index |=GMAP_SHADOW_FAKE_TABLE ; s_sgt = (unsigned long *)page_to_phys (page);/* Install shadow region second table */ -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); table =gmap_table_walk (sg, saddr,2 );/* get region-3 pointer */ if (!table) { rc = -EAGAIN ;/* Race with unshadow */
@@ -2147,7 +2147,7 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt, offset = ((sgt & _REGION_ENTRY_OFFSET) >>6 ) *PAGE_SIZE ; len = ((sgt & _REGION_ENTRY_LENGTH) +1 ) *PAGE_SIZE - offset; rc =gmap_protect_rmap (sg, raddr, origin + offset, len); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); if (!rc) { table = gmap_table_walk(sg, saddr, 2); if (!table || (*table & _REGION_ENTRY_ORIGIN) !=
@@ -2203,7 +2203,7 @@ int gmap_shadow_sgt_lookup(struct gmap *sg, unsigned long saddr,int rc = -EAGAIN ;BUG_ON (!gmap_is_shadow (sg)); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW );if (sg->asce & _ASCE_TYPE_MASK) {/* >2 GB guest */ r3e = (unsigned long *)gmap_table_walk (sg, saddr,2 );
@@ -2270,7 +2270,7 @@ int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt, page->index |=GMAP_SHADOW_FAKE_TABLE ; s_pgt = (unsigned long *)page_to_phys (page);/* Install shadow page table */ -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); table =gmap_table_walk (sg, saddr,1 );/* get segment pointer */ if (!table) { rc = -EAGAIN ;/* Race with unshadow */
@@ -2298,7 +2298,7 @@ int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt, raddr = (saddr & _SEGMENT_MASK) | _SHADOW_RMAP_SEGMENT; origin = pgt & _SEGMENT_ENTRY_ORIGIN &PAGE_MASK ; rc =gmap_protect_rmap (sg, raddr, origin,PAGE_SIZE ); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW ); if (!rc) { table = gmap_table_walk(sg, saddr, 1); if (!table || (*table & _SEGMENT_ENTRY_ORIGIN) !=
@@ -2359,7 +2359,7 @@ int gmap_shadow_segment(struct gmap *sg, unsigned long saddr, pmd_t pmd) rc = -EISDIR ;break ; } -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW );/* Get shadow segment table pointer */ tpmdp = (pmd_t *)gmap_table_walk (sg, saddr,1 );if (!tpmdp) {
@@ -2464,7 +2464,8 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte)if (pmd_large (*spmdp)) {pte_t spte;if (!(pmd_val (*spmdp) & _SEGMENT_ENTRY_PROTECT)) { -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock, +GMAP_LOCK_SHADOW ); spte =__pte ((pmd_val (*spmdp) & _SEGMENT_ENTRY_ORIGIN_LARGE) + (pte_index (paddr) <<12 ));
@@ -2477,7 +2478,8 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) } }else { sptep =gmap_pte_from_pmd (parent, spmdp, paddr, &ptl_pte); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock, +GMAP_LOCK_SHADOW );if (sptep) { rc =ptep_shadow_pte (sg->mm, saddr, sptep, tptep, pte);if (rc >0 ) {
@@ -2535,7 +2537,7 @@ static void gmap_shadow_notify_pmd(struct gmap *sg, unsigned long vmaddr,BUG_ON (!gmap_is_shadow (sg)); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW );if (sg->removed) {spin_unlock (&sg->guest_table_lock);return ;
@@ -2586,7 +2588,7 @@ static void gmap_shadow_notify(struct gmap *sg, unsigned long vmaddr,BUG_ON (!gmap_is_shadow (sg)); -spin_lock (&sg->guest_table_lock); +spin_lock_nested (&sg->guest_table_lock,GMAP_LOCK_SHADOW );if (sg->removed) {spin_unlock (&sg->guest_table_lock);return ;
@@ -2761,7 +2763,7 @@ static void gmap_pmdp_clear(struct mm_struct *mm, unsigned long vmaddr,rcu_read_lock ();list_for_each_entry_rcu (gmap, &mm->context.gmap_list, list) { -spin_lock (&gmap->guest_table_lock); +spin_lock_nested (&gmap->guest_table_lock,GMAP_LOCK_PARENT ); pmdp = (pmd_t *)radix_tree_delete (&gmap->host_to_guest, vmaddr >>PMD_SHIFT );if (pmdp) {
@@ -2816,7 +2818,7 @@ void gmap_pmdp_idte_local(struct mm_struct *mm, unsigned long vmaddr)rcu_read_lock ();list_for_each_entry_rcu (gmap, &mm->context.gmap_list, list) { -spin_lock (&gmap->guest_table_lock); +spin_lock_nested (&gmap->guest_table_lock,GMAP_LOCK_PARENT ); entry =radix_tree_delete (&gmap->host_to_guest, vmaddr >>PMD_SHIFT );if (entry) {
@@ -2852,7 +2854,7 @@ void gmap_pmdp_idte_global(struct mm_struct *mm, unsigned long vmaddr)rcu_read_lock ();list_for_each_entry_rcu (gmap, &mm->context.gmap_list, list) { -spin_lock (&gmap->guest_table_lock); +spin_lock_nested (&gmap->guest_table_lock,GMAP_LOCK_PARENT ); entry =radix_tree_delete (&gmap->host_to_guest, vmaddr >>PMD_SHIFT );if (entry) {
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 13/14 s390/mm: Pull pmd invalid check in gmap_pmd_op_walk
In reply to Janosch FrankNot yet sure if I'll keep this.
The walk should only walk and not check I, but then it looks way nicer.
--- arch/s390/mm/gmap.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)
arch/s390/mm/gmap.c +7 -10
@@ -956,7 +956,8 @@ static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr, } pmdp = (pmd_t *)gmap_table_walk (gmap, gaddr,1 ); -if (!pmdp ||pmd_none (*pmdp)) { +if (!pmdp ||pmd_none (*pmdp) || +pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID) {if (*ptl)spin_unlock (*ptl); pmdp =NULL ;
@@ -1165,7 +1166,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr,return vmaddr; vmaddr |= gaddr & ~PMD_MASK ; pmdp =gmap_pmd_op_walk (gmap, gaddr, vmaddr, &ptl_pmd); -if (pmdp && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)) { +if (pmdp) {if (!pmd_large (*pmdp)) { ptep =gmap_pte_from_pmd (gmap, pmdp, gaddr, &ptl_pte);
@@ -1274,7 +1275,7 @@ int gmap_read_table(struct gmap *gmap, unsigned long gaddr, unsigned long *val)if (IS_ERR_VALUE (vmaddr))return vmaddr; pmdp =gmap_pmd_op_walk (gmap, gaddr, vmaddr, &ptl_pmd); -if (pmdp && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)) { +if (pmdp) {if (!pmd_large (*pmdp)) { ptep =gmap_pte_from_pmd (gmap, pmdp, vmaddr, &ptl_pte);if (ptep) {
@@ -1388,7 +1389,7 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr,return vmaddr; vmaddr |= paddr & ~PMD_MASK ; pmdp =gmap_pmd_op_walk (parent, paddr, vmaddr, &ptl_pmd); -if (pmdp && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_INVALID)) { +if (pmdp) {if (!pmd_large (*pmdp)) { ptl_pte =NULL ; ptep = gmap_pte_from_pmd(parent, pmdp, paddr,
@@ -2378,8 +2379,7 @@ int gmap_shadow_segment(struct gmap *sg, unsigned long saddr, pmd_t pmd)break ; } spmd = *spmdp; -if (!(pmd_val (spmd) & _SEGMENT_ENTRY_INVALID) && - !((pmd_val (spmd) & _SEGMENT_ENTRY_PROTECT) && +if (!((pmd_val (spmd) & _SEGMENT_ENTRY_PROTECT) && !(pmd_val (pmd) & _SEGMENT_ENTRY_PROTECT))) {pmd_val (*spmdp) |= _SEGMENT_ENTRY_GMAP_VSIE;
@@ -2452,7 +2452,7 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte)break ; rc = -EAGAIN ; spmdp =gmap_pmd_op_walk (parent, paddr, vmaddr, &ptl_pmd); -if (spmdp && !(pmd_val (*spmdp) & _SEGMENT_ENTRY_INVALID)) { +if (spmdp) {/* Get page table pointer */ tptep = (pte_t *)gmap_table_walk (sg, saddr,0 );if (!tptep) {
@@ -2886,9 +2886,6 @@ static bool gmap_test_and_clear_dirty_pmd(struct gmap *gmap, pmd_t *pmdp, unsigned long gaddr, unsigned long vmaddr) { - if (pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID) - return false; -/* Already protected memory, which did not change is clean */ if (pmd_val (*pmdp) & _SEGMENT_ENTRY_PROTECT && !(pmd_val (*pmdp) & _SEGMENT_ENTRY_GMAP_UC))
Signature · 1 line
2.27.0
Janosch Frank
maintainerPATCH 14/14 KVM: s390: Allow the VSIE to be used with huge pages
In reply to Janosch FrankNow that we have VSIE support for VMs with huge memory backing, let's make both features usable at the same time.
--- Documentation/virt/kvm/api.rst | 9 ++++----- arch/s390/kvm/kvm-s390.c | 14 ++------------ arch/s390/mm/gmap.c | 1 - 3 files changed, 6 insertions(+), 18 deletions(-)
Documentation/virt/kvm/api.rst +4 -5
@@ -5876,15 +5876,14 @@ Do not enable KVM_FEATURE_PV_UNHALT if you disable HLT exits. :Architectures: s390 :Parameters: none -:Returns: 0 on success, -EINVAL if hpage module parameter was not set - or cmma is enabled, or the VM has the KVM_VM_S390_UCONTROL - flag set +:Returns: 0 on success, -EINVAL if cmma is enabled, or the VM has the + KVM_VM_S390_UCONTROL flag set With this capability the KVM support for memory backing with 1m pages through hugetlbfs can be enabled for a VM. After the capability is enabled, cmma can't be enabled anymore and pfmfi and the storage key -interpretation are disabled. If cmma has already been enabled or the -hpage module parameter is not set to 1, -EINVAL is returned. +interpretation are disabled. If cmma has already been enabled, -EINVAL +is returned. While it is generally possible to create a huge page backed VM without this capability, the VM will not be able to run.
arch/s390/kvm/kvm-s390.c +2 -12
@@ -175,11 +175,6 @@ static int nested;module_param (nested, int,S_IRUGO );MODULE_PARM_DESC (nested,"Nested virtualization support" ); -/* allow 1m huge page guest backing, if !nested */ -static int hpage; -module_param (hpage, int,0444 ); -MODULE_PARM_DESC (hpage,"1m huge page backing support" ); -/* maximum percentage of steal time for polling. >100 is treated like 100 */ static u8 halt_poll_max_steal =10 ;module_param (halt_poll_max_steal, byte,0644 );
@@ -551,7 +546,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)break ;case KVM_CAP_S390_HPAGE_1M : r =0 ; -if (hpage && !kvm_is_ucontrol (kvm)) +if (!kvm_is_ucontrol (kvm)) r =1 ;break ;case KVM_CAP_S390_MEM_OP :
@@ -761,7 +756,7 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)mutex_lock (&kvm->lock);if (kvm->created_vcpus) r = -EBUSY ; -else if (!hpage || kvm->arch.use_cmma ||kvm_is_ucontrol (kvm)) +else if (kvm->arch.use_cmma ||kvm_is_ucontrol (kvm)) r = -EINVAL ;else { r =0 ;
@@ -5042,11 +5037,6 @@ static int __init kvm_s390_init(void)return -ENODEV ; } -if (nested && hpage) { -pr_info ("A KVM host that supports nesting cannot back its KVM guests with huge pages\n" ); -return -EINVAL ; - } -for (i =0 ; i <16 ; i++) kvm_s390_fac_base[i] |= S390_lowcore.stfle_fac_list[i] &nonhyp_mask (i);
arch/s390/mm/gmap.c +0 -1
@@ -1844,7 +1844,6 @@ struct gmap *gmap_shadow(struct gmap *parent, unsigned long asce,unsigned long limit;int rc; -BUG_ON (parent->mm->context.allow_gmap_hpage_1m);BUG_ON (gmap_is_shadow (parent));spin_lock (&parent->shadow_lock); sg =gmap_find_shadow (parent, asce, edat_level);
Signature · 1 line
2.27.0