Sync unmappings in vmalloc/ioremap areas
Subsystems (4)
v3 cover Sync unmappings in vmalloc/ioremap areas
Hi,
here is a small patch-set to sync unmappings in the vmalloc/ioremap areas between page-tables in the system.
This is only needed x86-32 with !SHARED_KERNEL_PMD, which is the case on a PAE kernel with PTI enabled.
On affected systems the missing sync causes old mappings to persist in some page-tables, causing data corruption and other undefined behavior.
Please review.
Thanks,
Joerg
Changes v2 -> v3:
- Moved the vmalloc_sync_all() call to the lazy vmap purge function as requested by Andy Lutomirski
- Made sure that the code in vmalloc_sync_all() really iterates over all pgds (pointed out by Thomas Gleixner)
- Added a couple of comments
Changes v1 -> v2:
- Added correct Fixes-tags to all patches
Joerg Roedel (3):
x86/mm: Check for pfn instead of page in vmalloc_sync_one() x86/mm: Sync also unmappings in vmalloc_sync_all() mm/vmalloc: Sync unmappings in vunmap_page_range()
arch/x86/mm/fault.c | 15 ++++++--------- mm/vmalloc.c | 9 +++++++++ 2 files changed, 15 insertions(+), 9 deletions(-)
Signature · 1 line
2.17.1
PATCH v3 1/3 x86/mm: Check for pfn instead of page in vmalloc_sync_one()
From: Joerg Roedel <[email protected]>
Do not require a struct page for the mapped memory location because it might not exist. This can happen when an ioremapped region is mapped with 2MB pages.
--- arch/x86/mm/fault.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/fault.c +1 -1
@@ -183,7 +183,7 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address)if (!pmd_present (*pmd))set_pmd (pmd, *pmd_k);else -BUG_ON (pmd_page (*pmd) !=pmd_page (*pmd_k)); +BUG_ON (pmd_pfn (*pmd) !=pmd_pfn (*pmd_k));return pmd_k; }
Signature · 1 line
2.17.1
tip-bot for Joerg Roedel
merge bot[tip:x86/urgent] x86/mm: Check for pfn instead of page in vmalloc_sync_one()
x86/mm: Check for pfn instead of page in vmalloc_sync_one()
Commit-ID: 51b75b5b563a2637f9d8dc5bd02a31b2ff9e5ea0 Gitweb: https://git.kernel.org/tip/51b75b5b563a2637f9d8dc5bd02a31b2ff9e5ea0 Author: Joerg Roedel <[email protected]> AuthorDate: Fri, 19 Jul 2019 20:46:50 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 22 Jul 2019 10:18:30 +0200
x86/mm: Check for pfn instead of page in vmalloc_sync_one()
Do not require a struct page for the mapped memory location because it might not exist. This can happen when an ioremapped region is mapped with 2MB pages.
--- arch/x86/mm/fault.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/fault.c +1 -1
@@ -183,7 +183,7 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address)if (!pmd_present (*pmd))set_pmd (pmd, *pmd_k);else -BUG_ON (pmd_page (*pmd) !=pmd_page (*pmd_k)); +BUG_ON (pmd_pfn (*pmd) !=pmd_pfn (*pmd_k));return pmd_k; }
PATCH v3 2/3 x86/mm: Sync also unmappings in vmalloc_sync_all()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
With huge-page ioremap areas the unmappings also need to be synced between all page-tables. Otherwise it can cause data corruption when a region is unmapped and later re-used.
Make the vmalloc_sync_one() function ready to sync unmappings and make sure vmalloc_sync_all() iterates over all page-tables even when an unmapped PMD is found.
--- arch/x86/mm/fault.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)
arch/x86/mm/fault.c +5 -8
@@ -177,11 +177,12 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address) pmd =pmd_offset (pud, address); pmd_k =pmd_offset (pud_k, address); -if (!pmd_present (*pmd_k)) -return NULL ; -if (!pmd_present (*pmd)) +if (pmd_present (*pmd) !=pmd_present (*pmd_k))set_pmd (pmd, *pmd_k); + +if (!pmd_present (*pmd_k)) +return NULL ;else BUG_ON (pmd_pfn (*pmd) !=pmd_pfn (*pmd_k));
@@ -203,17 +204,13 @@ void vmalloc_sync_all(void)spin_lock (&pgd_lock);list_for_each_entry (page, &pgd_list, lru) {spinlock_t *pgt_lock; -pmd_t *ret;/* the pgt_lock only for Xen */ pgt_lock = &pgd_page_get_mm (page)->page_table_lock;spin_lock (pgt_lock); - ret =vmalloc_sync_one (page_address (page), address); +vmalloc_sync_one (page_address (page), address);spin_unlock (pgt_lock); - -if (!ret) -break ; }spin_unlock (&pgd_lock); }
Signature · 1 line
2.17.1
tip-bot for Joerg Roedel
merge bot[tip:x86/urgent] x86/mm: Sync also unmappings in vmalloc_sync_all()
x86/mm: Sync also unmappings in vmalloc_sync_all()
Commit-ID: 8e998fc24de47c55b47a887f6c95ab91acd4a720 Gitweb: https://git.kernel.org/tip/8e998fc24de47c55b47a887f6c95ab91acd4a720 Author: Joerg Roedel <[email protected]> AuthorDate: Fri, 19 Jul 2019 20:46:51 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 22 Jul 2019 10:18:30 +0200
x86/mm: Sync also unmappings in vmalloc_sync_all()
With huge-page ioremap areas the unmappings also need to be synced between all page-tables. Otherwise it can cause data corruption when a region is unmapped and later re-used.
Make the vmalloc_sync_one() function ready to sync unmappings and make sure vmalloc_sync_all() iterates over all page-tables even when an unmapped PMD is found.
--- arch/x86/mm/fault.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)
arch/x86/mm/fault.c +5 -8
@@ -177,11 +177,12 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address) pmd =pmd_offset (pud, address); pmd_k =pmd_offset (pud_k, address); -if (!pmd_present (*pmd_k)) -return NULL ; -if (!pmd_present (*pmd)) +if (pmd_present (*pmd) !=pmd_present (*pmd_k))set_pmd (pmd, *pmd_k); + +if (!pmd_present (*pmd_k)) +return NULL ;else BUG_ON (pmd_pfn (*pmd) !=pmd_pfn (*pmd_k));
@@ -203,17 +204,13 @@ void vmalloc_sync_all(void)spin_lock (&pgd_lock);list_for_each_entry (page, &pgd_list, lru) {spinlock_t *pgt_lock; -pmd_t *ret;/* the pgt_lock only for Xen */ pgt_lock = &pgd_page_get_mm (page)->page_table_lock;spin_lock (pgt_lock); - ret =vmalloc_sync_one (page_address (page), address); +vmalloc_sync_one (page_address (page), address);spin_unlock (pgt_lock); - -if (!ret) -break ; }spin_unlock (&pgd_lock); }
PATCH v3 3/3 mm/vmalloc: Sync unmappings in vunmap_page_range()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
On x86-32 with PTI enabled, parts of the kernel page-tables are not shared between processes. This can cause mappings in the vmalloc/ioremap area to persist in some page-tables after the region is unmapped and released.
When the region is re-used the processes with the old mappings do not fault in the new mappings but still access the old ones.
This causes undefined behavior, in reality often data corruption, kernel oopses and panics and even spontaneous reboots.
Fix this problem by activly syncing unmaps in the vmalloc/ioremap area to all page-tables in the system before the regions can be re-used.
References: https://bugzilla.suse.com/show_bug.cgi?id=1118689
--- mm/vmalloc.c | 9 +++++++++ 1 file changed, 9 insertions(+)
mm/vmalloc.c +9 -0
@@ -1258,6 +1258,12 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)if (unlikely (valist ==NULL ))return false; +/* +* First make sure the mappings are removed from all page-tables +* before they are freed. +*/ +vmalloc_sync_all (); +/* * TODO: to calculate a flush range without looping. * The list can be up to lazy_max_pages() elements.
@@ -3038,6 +3044,9 @@ EXPORT_SYMBOL(remap_vmalloc_range);/* * Implement a stub for vmalloc_sync_all() if the architecture chose not to * have one. +* +* The purpose of this function is to make sure the vmalloc area +* mappings are identical in all page-tables in the system. */ void __weak vmalloc_sync_all (void ) {
Signature · 1 line
2.17.1
Re: [PATCH 3/3] mm/vmalloc: Sync unmappings in vunmap_page_range()
Srewed up the subject :(, it needs to be
"mm/vmalloc: Sync unmappings in __purge_vmap_area_lazy()"
of course.
Quoted 56 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
On x86-32 with PTI enabled, parts of the kernel page-tables are not shared between processes. This can cause mappings in the vmalloc/ioremap area to persist in some page-tables after the region is unmapped and released.
When the region is re-used the processes with the old mappings do not fault in the new mappings but still access the old ones.
This causes undefined behavior, in reality often data corruption, kernel oopses and panics and even spontaneous reboots.
Fix this problem by activly syncing unmaps in the vmalloc/ioremap area to all page-tables in the system before the regions can be re-used.
References: https://bugzilla.suse.com/show_bug.cgi?id=1118689
Reviewed-by Dave Hansen <[email protected]>Fixes 5d72b4fba40ef ('x86, mm: support huge I/O mapping capability I/F')Signed-off-by Joerg Roedel <[email protected]>--- mm/vmalloc.c | 9 +++++++++ 1 file changed, 9 insertions(+)mm/vmalloc.c +9
-0@@ -1258,6 +1258,12 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)if (unlikely (valist ==NULL ))return false; +/* +* First make sure the mappings are removed from all page-tables +* before they are freed. +*/ +vmalloc_sync_all (); +/* * TODO: to calculate a flush range without looping. * The list can be up to lazy_max_pages() elements. @@ -3038,6 +3044,9 @@ EXPORT_SYMBOL(remap_vmalloc_range);/* * Implement a stub for vmalloc_sync_all() if the architecture chose not to * have one. +* +* The purpose of this function is to make sure the vmalloc area +* mappings are identical in all page-tables in the system. */ void __weak vmalloc_sync_all (void ) {Signature · 1 line
2.17.1
Re: [PATCH 3/3] mm/vmalloc: Sync unmappings in vunmap_page_range()
Quoted 1 line · Joerg Roedel
Srewed up the subject :(, it needs to be
Un-Srewed it :)
^^^^^^
Re: [PATCH 3/3] mm/vmalloc: Sync unmappings in vunmap_page_range()
Quoted 5 lines · Thomas Gleixner
Quoted 1 line · Joerg Roedel
Srewed up the subject :(, it needs to be
Un-Srewed it :)
Thanks a lot :)
tip-bot for Joerg Roedel
merge bot[tip:x86/urgent] mm/vmalloc: Sync unmappings in __purge_vmap_area_lazy()
In reply to Joerg Roedelmm/vmalloc: Sync unmappings in __purge_vmap_area_lazy()
Commit-ID: 3f8fd02b1bf1d7ba964485a56f2f4b53ae88c167 Gitweb: https://git.kernel.org/tip/3f8fd02b1bf1d7ba964485a56f2f4b53ae88c167 Author: Joerg Roedel <[email protected]> AuthorDate: Fri, 19 Jul 2019 20:46:52 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 22 Jul 2019 10:18:30 +0200
mm/vmalloc: Sync unmappings in __purge_vmap_area_lazy()
On x86-32 with PTI enabled, parts of the kernel page-tables are not shared between processes. This can cause mappings in the vmalloc/ioremap area to persist in some page-tables after the region is unmapped and released.
When the region is re-used the processes with the old mappings do not fault in the new mappings but still access the old ones.
This causes undefined behavior, in reality often data corruption, kernel oopses and panics and even spontaneous reboots.
Fix this problem by activly syncing unmaps in the vmalloc/ioremap area to all page-tables in the system before the regions can be re-used.
References: https://bugzilla.suse.com/show_bug.cgi?id=1118689
--- mm/vmalloc.c | 9 +++++++++ 1 file changed, 9 insertions(+)
mm/vmalloc.c +9 -0
@@ -1258,6 +1258,12 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)if (unlikely (valist ==NULL ))return false; +/* +* First make sure the mappings are removed from all page-tables +* before they are freed. +*/ +vmalloc_sync_all (); +/* * TODO: to calculate a flush range without looping. * The list can be up to lazy_max_pages() elements.
@@ -3038,6 +3044,9 @@ EXPORT_SYMBOL(remap_vmalloc_range);/* * Implement a stub for vmalloc_sync_all() if the architecture chose not to * have one. +* +* The purpose of this function is to make sure the vmalloc area +* mappings are identical in all page-tables in the system. */ void __weak vmalloc_sync_all (void ) {