Fix kvm misconceives NVDIMM pages as reserved mmio
Subsystems (5)
cover Fix kvm misconceives NVDIMM pages as reserved mmio
For device specific memory space, when we move these area of pfn to memory zone, we will set the page reserved flag at that time, some of these reserved for device mmio, and some of these are not, such as NVDIMM pmem.
Now, we map these dev_dax or fs_dax pages to kvm for DIMM/NVDIMM backend, since these pages are reserved. the check of kvm_is_reserved_pfn() misconceives those pages as MMIO. Therefor, we introduce 2 page map types, MEMORY_DEVICE_FS_DAX/MEMORY_DEVICE_DEV_DAX, to indentify these pages are from NVDIMM pmem. and let kvm treat these as normal pages.
Without this patch, Many operations will be missed due to this mistreatment to pmem pages. For example, a page may not have chance to be unpinned for KVM guest(in kvm_release_pfn_clean); not able to be marked as dirty/accessed(in kvm_set_pfn_dirty/accessed) etc.
Zhang Yi (3):
kvm: remove redundant reserved page check
mm: introduce memory type MEMORY_DEVICE_DEV_DAX
kvm: add a function to check if page is from NVDIMM pmem.
drivers/dax/pmem.c | 1 + include/linux/memremap.h | 1 + virt/kvm/kvm_main.c | 25 +++++++++++++++++-------- 3 files changed, 19 insertions(+), 8 deletions(-)
Signature · 6 lines
2.7.4 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 1/3 kvm: remove redundant reserved page check
PageReserved() is already checked inside kvm_is_reserved_pfn(), remove it from kvm_set_pfn_dirty().
--- virt/kvm/kvm_main.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
virt/kvm/kvm_main.c +2 -6
@@ -1673,12 +1673,8 @@ EXPORT_SYMBOL_GPL(kvm_release_pfn_dirty);void kvm_set_pfn_dirty (kvm_pfn_t pfn) { -if (!kvm_is_reserved_pfn (pfn)) { -struct page *page =pfn_to_page (pfn); - -if (!PageReserved (page)) -SetPageDirty (page); - } +if (!kvm_is_reserved_pfn (pfn)) +SetPageDirty (pfn_to_page (pfn)); }EXPORT_SYMBOL_GPL (kvm_set_pfn_dirty);
Signature · 6 lines
2.7.4 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 2/3 mm: introduce memory type MEMORY_DEVICE_DEV_DAX
In reply to Zhang YiCurrently, NVDIMM pages will be marked 'PageReserved'. However, unlike other reserved PFNs, pages on NVDIMM shall still behave like normal ones in many cases, i.e. when used as backend memory of KVM guest. This patch introduces a new memory type, MEMORY_DEVICE_DEV_DAX. Together with the existing type MEMORY_DEVICE_FS_DAX, we can differentiate the pages on NVDIMM with the normal reserved pages.
--- drivers/dax/pmem.c | 1 + include/linux/memremap.h | 1 + 2 files changed, 2 insertions(+)
drivers/dax/pmem.c +1 -0
@@ -111,6 +111,7 @@ static int dax_pmem_probe(struct device *dev)return rc; dax_pmem->pgmap.ref = &dax_pmem->ref; + dax_pmem->pgmap.type =MEMORY_DEVICE_DEV_DAX ; addr =devm_memremap_pages (dev, &dax_pmem->pgmap);if (IS_ERR (addr))return PTR_ERR (addr);
include/linux/memremap.h +1 -0
@@ -58,6 +58,7 @@ enum memory_type {MEMORY_DEVICE_PRIVATE =1 ,MEMORY_DEVICE_PUBLIC ,MEMORY_DEVICE_FS_DAX , + MEMORY_DEVICE_DEV_DAX, };/*
Signature · 6 lines
2.7.4 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [PATCH 2/3] mm: introduce memory type MEMORY_DEVICE_DEV_DAX
Quoted 31 lines · Zhang Yi
Currently, NVDIMM pages will be marked 'PageReserved'. However, unlike other reserved PFNs, pages on NVDIMM shall still behave like normal ones in many cases, i.e. when used as backend memory of KVM guest. This patch introduces a new memory type,
MEMORY_DEVICE_DEV_DAX. Together with the existing typeMEMORY_DEVICE_FS_DAX, we can differentiate the pages on NVDIMM with the normal reserved pages.--- drivers/dax/pmem.c | 1 + include/linux/memremap.h | 1 + 2 files changed, 2 insertions(+)drivers/dax/pmem.c +1
-0@@ -111,6 +111,7 @@ static int dax_pmem_probe(struct device *dev)return rc; dax_pmem->pgmap.ref = &dax_pmem->ref; + dax_pmem->pgmap.type =MEMORY_DEVICE_DEV_DAX ; addr =devm_memremap_pages (dev, &dax_pmem->pgmap);if (IS_ERR (addr))return PTR_ERR (addr);include/linux/memremap.h +0
-0
include/linux/memremap.h enum memory_type { · line 61
Quoted 4 lines
include/linux/memremap.h +1
-0@@ -58,6 +58,7 @@ enum memory_type {MEMORY_DEVICE_PRIVATE =1 ,MEMORY_DEVICE_PUBLIC ,MEMORY_DEVICE_FS_DAX , + MEMORY_DEVICE_DEV_DAX,
Please add documentation for this new type to the comment block about this definition.
Re: [PATCH 2/3] mm: introduce memory type MEMORY_DEVICE_DEV_DAX
Quoted 22 lines · Dan Williams
Quoted 21 lines · Zhang Yi
Currently, NVDIMM pages will be marked 'PageReserved'. However, unlike other reserved PFNs, pages on NVDIMM shall still behave like normal ones in many cases, i.e. when used as backend memory of KVM guest. This patch introduces a new memory type,
MEMORY_DEVICE_DEV_DAX. Together with the existing typeMEMORY_DEVICE_FS_DAX, we can differentiate the pages on NVDIMM with the normal reserved pages.Signed-off-by Zhang Yi <[email protected]>Signed-off-by Zhang Yu <[email protected]>--- drivers/dax/pmem.c | 1 + include/linux/memremap.h | 1 + 2 files changed, 2 insertions(+)drivers/dax/pmem.c +0
-0@@ -111,6 +111,7 @@ static int dax_pmem_probe(struct device *dev)return rc;
Quoted 16 lines
Quoted 10 lines
drivers/dax/pmem.c +1
-0dax_pmem->pgmap.ref = &dax_pmem->ref; + dax_pmem->pgmap.type =MEMORY_DEVICE_DEV_DAX ; addr =devm_memremap_pages (dev, &dax_pmem->pgmap);if (IS_ERR (addr))return PTR_ERR (addr);include/linux/memremap.h +0
-0include/linux/memremap.h enum memory_type { · line 61
Quoted 4 lines
include/linux/memremap.h +1
-0@@ -58,6 +58,7 @@ enum memory_type {MEMORY_DEVICE_PRIVATE =1 ,MEMORY_DEVICE_PUBLIC ,MEMORY_DEVICE_FS_DAX , + MEMORY_DEVICE_DEV_DAX,Please add documentation for this new type to the comment block about this definition.
Thanks for your comments Dan, Will add it in next version, Regards
Yi.
_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 3/3 kvm: add a function to check if page is from NVDIMM pmem.
In reply to Zhang YiFor device specific memory space, when we move these area of pfn to memory zone, we will set the page reserved flag at that time, some of these reserved for device mmio, and some of these are not, such as NVDIMM pmem.
Now, we map these dev_dax or fs_dax pages to kvm for DIMM/NVDIMM backend, since these pages are reserved. the check of kvm_is_reserved_pfn() misconceives those pages as MMIO. Therefor, we introduce 2 page map types, MEMORY_DEVICE_FS_DAX/MEMORY_DEVICE_DEV_DAX, to indentify these pages are from NVDIMM pmem. and let kvm treat these as normal pages.
Without this patch, Many operations will be missed due to this mistreatment to pmem pages. For example, a page may not have chance to be unpinned for KVM guest(in kvm_release_pfn_clean); not able to be marked as dirty/accessed(in kvm_set_pfn_dirty/accessed) etc.
--- virt/kvm/kvm_main.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
virt/kvm/kvm_main.c +15 -2
@@ -140,10 +140,23 @@ __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, { } +static bool kvm_is_nd_pfn (kvm_pfn_t pfn) +{ +struct page *page =pfn_to_page (pfn); + +return is_zone_device_page (page) && + ((page->pgmap->type ==MEMORY_DEVICE_FS_DAX ) || + (page->pgmap->type ==MEMORY_DEVICE_DEV_DAX )); +} +bool kvm_is_reserved_pfn (kvm_pfn_t pfn) { -if (pfn_valid (pfn)) -return PageReserved (pfn_to_page (pfn)); +struct page *page; + +if (pfn_valid (pfn)) { + page =pfn_to_page (pfn); +return kvm_is_nd_pfn (pfn) ? false :PageReserved (page); + }return true; }
Signature · 6 lines
2.7.4 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [PATCH 3/3] kvm: add a function to check if page is from NVDIMM pmem.
[ adding Jerome ]
Quoted 28 lines · Zhang Yi
For device specific memory space, when we move these area of pfn to memory zone, we will set the page reserved flag at that time, some of these reserved for device mmio, and some of these are not, such as NVDIMM pmem.
Now, we map these
dev_daxorfs_daxpages to kvm for DIMM/NVDIMM backend, since these pages are reserved. the check ofkvm_is_reserved_pfn()misconceives those pages as MMIO. Therefor, we introduce 2 page map types, MEMORY_DEVICE_FS_DAX/MEMORY_DEVICE_DEV_DAX, to indentify these pages are from NVDIMM pmem. and let kvm treat these as normal pages.Without this patch, Many operations will be missed due to this mistreatment to pmem pages. For example, a page may not have chance to be unpinned for KVM
guest(in kvm_release_pfn_clean); not able to be marked as dirty/accessed(in kvm_set_pfn_dirty/accessed) etc.--- virt/kvm/kvm_main.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +0
-0
virt/kvm/kvm_main.c __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, · line 149
Quoted 10 lines
virt/kvm/kvm_main.c +7
-0@@ -140,10 +140,23 @@ __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, { } +static bool kvm_is_nd_pfn (kvm_pfn_t pfn) +{ +struct page *page =pfn_to_page (pfn); + +return is_zone_device_page (page) && + ((page->pgmap->type ==MEMORY_DEVICE_FS_DAX ) || + (page->pgmap->type ==MEMORY_DEVICE_DEV_DAX ));
Jerome, might there be any use case to pass MEMORY_DEVICE_PUBLIC memory to a guest vm?
Quoted 18 lines
virt/kvm/kvm_main.c +8
-3+} +bool kvm_is_reserved_pfn (kvm_pfn_t pfn) { -if (pfn_valid (pfn)) -return PageReserved (pfn_to_page (pfn)); +struct page *page; + +if (pfn_valid (pfn)) { + page =pfn_to_page (pfn); +return kvm_is_nd_pfn (pfn) ? false :PageReserved (page); + }return true; } --2.7.4
Paolo Bonzini
maintainerRe: [PATCH 3/3] kvm: add a function to check if page is from NVDIMM pmem.
Quoted 6 lines · Dan Williams
virt/kvm/kvm_main.c __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, · line 149
Quoted 3 lines
virt/kvm/kvm_main.c +3
-0+return is_zone_device_page (page) && + ((page->pgmap->type ==MEMORY_DEVICE_FS_DAX ) || + (page->pgmap->type ==MEMORY_DEVICE_DEV_DAX ));Jerome, might there be any use case to pass
MEMORY_DEVICE_PUBLICmemory to a guest vm?
An even better reason to place this in mm.h. :) There should be an function to tell you if a reserved page has accessed/dirty bits etc., that's all that KVM needs to know.
Paolo
Paolo Bonzini
maintainerRe: [PATCH 3/3] kvm: add a function to check if page is from NVDIMM pmem.
In reply to Zhang YiQuoted 29 lines · Zhang Yi
For device specific memory space, when we move these area of pfn to memory zone, we will set the page reserved flag at that time, some of these reserved for device mmio, and some of these are not, such as NVDIMM pmem.
Now, we map these
dev_daxorfs_daxpages to kvm for DIMM/NVDIMM backend, since these pages are reserved. the check ofkvm_is_reserved_pfn()misconceives those pages as MMIO. Therefor, we introduce 2 page map types, MEMORY_DEVICE_FS_DAX/MEMORY_DEVICE_DEV_DAX, to indentify these pages are from NVDIMM pmem. and let kvm treat these as normal pages.Without this patch, Many operations will be missed due to this mistreatment to pmem pages. For example, a page may not have chance to be unpinned for KVM
guest(in kvm_release_pfn_clean); not able to be marked as dirty/accessed(in kvm_set_pfn_dirty/accessed) etc.--- virt/kvm/kvm_main.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +0
-0@@ -140,10 +140,23 @@ __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, {
virt/kvm/kvm_main.c __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, · line 150
Quoted 10 lines
virt/kvm/kvm_main.c +8
-0} +static bool kvm_is_nd_pfn (kvm_pfn_t pfn) +{ +struct page *page =pfn_to_page (pfn); + +return is_zone_device_page (page) && + ((page->pgmap->type ==MEMORY_DEVICE_FS_DAX ) || + (page->pgmap->type ==MEMORY_DEVICE_DEV_DAX )); +}
If the mm people agree, I'd prefer something that takes a struct page * and is exported by include/linux/mm.h. Then KVM can just do something like
struct page *page; if (!pfn_valid(pfn)) return true; page = pfn_to_page(pfn); return PageReserved(page) && !is_dax_page(page);
Thanks,
Paolo
Re: [PATCH 3/3] kvm: add a function to check if page is from NVDIMM pmem.
Quoted 50 lines · Paolo Bonzini
Quoted 29 lines · Zhang Yi
For device specific memory space, when we move these area of pfn to memory zone, we will set the page reserved flag at that time, some of these reserved for device mmio, and some of these are not, such as NVDIMM pmem.
Now, we map these
dev_daxorfs_daxpages to kvm for DIMM/NVDIMM backend, since these pages are reserved. the check ofkvm_is_reserved_pfn()misconceives those pages as MMIO. Therefor, we introduce 2 page map types, MEMORY_DEVICE_FS_DAX/MEMORY_DEVICE_DEV_DAX, to indentify these pages are from NVDIMM pmem. and let kvm treat these as normal pages.Without this patch, Many operations will be missed due to this mistreatment to pmem pages. For example, a page may not have chance to be unpinned for KVM
guest(in kvm_release_pfn_clean); not able to be marked as dirty/accessed(in kvm_set_pfn_dirty/accessed) etc.Signed-off-by Zhang Yi <[email protected]>Signed-off-by Zhang Yu <[email protected]>--- virt/kvm/kvm_main.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +0
-0@@ -140,10 +140,23 @@ __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, {virt/kvm/kvm_main.c __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, · line 150
Quoted 10 lines
virt/kvm/kvm_main.c +8
-0} +static bool kvm_is_nd_pfn (kvm_pfn_t pfn) +{ +struct page *page =pfn_to_page (pfn); + +return is_zone_device_page (page) && + ((page->pgmap->type ==MEMORY_DEVICE_FS_DAX ) || + (page->pgmap->type ==MEMORY_DEVICE_DEV_DAX )); +}If the mm people agree, I'd prefer something that takes a struct page * and is exported by
include/linux/mm.h. Then KVM can just do something likestruct page *page; if (!pfn_valid(pfn)) return true; page = pfn_to_page(pfn); return PageReserved(page) && !is_dax_page(page);
Quoted 3 lines
Thanks,
Paolo
Yeah, that could be much better. Thanks for your comments Paolo.
Hi Kara, Do u have Any opinions/ideas to add such definition in mm?
Regards,
Yi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [PATCH 3/3] kvm: add a function to check if page is from NVDIMM pmem.
Quoted 53 lines · Zhang,Yi
Quoted 50 lines · Paolo Bonzini
Quoted 29 lines · Zhang Yi
For device specific memory space, when we move these area of pfn to memory zone, we will set the page reserved flag at that time, some of these reserved for device mmio, and some of these are not, such as NVDIMM pmem.
Now, we map these
dev_daxorfs_daxpages to kvm for DIMM/NVDIMM backend, since these pages are reserved. the check ofkvm_is_reserved_pfn()misconceives those pages as MMIO. Therefor, we introduce 2 page map types, MEMORY_DEVICE_FS_DAX/MEMORY_DEVICE_DEV_DAX, to indentify these pages are from NVDIMM pmem. and let kvm treat these as normal pages.Without this patch, Many operations will be missed due to this mistreatment to pmem pages. For example, a page may not have chance to be unpinned for KVM
guest(in kvm_release_pfn_clean); not able to be marked as dirty/accessed(in kvm_set_pfn_dirty/accessed) etc.Signed-off-by Zhang Yi <[email protected]>Signed-off-by Zhang Yu <[email protected]>--- virt/kvm/kvm_main.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)virt/kvm/kvm_main.c +0
-0@@ -140,10 +140,23 @@ __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, {virt/kvm/kvm_main.c __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, · line 150
Quoted 10 lines
virt/kvm/kvm_main.c +8
-0} +static bool kvm_is_nd_pfn (kvm_pfn_t pfn) +{ +struct page *page =pfn_to_page (pfn); + +return is_zone_device_page (page) && + ((page->pgmap->type ==MEMORY_DEVICE_FS_DAX ) || + (page->pgmap->type ==MEMORY_DEVICE_DEV_DAX )); +}If the mm people agree, I'd prefer something that takes a struct page * and is exported by
include/linux/mm.h. Then KVM can just do something likestruct page *page; if (!pfn_valid(pfn)) return true; page = pfn_to_page(pfn); return PageReserved(page) && !is_dax_page(page);
Quoted 6 lines
Quoted 3 lines
Thanks,
Paolo
Yeah, that could be much better. Thanks for your comments Paolo.
Hi Kara, Do u have Any opinions/ideas to add such definition in mm?
What Paolo suggests sounds good to me.
Honza
Signature · 6 lines
Jan Kara <[email protected]> SUSE Labs, CR _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm