hmm: fixes and documentations v4
cover hmm: fixes and documentations v4
From: JA(C)rA'me Glisse <[email protected]>
Fixes and improvement to HMM only impact HMM user. Changes since the last posting:
Allow release callback to wait on any device driver workqueue which is processing fault without having to worry for deadlock. Some driver do call migrate_vma() from their page fault workqueue which itself will trigger calls to mmu_notifier callback. If we take and keep the HMM semaphore in write mode while calling device driver release call back then those can not wait on their page fault workqueue as the workqueue might end up deadlocking (waiting to be able to take the HMM semaphore in read mode). Avoid this by dropping HMM semaphore while calling the device driver release callback.
Other changes from previous post are solely typos fixes (in comment not in code), s/pfns/pfn for one function argument, and usual patch formatting fixes (also added review tag for review receive so far).
Below are previous cover letter (everything in them are still true):
----------------------------------------------------------------------
cover letter for v3:
Added a patch to fix zombie mm_struct (missing call to mmu notifier unregister) this was lost in translation at some point. Included all typos and comments received so far (and even more typos fixes). Added more comments. Updated individual patch version to reflect changes.
----------------------------------------------------------------------
cover letter for v2:
Removed pointless VM_BUG_ON() cced stable when appropriate and splited the last patch into _many_ smaller patches to make it easier to review. The end result is same modulo comments i received so far and the extra documentation i added while splitting thing up. Below is previous cover letter (everything in it is still true):
----------------------------------------------------------------------
cover letter for v1:
All patches only impact HMM user, there is no implication outside HMM.
First patch improve documentation to better reflect what HMM is. Second patch fix #if/#else placement in hmm.h. The third patch add a call on mm release which helps device driver who use HMM to clean up early when a process quit. Finaly last patch modify the CPU snapshot and page fault helper to simplify device driver. The nouveau patchset i posted last week already depends on all of those patches.
You can find them in a hmm-for-4.17 branch:
git://people.freedesktop.org/~glisse/linux
https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-for-4.17
JA(C)rA'me Glisse (13):
mm/hmm: fix header file if/else/endif maze v2
mm/hmm: unregister mmu_notifier when last HMM client quit v3 mm/hmm: hmm_pfns_bad() was accessing wrong struct
mm/hmm: use struct for hmm_vma_fault(), hmm_vma_get_pfns() parameters
v2
mm/hmm: remove HMM_PFN_READ flag and ignore peculiar architecture v2
mm/hmm: use uint64_t for HMM pfn instead of defining hmm_pfn_t to
ulong v2
mm/hmm: cleanup special vma handling (VM_SPECIAL)
mm/hmm: do not differentiate between empty entry or missing directory
v3
mm/hmm: rename HMM_PFN_DEVICE_UNADDRESSABLE to HMM_PFN_DEVICE_PRIVATE
mm/hmm: move hmm_pfns_clear() closer to where it is use
mm/hmm: factor out pte and pmd handling to simplify hmm_vma_walk_pmd()
v2 mm/hmm: change hmm_vma_fault() to allow write fault on page basis mm/hmm: use device driver encoding for HMM pfn v2
Ralph Campbell (2):
mm/hmm: documentation editorial update to HMM documentation mm/hmm: HMM should have a callback before MM is destroyed v3
Documentation/vm/hmm.txt | 360 +++++++++++++++--------------- MAINTAINERS | 1 + include/linux/hmm.h | 201 ++++++++++------- mm/hmm.c | 556 +++++++++++++++++++++++++++++++---------------- 4 files changed, 675 insertions(+), 443 deletions(-)
Signature · 1 line
2.14.3
PATCH 1/15 mm/hmm: documentation editorial update to HMM documentation
From: Ralph Campbell <[email protected]>
This patch updates the documentation for HMM to fix minor typos and phrasing to be a bit more readable.
Signed-off-by Ralph Campbell <[email protected]>
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Cc Stephen Bates <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Mark Hairgrove <[email protected]>
Cc John Hubbard <[email protected]>
--- Documentation/vm/hmm.txt | 360 ++++++++++++++++++++++++----------------------- MAINTAINERS | 1 + 2 files changed, 187 insertions(+), 174 deletions(-)
Documentation/vm/hmm.txt +186 -174
@@ -1,151 +1,159 @@ Heterogeneous Memory Management (HMM) -Transparently allow any component of a program to use any memory region of said -program with a device without using device specific memory allocator. This is -becoming a requirement to simplify the use of advance heterogeneous computing -where GPU, DSP or FPGA are use to perform various computations. - -This document is divided as follow, in the first section i expose the problems -related to the use of a device specific allocator. The second section i expose -the hardware limitations that are inherent to many platforms. The third section -gives an overview of HMM designs. The fourth section explains how CPU page- -table mirroring works and what is HMM purpose in this context. Fifth section -deals with how device memory is represented inside the kernel. Finaly the last -section present the new migration helper that allow to leverage the device DMA -engine. - - -1) Problems of using device specific memory allocator: -2) System bus, device memory characteristics -3) Share address space and migration +Provide infrastructure and helpers to integrate non conventional memory (device +memory like GPU on board memory) into regular kernel code path. Corner stone of +this being specialize struct page for such memory (see sections 5 to 7 of this +document). + +HMM also provide optional helpers for SVM (Share Virtual Memory) ie allowing a +device to transparently access program address coherently with the CPU meaning +that any valid pointer on the CPU is also a valid pointer for the device. This +is becoming a mandatory to simplify the use of advance heterogeneous computing +where GPU, DSP, or FPGA are used to perform various computations on behalf of +a process. + +This document is divided as follows: in the first section I expose the problems +related to using device specific memory allocators. In the second section, I +expose the hardware limitations that are inherent to many platforms. The third +section gives an overview of the HMM design. The fourth section explains how +CPU page-table mirroring works and what is HMM's purpose in this context. The +fifth section deals with how device memory is represented inside the kernel. +Finally, the last section presents a new migration helper that allows lever- +aging the device DMA engine. + + +1) Problems of using a device specific memory allocator: +2) I/O bus, device memory characteristics +3) Shared address space and migration 4) Address space mirroring implementation and API 5) Represent and manage device memory from core kernel point of view -6) Migrate to and from device memory +6) Migration to and from device memory 7) Memory cgroup (memcg) and rss accounting ------------------------------------------------------------------------------- -1) Problems of using device specific memory allocator: +1) Problems of using a device specific memory allocator: -Device with large amount of on board memory (several giga bytes) like GPU have -historically manage their memory through dedicated driver specific API. This -creates a disconnect between memory allocated and managed by device driver and -regular application memory (private anonymous, share memory or regular file -back memory). From here on i will refer to this aspect as split address space. -I use share address space to refer to the opposite situation ie one in which -any memory region can be use by device transparently. +Devices with a large amount of on board memory (several giga bytes) like GPUs +have historically managed their memory through dedicated driver specific APIs. +This creates a disconnect between memory allocated and managed by a device +driver and regular application memory (private anonymous, shared memory, or +regular file backed memory). From here on I will refer to this aspect as split +address space. I use shared address space to refer to the opposite situation: +i.e., one in which any application memory region can be used by a device +transparently. Split address space because device can only access memory allocated through the -device specific API. This imply that all memory object in a program are not -equal from device point of view which complicate large program that rely on a -wide set of libraries. +device specific API. This implies that all memory objects in a program are not +equal from the device point of view which complicates large programs that rely +on a wide set of libraries. -Concretly this means that code that wants to leverage device like GPU need to +Concretly this means that code that wants to leverage devices like GPUs need to copy object between genericly allocated memory (malloc, mmap private/share/) and memory allocated through the device driver API (this still end up with an mmap but of the device file). -For flat dataset (array, grid, image, ...) this isn't too hard to achieve but -complex data-set (list, tree, ...) are hard to get right. Duplicating a complex -data-set need to re-map all the pointer relations between each of its elements. -This is error prone and program gets harder to debug because of the duplicate -data-set. +For flat data-sets (array, grid, image, ...) this isn't too hard to achieve but +complex data-sets (list, tree, ...) are hard to get right. Duplicating a +complex data-set needs to re-map all the pointer relations between each of its +elements. This is error prone and program gets harder to debug because of the +duplicate data-set and addresses. -Split address space also means that library can not transparently use data they -are getting from core program or other library and thus each library might have -to duplicate its input data-set using specific memory allocator. Large project -suffer from this and waste resources because of the various memory copy. +Split address space also means that libraries can not transparently use data +they are getting from the core program or another library and thus each library +might have to duplicate its input data-set using the device specific memory +allocator. Large projects suffer from this and waste resources because of the +various memory copies. Duplicating each library API to accept as input or output memory allocted by each device specific allocator is not a viable option. It would lead to a -combinatorial explosions in the library entry points. +combinatorial explosion in the library entry points. -Finaly with the advance of high level language constructs (in C++ but in other -language too) it is now possible for compiler to leverage GPU or other devices -without even the programmer knowledge. Some of compiler identified patterns are -only do-able with a share address. It is as well more reasonable to use a share -address space for all the other patterns. +Finally, with the advance of high level language constructs (in C++ but in +other languages too) it is now possible for the compiler to leverage GPUs and +other devices without programmer knowledge. Some compiler identified patterns +are only do-able with a shared address space. It is also more reasonable to use +a shared address space for all other patterns. ------------------------------------------------------------------------------- -2) System bus, device memory characteristics +2) I/O bus, device memory characteristics -System bus cripple share address due to few limitations. Most system bus only +I/O buses cripple shared address due to few limitations. Most I/O buses only allow basic memory access from device to main memory, even cache coherency is -often optional. Access to device memory from CPU is even more limited, most -often than not it is not cache coherent. +often optional. Access to device memory from CPU is even more limited. More +often than not, it is not cache coherent. -If we only consider the PCIE bus than device can access main memory (often -through an IOMMU) and be cache coherent with the CPUs. However it only allows -a limited set of atomic operation from device on main memory. This is worse -in the other direction the CPUs can only access a limited range of the device +If we only consider the PCIE bus, then a device can access main memory (often +through an IOMMU) and be cache coherent with the CPUs. However, it only allows +a limited set of atomic operations from device on main memory. This is worse +in the other direction, the CPU can only access a limited range of the device memory and can not perform atomic operations on it. Thus device memory can not -be consider like regular memory from kernel point of view. +be considered the same as regular memory from the kernel point of view. Another crippling factor is the limited bandwidth (~32GBytes/s with PCIE 4.0 -and 16 lanes). This is 33 times less that fastest GPU memory (1 TBytes/s). -The final limitation is latency, access to main memory from the device has an -order of magnitude higher latency than when the device access its own memory. +and 16 lanes). This is 33 times less than the fastest GPU memory (1 TBytes/s). +The final limitation is latency. Access to main memory from the device has an +order of magnitude higher latency than when the device accesses its own memory. -Some platform are developing new system bus or additions/modifications to PCIE -to address some of those limitations (OpenCAPI, CCIX). They mainly allow two +Some platforms are developing new I/O buses or additions/modifications to PCIE +to address some of these limitations (OpenCAPI, CCIX). They mainly allow two way cache coherency between CPU and device and allow all atomic operations the -architecture supports. Saddly not all platform are following this trends and -some major architecture are left without hardware solutions to those problems. +architecture supports. Saddly, not all platforms are following this trend and +some major architectures are left without hardware solutions to these problems. -So for share address space to make sense not only we must allow device to +So for shared address space to make sense, not only must we allow device to access any memory memory but we must also permit any memory to be migrated to device memory while device is using it (blocking CPU access while it happens). ------------------------------------------------------------------------------- -3) Share address space and migration +3) Shared address space and migration HMM intends to provide two main features. First one is to share the address -space by duplication the CPU page table into the device page table so same -address point to same memory and this for any valid main memory address in +space by duplicating the CPU page table in the device page table so the same +address points to the same physical memory for any valid main memory address in the process address space. -To achieve this, HMM offer a set of helpers to populate the device page table +To achieve this, HMM offers a set of helpers to populate the device page table while keeping track of CPU page table updates. Device page table updates are -not as easy as CPU page table updates. To update the device page table you must -allow a buffer (or use a pool of pre-allocated buffer) and write GPU specifics -commands in it to perform the update (unmap, cache invalidations and flush, -...). This can not be done through common code for all device. Hence why HMM -provides helpers to factor out everything that can be while leaving the gory -details to the device driver. - -The second mechanism HMM provide is a new kind of ZONE_DEVICE memory that does -allow to allocate a struct page for each page of the device memory. Those page -are special because the CPU can not map them. They however allow to migrate -main memory to device memory using exhisting migration mechanism and everything -looks like if page was swap out to disk from CPU point of view. Using a struct -page gives the easiest and cleanest integration with existing mm mechanisms. -Again here HMM only provide helpers, first to hotplug new ZONE_DEVICE memory -for the device memory and second to perform migration. Policy decision of what -and when to migrate things is left to the device driver. - -Note that any CPU access to a device page trigger a page fault and a migration -back to main memory ie when a page backing an given address A is migrated from -a main memory page to a device page then any CPU access to address A trigger a -page fault and initiate a migration back to main memory. - - -With this two features, HMM not only allow a device to mirror a process address -space and keeps both CPU and device page table synchronize, but also allow to -leverage device memory by migrating part of data-set that is actively use by a -device. +not as easy as CPU page table updates. To update the device page table, you must +allocate a buffer (or use a pool of pre-allocated buffers) and write GPU +specific commands in it to perform the update (unmap, cache invalidations, and +flush, ...). This can not be done through common code for all devices. Hence +why HMM provides helpers to factor out everything that can be while leaving the +hardware specific details to the device driver. + +The second mechanism HMM provides, is a new kind of ZONE_DEVICE memory that +allows allocating a struct page for each page of the device memory. Those pages +are special because the CPU can not map them. However, they allow migrating +main memory to device memory using existing migration mechanisms and everything +looks like a page is swapped out to disk from the CPU point of view. Using a +struct page gives the easiest and cleanest integration with existing mm mech- +anisms. Here again, HMM only provides helpers, first to hotplug new ZONE_DEVICE +memory for the device memory and second to perform migration. Policy decisions +of what and when to migrate things is left to the device driver. + +Note that any CPU access to a device page triggers a page fault and a migration +back to main memory. For example, when a page backing a given CPU address A is +migrated from a main memory page to a device page, then any CPU access to +address A triggers a page fault and initiates a migration back to main memory. + +With these two features, HMM not only allows a device to mirror process address +space and keeping both CPU and device page table synchronized, but also lever- +ages device memory by migrating the part of the data-set that is actively being +used by the device. ------------------------------------------------------------------------------- 4) Address space mirroring implementation and API -Address space mirroring main objective is to allow to duplicate range of CPU -page table into a device page table and HMM helps keeping both synchronize. A +Address space mirroring's main objective is to allow duplication of a range of +CPU page table into a device page table; HMM helps keep both synchronized. A device driver that want to mirror a process address space must start with the registration of an hmm_mirror struct:
@@ -155,8 +163,8 @@ device driver that want to mirror a process address space must start with the struct mm_struct *mm); The locked variant is to be use when the driver is already holding the mmap_sem -of the mm in write mode. The mirror struct has a set of callback that are use -to propagate CPU page table: +of the mm in write mode. The mirror struct has a set of callbacks that are used +to propagate CPU page tables: struct hmm_mirror_ops { /* sync_cpu_device_pagetables() - synchronize page tables
@@ -181,13 +189,13 @@ of the mm in write mode. The mirror struct has a set of callback that are use unsigned long end); }; -Device driver must perform update to the range following action (turn range -read only, or fully unmap, ...). Once driver callback returns the device must -be done with the update. +The device driver must perform the update action to the range (mark range +read only, or fully unmap, ...). The device must be done with the update before +the driver callback returns. -When device driver wants to populate a range of virtual address it can use -either: +When the device driver wants to populate a range of virtual addresses, it can +use either: int hmm_vma_get_pfns(struct vm_area_struct *vma, struct hmm_range *range, unsigned long start,
@@ -201,17 +209,19 @@ When device driver wants to populate a range of virtual address it can use bool write, bool block); -First one (hmm_vma_get_pfns()) will only fetch present CPU page table entry and -will not trigger a page fault on missing or non present entry. The second one -do trigger page fault on missing or read only entry if write parameter is true. -Page fault use the generic mm page fault code path just like a CPU page fault. +The first one (hmm_vma_get_pfns()) will only fetch present CPU page table +entries and will not trigger a page fault on missing or non present entries. +The second one does trigger a page fault on missing or read only entry if the +write parameter is true. Page faults use the generic mm page fault code path +just like a CPU page fault. -Both function copy CPU page table into their pfns array argument. Each entry in -that array correspond to an address in the virtual range. HMM provide a set of -flags to help driver identify special CPU page table entries. +Both functions copy CPU page table entries into their pfns array argument. Each +entry in that array corresponds to an address in the virtual range. HMM +provides a set of flags to help the driver identify special CPU page table +entries. Locking with the update() callback is the most important aspect the driver must -respect in order to keep things properly synchronize. The usage pattern is : +respect in order to keep things properly synchronized. The usage pattern is: int driver_populate_range(...) {
@@ -233,43 +243,44 @@ Locking with the update() callback is the most important aspect the driver must return 0; } -The driver->update lock is the same lock that driver takes inside its update() -callback. That lock must be call before hmm_vma_range_done() to avoid any race -with a concurrent CPU page table update. +The driver->update lock is the same lock that the driver takes inside its +update() callback. That lock must be held before hmm_vma_range_done() to avoid +any race with a concurrent CPU page table update. -HMM implements all this on top of the mmu_notifier API because we wanted to a -simpler API and also to be able to perform optimization latter own like doing -concurrent device update in multi-devices scenario. +HMM implements all this on top of the mmu_notifier API because we wanted a +simpler API and also to be able to perform optimizations latter on like doing +concurrent device updates in multi-devices scenario. -HMM also serve as an impedence missmatch between how CPU page table update are -done (by CPU write to the page table and TLB flushes) from how device update -their own page table. Device update is a multi-step process, first appropriate -commands are write to a buffer, then this buffer is schedule for execution on -the device. It is only once the device has executed commands in the buffer that -the update is done. Creating and scheduling update command buffer can happen -concurrently for multiple devices. Waiting for each device to report commands -as executed is serialize (there is no point in doing this concurrently). +HMM also serves as an impedence mismatch between how CPU page table updates +are done (by CPU write to the page table and TLB flushes) and how devices +update their own page table. Device updates are a multi-step process. First, +appropriate commands are writen to a buffer, then this buffer is scheduled for +execution on the device. It is only once the device has executed commands in +the buffer that the update is done. Creating and scheduling the update command +buffer can happen concurrently for multiple devices. Waiting for each device to +report commands as executed is serialized (there is no point in doing this +concurrently). ------------------------------------------------------------------------------- 5) Represent and manage device memory from core kernel point of view -Several differents design were try to support device memory. First one use -device specific data structure to keep information about migrated memory and -HMM hooked itself in various place of mm code to handle any access to address -that were back by device memory. It turns out that this ended up replicating -most of the fields of struct page and also needed many kernel code path to be -updated to understand this new kind of memory. +Several different designs were tried to support device memory. First one used +a device specific data structure to keep information about migrated memory and +HMM hooked itself in various places of mm code to handle any access to +addresses that were backed by device memory. It turns out that this ended up +replicating most of the fields of struct page and also needed many kernel code +paths to be updated to understand this new kind of memory. -Thing is most kernel code path never try to access the memory behind a page -but only care about struct page contents. Because of this HMM switchted to -directly using struct page for device memory which left most kernel code path -un-aware of the difference. We only need to make sure that no one ever try to -map those page from the CPU side. +Most kernel code paths never try to access the memory behind a page +but only care about struct page contents. Because of this, HMM switched to +directly using struct page for device memory which left most kernel code paths +unaware of the difference. We only need to make sure that no one ever tries to +map those pages from the CPU side. -HMM provide a set of helpers to register and hotplug device memory as a new -region needing struct page. This is offer through a very simple API: +HMM provides a set of helpers to register and hotplug device memory as a new +region needing a struct page. This is offered through a very simple API: struct hmm_devmem *hmm_devmem_add(const struct hmm_devmem_ops *ops, struct device *device,
@@ -289,18 +300,19 @@ HMM provide a set of helpers to register and hotplug device memory as a new }; The first callback (free()) happens when the last reference on a device page is -drop. This means the device page is now free and no longer use by anyone. The -second callback happens whenever CPU try to access a device page which it can -not do. This second callback must trigger a migration back to system memory. +dropped. This means the device page is now free and no longer used by anyone. +The second callback happens whenever the CPU tries to access a device page +which it can not do. This second callback must trigger a migration back to +system memory. ------------------------------------------------------------------------------- -6) Migrate to and from device memory +6) Migration to and from device memory -Because CPU can not access device memory, migration must use device DMA engine -to perform copy from and to device memory. For this we need a new migration -helper: +Because the CPU can not access device memory, migration must use the device DMA +engine to perform copy from and to device memory. For this we need a new +migration helper: int migrate_vma(const struct migrate_vma_ops *ops, struct vm_area_struct *vma,
@@ -311,15 +323,15 @@ to perform copy from and to device memory. For this we need a new migration unsigned long *dst, void *private); -Unlike other migration function it works on a range of virtual address, there -is two reasons for that. First device DMA copy has a high setup overhead cost +Unlike other migration functions it works on a range of virtual address, there +are two reasons for that. First, device DMA copy has a high setup overhead cost and thus batching multiple pages is needed as otherwise the migration overhead -make the whole excersie pointless. The second reason is because driver trigger -such migration base on range of address the device is actively accessing. +makes the whole exersize pointless. The second reason is because the +migration might be for a range of addresses the device is actively accessing. -The migrate_vma_ops struct define two callbacks. First one (alloc_and_copy()) -control destination memory allocation and copy operation. Second one is there -to allow device driver to perform cleanup operation after migration. +The migrate_vma_ops struct defines two callbacks. First one (alloc_and_copy()) +controls destination memory allocation and copy operation. Second one is there +to allow the device driver to perform cleanup operations after migration. struct migrate_vma_ops { void (*alloc_and_copy)(struct vm_area_struct *vma,
@@ -336,19 +348,19 @@ to allow device driver to perform cleanup operation after migration. void *private); }; -It is important to stress that this migration helpers allow for hole in the +It is important to stress that these migration helpers allow for holes in the virtual address range. Some pages in the range might not be migrated for all -the usual reasons (page is pin, page is lock, ...). This helper does not fail -but just skip over those pages. +the usual reasons (page is pinned, page is locked, ...). This helper does not +fail but just skips over those pages. -The alloc_and_copy() might as well decide to not migrate all pages in the -range (for reasons under the callback control). For those the callback just -have to leave the corresponding dst entry empty. +The alloc_and_copy() might decide to not migrate all pages in the +range (for reasons under the callback control). For those, the callback just +has to leave the corresponding dst entry empty. -Finaly the migration of the struct page might fails (for file back page) for +Finally, the migration of the struct page might fail (for file backed page) for various reasons (failure to freeze reference, or update page cache, ...). If -that happens then the finalize_and_map() can catch any pages that was not -migrated. Note those page were still copied to new page and thus we wasted +that happens, then the finalize_and_map() can catch any pages that were not +migrated. Note those pages were still copied to a new page and thus we wasted bandwidth but this is considered as a rare event and a price that we are willing to pay to keep all the code simpler.
@@ -358,27 +370,27 @@ willing to pay to keep all the code simpler. 7) Memory cgroup (memcg) and rss accounting For now device memory is accounted as any regular page in rss counters (either -anonymous if device page is use for anonymous, file if device page is use for -file back page or shmem if device page is use for share memory). This is a -deliberate choice to keep existing application that might start using device -memory without knowing about it to keep runing unimpacted. - -Drawbacks is that OOM killer might kill an application using a lot of device -memory and not a lot of regular system memory and thus not freeing much system -memory. We want to gather more real world experience on how application and -system react under memory pressure in the presence of device memory before +anonymous if device page is used for anonymous, file if device page is used for +file backed page or shmem if device page is used for shared memory). This is a +deliberate choice to keep existing applications, that might start using device +memory without knowing about it, running unimpacted. + +A Drawback is that the OOM killer might kill an application using a lot of +device memory and not a lot of regular system memory and thus not freeing much +system memory. We want to gather more real world experience on how applications +and system react under memory pressure in the presence of device memory before deciding to account device memory differently. -Same decision was made for memory cgroup. Device memory page are accounted +Same decision was made for memory cgroup. Device memory pages are accounted against same memory cgroup a regular page would be accounted to. This does simplify migration to and from device memory. This also means that migration back from device memory to regular memory can not fail because it would go above memory cgroup limit. We might revisit this choice latter on once we -get more experience in how device memory is use and its impact on memory +get more experience in how device memory is used and its impact on memory resource control. -Note that device memory can never be pin nor by device driver nor through GUP +Note that device memory can never be pinned by device driver nor through GUP and thus such memory is always free upon process exit. Or when last reference -is drop in case of share memory or file back memory. +is dropped in case of shared memory or file backed memory.
MAINTAINERS +1 -0
@@ -6466,6 +6466,7 @@ L: [email protected] S: Maintained F: mm/hmm* F: include/linux/hmm* +F: Documentation/vm/hmm.txt HOST AP DRIVER M: Jouni Malinen <[email protected]>
Signature · 1 line
2.14.3
Randy Dunlap
reviewerRe: [PATCH 01/15] mm/hmm: documentation editorial update to HMM documentation
Quoted 34 lines · On 03/22/2018 05:55 PM, [email protected] wrote:
From: Ralph Campbell <[email protected]>
This patch updates the documentation for HMM to fix minor typos and phrasing to be a bit more readable.
Signed-off-by Ralph Campbell <[email protected]>Signed-off-by Jérôme Glisse <[email protected]>Cc Stephen Bates <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Logan Gunthorpe <[email protected]>Cc Evgeny Baskakov <[email protected]>Cc Mark Hairgrove <[email protected]>Cc John Hubbard <[email protected]>--- Documentation/vm/hmm.txt | 360 ++++++++++++++++++++++++----------------------- MAINTAINERS | 1 + 2 files changed, 187 insertions(+), 174 deletions(-)Documentation/vm/hmm.txt +0
-9@@ -1,151 +1,159 @@ Heterogeneous Memory Management (HMM) -Transparently allow any component of a program to use any memory region of said -program with a device without using device specific memory allocator. This is -becoming a requirement to simplify the use of advance heterogeneous computing -where GPU, DSP or FPGA are use to perform various computations. - -This document is divided as follow, in the first section i expose the problems -related to the use of a device specific allocator. The second section i expose -the hardware limitations that are inherent to many platforms. The third section -gives an overview of HMM designs. The fourth section explains how CPU page-
Documentation/vm/hmm.txt line 3
Quoted 10 lines
Documentation/vm/hmm.txt +1
-9-table mirroring works and what is HMM purpose in this context. Fifth section -deals with how device memory is represented inside the kernel. Finaly the last -section present the new migration helper that allow to leverage the device DMA -engine. - - -1) Problems of using device specific memory allocator: -2) System bus, device memory characteristics -3) Share address space and migration +Provide infrastructure and helpers to integrate non conventional memory (device
non-conventional
Documentation/vm/hmm.txt line 4
Quoted 1 line
Documentation/vm/hmm.txt +1
-0+memory like GPU on board memory) into regular kernel code path. Corner stone of
path, with the cornerstone of
Documentation/vm/hmm.txt line 5
Quoted 1 line
Documentation/vm/hmm.txt +1
-0+this being specialize struct page for such memory (see sections 5 to 7 of this
specialized
Documentation/vm/hmm.txt line 8
Quoted 3 lines
Documentation/vm/hmm.txt +3
-0+document). + +HMM also provide optional helpers for SVM (Share Virtual Memory) ie allowing a
provides Memory), i.e.,
Documentation/vm/hmm.txt line 11
Quoted 3 lines
Documentation/vm/hmm.txt +3
-0+device to transparently access program address coherently with the CPU meaning +that any valid pointer on the CPU is also a valid pointer for the device. This +is becoming a mandatory to simplify the use of advance heterogeneous computing
becoming mandatory advanced
Documentation/vm/hmm.txt line 19
Quoted 8 lines
Documentation/vm/hmm.txt +8
-0+where GPU, DSP, or FPGA are used to perform various computations on behalf of +a process. + +This document is divided as follows: in the first section I expose the problems +related to using device specific memory allocators. In the second section, I +expose the hardware limitations that are inherent to many platforms. The third +section gives an overview of the HMM design. The fourth section explains how +CPU page-table mirroring works and what is HMM's purpose in this context. The
and the purpose of HMM in this context.
Quoted 18 lines
Documentation/vm/hmm.txt +9
-2+fifth section deals with how device memory is represented inside the kernel. +Finally, the last section presents a new migration helper that allows lever- +aging the device DMA engine. + + +1) Problems of using a device specific memory allocator: +2) I/O bus, device memory characteristics +3) Shared address space and migration 4) Address space mirroring implementation and API 5) Represent and manage device memory from core kernel point of view -6) Migrate to and from device memory +6) Migration to and from device memory 7) Memory cgroup (memcg) and rss accounting ------------------------------------------------------------------------------- -1) Problems of using device specific memory allocator:
Documentation/vm/hmm.txt line 38
Quoted 10 lines
Documentation/vm/hmm.txt +2
-7+1) Problems of using a device specific memory allocator: -Device with large amount of on board memory (several giga bytes) like GPU have -historically manage their memory through dedicated driver specific API. This -creates a disconnect between memory allocated and managed by device driver and -regular application memory (private anonymous, share memory or regular file -back memory). From here on i will refer to this aspect as split address space. -I use share address space to refer to the opposite situation ie one in which -any memory region can be use by device transparently. +Devices with a large amount of on board memory (several giga bytes) like GPUs
gigabytes)
Documentation/vm/hmm.txt line 47
Quoted 9 lines
Documentation/vm/hmm.txt +7
-0+have historically managed their memory through dedicated driver specific APIs. +This creates a disconnect between memory allocated and managed by a device +driver and regular application memory (private anonymous, shared memory, or +regular file backed memory). From here on I will refer to this aspect as split +address space. I use shared address space to refer to the opposite situation: +i.e., one in which any application memory region can be used by a device +transparently. Split address space because device can only access memory allocated through the
Awkward sentence: maybe:
Split address space happens because
Documentation/vm/hmm.txt line 52
Quoted 9 lines
Documentation/vm/hmm.txt +4
-4-device specific API. This imply that all memory object in a program are not -equal from device point of view which complicate large program that rely on a -wide set of libraries. +device specific API. This implies that all memory objects in a program are not +equal from the device point of view which complicates large programs that rely +on a wide set of libraries. -Concretly this means that code that wants to leverage device like GPU need to +Concretly this means that code that wants to leverage devices like GPUs need to
Concretely needs
Documentation/vm/hmm.txt line 53
Quoted 1 line
Documentation/vm/hmm.txt +0
-0copy object between genericly allocated memory (malloc, mmap private/share/)
object [or an object] between generically
Documentation/vm/hmm.txt line 54
Quoted 1 line
Documentation/vm/hmm.txt +0
-0and memory allocated through the device driver API (this still end up with an
ends up
Documentation/vm/hmm.txt line 57
Quoted 8 lines
Documentation/vm/hmm.txt +1
-5mmap but of the device file). -For flat dataset (array, grid, image, ...) this isn't too hard to achieve but -complex data-set (list, tree, ...) are hard to get right. Duplicating a complex -data-set need to re-map all the pointer relations between each of its elements. -This is error prone and program gets harder to debug because of the duplicate -data-set. +For flat data-sets (array, grid, image, ...) this isn't too hard to achieve but
data sets
Documentation/vm/hmm.txt line 58
Quoted 1 line
Documentation/vm/hmm.txt +1
-0+complex data-sets (list, tree, ...) are hard to get right. Duplicating a
data sets
Documentation/vm/hmm.txt line 59
Quoted 1 line
Documentation/vm/hmm.txt +1
-0+complex data-set needs to re-map all the pointer relations between each of its
data set
Documentation/vm/hmm.txt line 61
Quoted 2 lines
Documentation/vm/hmm.txt +2
-0+elements. This is error prone and program gets harder to debug because of the +duplicate data-set and addresses.
data set
Documentation/vm/hmm.txt line 63
Quoted 6 lines
Documentation/vm/hmm.txt +1
-4-Split address space also means that library can not transparently use data they -are getting from core program or other library and thus each library might have -to duplicate its input data-set using specific memory allocator. Large project -suffer from this and waste resources because of the various memory copy. +Split address space also means that libraries can not transparently use data
cannot
Documentation/vm/hmm.txt line 65
Quoted 2 lines
Documentation/vm/hmm.txt +2
-0+they are getting from the core program or another library and thus each library +might have to duplicate its input data-set using the device specific memory
data set
Documentation/vm/hmm.txt line 69
Quoted 4 lines
Documentation/vm/hmm.txt +2
-0+allocator. Large projects suffer from this and waste resources because of the +various memory copies. Duplicating each library API to accept as input or output memory allocted by
allocated
Quoted 13 lines
Documentation/vm/hmm.txt +5
-6each device specific allocator is not a viable option. It would lead to a -combinatorial explosions in the library entry points. +combinatorial explosion in the library entry points. -Finaly with the advance of high level language constructs (in C++ but in other -language too) it is now possible for compiler to leverage GPU or other devices -without even the programmer knowledge. Some of compiler identified patterns are -only do-able with a share address. It is as well more reasonable to use a share -address space for all the other patterns. +Finally, with the advance of high level language constructs (in C++ but in +other languages too) it is now possible for the compiler to leverage GPUs and +other devices without programmer knowledge. Some compiler identified patterns +are only do-able with a shared address space. It is also more reasonable to use
Documentation/vm/hmm.txt line 84
Quoted 10 lines
Documentation/vm/hmm.txt +3
-2+a shared address space for all other patterns. ------------------------------------------------------------------------------- -2) System bus, device memory characteristics +2) I/O bus, device memory characteristics -System bus cripple share address due to few limitations. Most system bus only +I/O buses cripple shared address due to few limitations. Most I/O buses only
shared address spaces due to a few limitations.
Documentation/vm/hmm.txt line 85
Quoted 1 line
Documentation/vm/hmm.txt +0
-0allow basic memory access from device to main memory, even cache coherency is
memory; even
Documentation/vm/hmm.txt line 92
Quoted 13 lines
Documentation/vm/hmm.txt +6
-6-often optional. Access to device memory from CPU is even more limited, most -often than not it is not cache coherent. +often optional. Access to device memory from CPU is even more limited. More +often than not, it is not cache coherent. -If we only consider the PCIE bus than device can access main memory (often -through an IOMMU) and be cache coherent with the CPUs. However it only allows -a limited set of atomic operation from device on main memory. This is worse -in the other direction the CPUs can only access a limited range of the device +If we only consider the PCIE bus, then a device can access main memory (often +through an IOMMU) and be cache coherent with the CPUs. However, it only allows +a limited set of atomic operations from device on main memory. This is worse +in the other direction, the CPU can only access a limited range of the device
other direction:
Documentation/vm/hmm.txt line 93
Quoted 1 line
Documentation/vm/hmm.txt +0
-0memory and can not perform atomic operations on it. Thus device memory can not
cannot cannot
Quoted 5 lines
Documentation/vm/hmm.txt +1
-2-be consider like regular memory from kernel point of view. +be considered the same as regular memory from the kernel point of view. Another crippling factor is the limited bandwidth (~32GBytes/s with PCIE 4.0 -and 16 lanes). This is 33 times less that fastest GPU memory (1 TBytes/s).
Documentation/vm/hmm.txt line 102
Quoted 10 lines
Documentation/vm/hmm.txt +5
-4-The final limitation is latency, access to main memory from the device has an -order of magnitude higher latency than when the device access its own memory. +and 16 lanes). This is 33 times less than the fastest GPU memory (1 TBytes/s). +The final limitation is latency. Access to main memory from the device has an +order of magnitude higher latency than when the device accesses its own memory. -Some platform are developing new system bus or additions/modifications to PCIE -to address some of those limitations (OpenCAPI, CCIX). They mainly allow two +Some platforms are developing new I/O buses or additions/modifications to PCIE +to address some of these limitations (OpenCAPI, CCIX). They mainly allow two
two-
Documentation/vm/hmm.txt line 104
Quoted 4 lines
Documentation/vm/hmm.txt +1
-2way cache coherency between CPU and device and allow all atomic operations the -architecture supports. Saddly not all platform are following this trends and -some major architecture are left without hardware solutions to those problems. +architecture supports. Saddly, not all platforms are following this trend and
Sadly,
Documentation/vm/hmm.txt line 107
Quoted 4 lines
Documentation/vm/hmm.txt +2
-1+some major architectures are left without hardware solutions to these problems. -So for share address space to make sense not only we must allow device to +So for shared address space to make sense, not only must we allow device to
devices to
Documentation/vm/hmm.txt line 108
Quoted 1 line
Documentation/vm/hmm.txt +0
-0access any memory memory but we must also permit any memory to be migrated to
any memory but
Quoted 39 lines
Documentation/vm/hmm.txt +4
-25device memory while device is using it (blocking CPU access while it happens). ------------------------------------------------------------------------------- -3) Share address space and migration +3) Shared address space and migration HMM intends to provide two main features. First one is to share the address -space by duplication the CPU page table into the device page table so same -address point to same memory and this for any valid main memory address in +space by duplicating the CPU page table in the device page table so the same +address points to the same physical memory for any valid main memory address in the process address space. -To achieve this, HMM offer a set of helpers to populate the device page table +To achieve this, HMM offers a set of helpers to populate the device page table while keeping track of CPU page table updates. Device page table updates are -not as easy as CPU page table updates. To update the device page table you must -allow a buffer (or use a pool of pre-allocated buffer) and write GPU specifics -commands in it to perform the update (unmap, cache invalidations and flush, -...). This can not be done through common code for all device. Hence why HMM -provides helpers to factor out everything that can be while leaving the gory -details to the device driver. - -The second mechanism HMM provide is a new kind of ZONE_DEVICE memory that does -allow to allocate a struct page for each page of the device memory. Those page -are special because the CPU can not map them. They however allow to migrate -main memory to device memory using exhisting migration mechanism and everything -looks like if page was swap out to disk from CPU point of view. Using a struct -page gives the easiest and cleanest integration with existing mm mechanisms. -Again here HMM only provide helpers, first to hotplug new ZONE_DEVICE memory -for the device memory and second to perform migration. Policy decision of what -and when to migrate things is left to the device driver. - -Note that any CPU access to a device page trigger a page fault and a migration -back to main memory ie when a page backing an given address A is migrated from -a main memory page to a device page then any CPU access to address A trigger a -page fault and initiate a migration back to main memory.
Documentation/vm/hmm.txt line 126
Quoted 10 lines
Documentation/vm/hmm.txt +4
-6- - -With this two features, HMM not only allow a device to mirror a process address -space and keeps both CPU and device page table synchronize, but also allow to -leverage device memory by migrating part of data-set that is actively use by a -device. +not as easy as CPU page table updates. To update the device page table, you must +allocate a buffer (or use a pool of pre-allocated buffers) and write GPU +specific commands in it to perform the update (unmap, cache invalidations, and +flush, ...). This can not be done through common code for all devices. Hence
cannot
Documentation/vm/hmm.txt line 130
Quoted 4 lines
Documentation/vm/hmm.txt +4
-0+why HMM provides helpers to factor out everything that can be while leaving the +hardware specific details to the device driver. + +The second mechanism HMM provides, is a new kind of ZONE_DEVICE memory that
provides is
Documentation/vm/hmm.txt line 132
Quoted 2 lines
Documentation/vm/hmm.txt +2
-0+allows allocating a struct page for each page of the device memory. Those pages +are special because the CPU can not map them. However, they allow migrating
cannot
Quoted 5 lines
Documentation/vm/hmm.txt +5
-0+main memory to device memory using existing migration mechanisms and everything +looks like a page is swapped out to disk from the CPU point of view. Using a +struct page gives the easiest and cleanest integration with existing mm mech- +anisms. Here again, HMM only provides helpers, first to hotplug new ZONE_DEVICE +memory for the device memory and second to perform migration. Policy decisions
Documentation/vm/hmm.txt line 147
Quoted 10 lines
Documentation/vm/hmm.txt +10
-0+of what and when to migrate things is left to the device driver. + +Note that any CPU access to a device page triggers a page fault and a migration +back to main memory. For example, when a page backing a given CPU address A is +migrated from a main memory page to a device page, then any CPU access to +address A triggers a page fault and initiates a migration back to main memory. + +With these two features, HMM not only allows a device to mirror process address +space and keeping both CPU and device page table synchronized, but also lever- +ages device memory by migrating the part of the data-set that is actively being
data set
Documentation/vm/hmm.txt line 157
Quoted 12 lines
Documentation/vm/hmm.txt +3
-2+used by the device. ------------------------------------------------------------------------------- 4) Address space mirroring implementation and API -Address space mirroring main objective is to allow to duplicate range of CPU -page table into a device page table and HMM helps keeping both synchronize. A +Address space mirroring's main objective is to allow duplication of a range of +CPU page table into a device page table; HMM helps keep both synchronized. A device driver that want to mirror a process address space must start with the
wants
Documentation/vm/hmm.txt device driver that want to mirror a process address space must start with the · line 165
Quoted 6 lines
Documentation/vm/hmm.txt +0
-0registration of an hmm_mirror struct:@@ -155,8 +163,8 @@ device driver that want to mirror a process address space must start with the struct mm_struct *mm); The locked variant is to be use when the driver is already holding the mmap_sem
to be used
Quoted 27 lines
Documentation/vm/hmm.txt +7
-7-of the mm in write mode. The mirror struct has a set of callback that are use -to propagate CPU page table: +of the mm in write mode. The mirror struct has a set of callbacks that are used +to propagate CPU page tables: struct hmm_mirror_ops { /* sync_cpu_device_pagetables() - synchronize page tables@@ -181,13 +189,13 @@ of the mm in write mode. The mirror struct has a set of callback that are use unsigned long end); }; -Device driver must perform update to the range following action (turn range -read only, or fully unmap, ...). Once driver callback returns the device must -be done with the update. +The device driver must perform the update action to the range (mark range +read only, or fully unmap, ...). The device must be done with the update before +the driver callback returns. -When device driver wants to populate a range of virtual address it can use -either: +When the device driver wants to populate a range of virtual addresses, it can +use either: int hmm_vma_get_pfns(struct vm_area_struct *vma, struct hmm_range *range, unsigned long start,
Documentation/vm/hmm.txt When device driver wants to populate a range of virtual address it can use · line 213
Quoted 9 lines
Documentation/vm/hmm.txt +2
-4@@ -201,17 +209,19 @@ When device driver wants to populate a range of virtual address it can use bool write, bool block); -First one (hmm_vma_get_pfns()) will only fetch present CPU page table entry and -will not trigger a page fault on missing or non present entry. The second one -do trigger page fault on missing or read only entry if write parameter is true. -Page fault use the generic mm page fault code path just like a CPU page fault. +The first one (hmm_vma_get_pfns()) will only fetch present CPU page table +entries and will not trigger a page fault on missing or non present entries.
non-present
Documentation/vm/hmm.txt When device driver wants to populate a range of virtual address it can use · line 214
Quoted 1 line
Documentation/vm/hmm.txt +1
-0+The second one does trigger a page fault on missing or read only entry if the
read-only
Quoted 34 lines
Documentation/vm/hmm.txt +13
-10+write parameter is true. Page faults use the generic mm page fault code path +just like a CPU page fault. -Both function copy CPU page table into their pfns array argument. Each entry in -that array correspond to an address in the virtual range. HMM provide a set of -flags to help driver identify special CPU page table entries. +Both functions copy CPU page table entries into their pfns array argument. Each +entry in that array corresponds to an address in the virtual range. HMM +provides a set of flags to help the driver identify special CPU page table +entries. Locking with the update() callback is the most important aspect the driver must -respect in order to keep things properly synchronize. The usage pattern is : +respect in order to keep things properly synchronized. The usage pattern is: int driver_populate_range(...) {@@ -233,43 +243,44 @@ Locking with the update() callback is the most important aspect the driver must return 0; } -The driver->update lock is the same lock that driver takes inside its update() -callback. That lock must be call before hmm_vma_range_done() to avoid any race -with a concurrent CPU page table update. +The driver->update lock is the same lock that the driver takes inside its +update() callback. That lock must be held before hmm_vma_range_done() to avoid +any race with a concurrent CPU page table update. -HMM implements all this on top of the mmu_notifier API because we wanted to a -simpler API and also to be able to perform optimization latter own like doing -concurrent device update in multi-devices scenario. +HMM implements all this on top of the mmu_notifier API because we wanted a +simpler API and also to be able to perform optimizations latter on like doing +concurrent device updates in multi-devices scenario.
Documentation/vm/hmm.txt Locking with the update() callback is the most important aspect the driver must · line 254
Quoted 10 lines
Documentation/vm/hmm.txt +1
-8-HMM also serve as an impedence missmatch between how CPU page table update are -done (by CPU write to the page table and TLB flushes) from how device update -their own page table. Device update is a multi-step process, first appropriate -commands are write to a buffer, then this buffer is schedule for execution on -the device. It is only once the device has executed commands in the buffer that -the update is done. Creating and scheduling update command buffer can happen -concurrently for multiple devices. Waiting for each device to report commands -as executed is serialize (there is no point in doing this concurrently). +HMM also serves as an impedence mismatch between how CPU page table updates
impedance
Documentation/vm/hmm.txt Locking with the update() callback is the most important aspect the driver must · line 257
Quoted 3 lines
Documentation/vm/hmm.txt +3
-0+are done (by CPU write to the page table and TLB flushes) and how devices +update their own page table. Device updates are a multi-step process. First, +appropriate commands are writen to a buffer, then this buffer is scheduled for
written
Quoted 43 lines
Documentation/vm/hmm.txt +18
-13+execution on the device. It is only once the device has executed commands in +the buffer that the update is done. Creating and scheduling the update command +buffer can happen concurrently for multiple devices. Waiting for each device to +report commands as executed is serialized (there is no point in doing this +concurrently). ------------------------------------------------------------------------------- 5) Represent and manage device memory from core kernel point of view -Several differents design were try to support device memory. First one use -device specific data structure to keep information about migrated memory and -HMM hooked itself in various place of mm code to handle any access to address -that were back by device memory. It turns out that this ended up replicating -most of the fields of struct page and also needed many kernel code path to be -updated to understand this new kind of memory. +Several different designs were tried to support device memory. First one used +a device specific data structure to keep information about migrated memory and +HMM hooked itself in various places of mm code to handle any access to +addresses that were backed by device memory. It turns out that this ended up +replicating most of the fields of struct page and also needed many kernel code +paths to be updated to understand this new kind of memory. -Thing is most kernel code path never try to access the memory behind a page -but only care about struct page contents. Because of this HMM switchted to -directly using struct page for device memory which left most kernel code path -un-aware of the difference. We only need to make sure that no one ever try to -map those page from the CPU side. +Most kernel code paths never try to access the memory behind a page +but only care about struct page contents. Because of this, HMM switched to +directly using struct page for device memory which left most kernel code paths +unaware of the difference. We only need to make sure that no one ever tries to +map those pages from the CPU side. -HMM provide a set of helpers to register and hotplug device memory as a new -region needing struct page. This is offer through a very simple API: +HMM provides a set of helpers to register and hotplug device memory as a new +region needing a struct page. This is offered through a very simple API: struct hmm_devmem *hmm_devmem_add(const struct hmm_devmem_ops *ops, struct device *device,
Documentation/vm/hmm.txt HMM provide a set of helpers to register and hotplug device memory as a new · line 305
Quoted 9 lines
Documentation/vm/hmm.txt +3
-3@@ -289,18 +300,19 @@ HMM provide a set of helpers to register and hotplug device memory as a new }; The first callback (free()) happens when the last reference on a device page is -drop. This means the device page is now free and no longer use by anyone. The -second callback happens whenever CPU try to access a device page which it can -not do. This second callback must trigger a migration back to system memory. +dropped. This means the device page is now free and no longer used by anyone. +The second callback happens whenever the CPU tries to access a device page +which it can not do. This second callback must trigger a migration back to
cannot
Documentation/vm/hmm.txt HMM provide a set of helpers to register and hotplug device memory as a new · line 313
Quoted 12 lines
Documentation/vm/hmm.txt +3
-4+system memory. ------------------------------------------------------------------------------- -6) Migrate to and from device memory +6) Migration to and from device memory -Because CPU can not access device memory, migration must use device DMA engine -to perform copy from and to device memory. For this we need a new migration -helper: +Because the CPU can not access device memory, migration must use the device DMA
cannot
Quoted 7 lines
Documentation/vm/hmm.txt +2
-0+engine to perform copy from and to device memory. For this we need a new +migration helper: int migrate_vma(const struct migrate_vma_ops *ops, struct vm_area_struct *vma,@@ -311,15 +323,15 @@ to perform copy from and to device memory. For this we need a new migration unsigned long *dst,
Documentation/vm/hmm.txt to perform copy from and to device memory. For this we need a new migration · line 329
Quoted 10 lines
Documentation/vm/hmm.txt +3
-4void *private); -Unlike other migration function it works on a range of virtual address, there -is two reasons for that. First device DMA copy has a high setup overhead cost +Unlike other migration functions it works on a range of virtual address, there +are two reasons for that. First, device DMA copy has a high setup overhead cost and thus batching multiple pages is needed as otherwise the migration overhead -make the whole excersie pointless. The second reason is because driver trigger -such migration base on range of address the device is actively accessing. +makes the whole exersize pointless. The second reason is because the
exercise
Quoted 49 lines
Documentation/vm/hmm.txt +13
-17+migration might be for a range of addresses the device is actively accessing. -The migrate_vma_ops struct define two callbacks. First one (alloc_and_copy()) -control destination memory allocation and copy operation. Second one is there -to allow device driver to perform cleanup operation after migration. +The migrate_vma_ops struct defines two callbacks. First one (alloc_and_copy()) +controls destination memory allocation and copy operation. Second one is there +to allow the device driver to perform cleanup operations after migration. struct migrate_vma_ops { void (*alloc_and_copy)(struct vm_area_struct *vma,@@ -336,19 +348,19 @@ to allow device driver to perform cleanup operation after migration. void *private); }; -It is important to stress that this migration helpers allow for hole in the +It is important to stress that these migration helpers allow for holes in the virtual address range. Some pages in the range might not be migrated for all -the usual reasons (page is pin, page is lock, ...). This helper does not fail -but just skip over those pages. +the usual reasons (page is pinned, page is locked, ...). This helper does not +fail but just skips over those pages. -The alloc_and_copy() might as well decide to not migrate all pages in the -range (for reasons under the callback control). For those the callback just -have to leave the corresponding dst entry empty. +The alloc_and_copy() might decide to not migrate all pages in the +range (for reasons under the callback control). For those, the callback just +has to leave the corresponding dst entry empty. -Finaly the migration of the struct page might fails (for file back page) for +Finally, the migration of the struct page might fail (for file backed page) for various reasons (failure to freeze reference, or update page cache, ...). If -that happens then the finalize_and_map() can catch any pages that was not -migrated. Note those page were still copied to new page and thus we wasted +that happens, then the finalize_and_map() can catch any pages that were not +migrated. Note those pages were still copied to a new page and thus we wasted bandwidth but this is considered as a rare event and a price that we are willing to pay to keep all the code simpler.@@ -358,27 +370,27 @@ willing to pay to keep all the code simpler. 7) Memory cgroup (memcg) and rss accounting For now device memory is accounted as any regular page in rss counters (either -anonymous if device page is use for anonymous, file if device page is use for -file back page or shmem if device page is use for share memory). This is a -deliberate choice to keep existing application that might start using device -memory without knowing about it to keep runing unimpacted. -
Documentation/vm/hmm.txt willing to pay to keep all the code simpler. · line 378
Quoted 10 lines
Documentation/vm/hmm.txt +6
-4-Drawbacks is that OOM killer might kill an application using a lot of device -memory and not a lot of regular system memory and thus not freeing much system -memory. We want to gather more real world experience on how application and -system react under memory pressure in the presence of device memory before +anonymous if device page is used for anonymous, file if device page is used for +file backed page or shmem if device page is used for shared memory). This is a +deliberate choice to keep existing applications, that might start using device +memory without knowing about it, running unimpacted. + +A Drawback is that the OOM killer might kill an application using a lot of
drawback
Documentation/vm/hmm.txt willing to pay to keep all the code simpler. · line 388
Quoted 11 lines
Documentation/vm/hmm.txt +4
-1+device memory and not a lot of regular system memory and thus not freeing much +system memory. We want to gather more real world experience on how applications +and system react under memory pressure in the presence of device memory before deciding to account device memory differently. -Same decision was made for memory cgroup. Device memory page are accounted +Same decision was made for memory cgroup. Device memory pages are accounted against same memory cgroup a regular page would be accounted to. This does simplify migration to and from device memory. This also means that migration back from device memory to regular memory can not fail because it would
cannot
Quoted 11 lines
Documentation/vm/hmm.txt +3
-3go above memory cgroup limit. We might revisit this choice latter on once we -get more experience in how device memory is use and its impact on memory +get more experience in how device memory is used and its impact on memory resource control. -Note that device memory can never be pin nor by device driver nor through GUP +Note that device memory can never be pinned by device driver nor through GUP and thus such memory is always free upon process exit. Or when last reference -is drop in case of share memory or file back memory. +is dropped in case of shared memory or file backed memory.
Signature · 1 line
~Randy
PATCH 2/15 mm/hmm: fix header file if/else/endif maze v2
In reply to [email protected]From: Jérôme Glisse <[email protected]>
The #if/#else/#endif for IS_ENABLED(CONFIG_HMM) were wrong. Because of this after multiple include there was multiple definition of both hmm_mm_init() and hmm_mm_destroy() leading to build failure if HMM was enabled (CONFIG_HMM set).
Changed since v1:
- Fix the maze when CONFIG_HMM is disabled not just when it is disabled. This fix bot build failure.
- Improved commit message.
Signed-off-by Jérôme Glisse <[email protected]>
Acked-by Balbir Singh <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
--- include/linux/hmm.h | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
include/linux/hmm.h +1 -8
@@ -498,23 +498,16 @@ struct hmm_device {struct hmm_device *hmm_device_new (void *drvdata);void hmm_device_put (struct hmm_device *hmm_device);#endif /* CONFIG_DEVICE_PRIVATE || CONFIG_DEVICE_PUBLIC */ -#endif /* IS_ENABLED(CONFIG_HMM) */ /* Below are for HMM internal use only! Not to be used by device driver! */ -#if IS_ENABLED (CONFIG_HMM_MIRROR )void hmm_mm_destroy (struct mm_struct *mm);static inline void hmm_mm_init (struct mm_struct *mm) { mm->hmm =NULL ; } -#else /* IS_ENABLED(CONFIG_HMM_MIRROR) */ -static inline void hmm_mm_destroy (struct mm_struct *mm) {} -static inline void hmm_mm_init (struct mm_struct *mm) {} -#endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */ - -#else /* IS_ENABLED(CONFIG_HMM) */ static inline void hmm_mm_destroy (struct mm_struct *mm) {}static inline void hmm_mm_init (struct mm_struct *mm) {} +#endif /* IS_ENABLED(CONFIG_HMM) */ #endif /* LINUX_HMM_H */
Signature · 1 line
2.14.3
PATCH 3/15 mm/hmm: HMM should have a callback before MM is destroyed v3
In reply to [email protected]From: Ralph Campbell <[email protected]>
The hmm_mirror_register() function registers a callback for when the CPU pagetable is modified. Normally, the device driver will call hmm_mirror_unregister() when the process using the device is finished. However, if the process exits uncleanly, the struct_mm can be destroyed with no warning to the device driver.
Changed since v1:
- dropped VM_BUG_ON()
- cc stable
Changed since v2:
- drop stable
- Split list removale and call to driver release callback. This allow the release callback to wait on any pending fault handler without deadlock.
Signed-off-by Ralph Campbell <[email protected]>
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- include/linux/hmm.h | 10 ++++++++++ mm/hmm.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 38 insertions(+), 1 deletion(-)
include/linux/hmm.h +10 -0
@@ -218,6 +218,16 @@ enum hmm_update_type {* @update: callback to update range on a device */ struct hmm_mirror_ops { +/* release() - release hmm_mirror +* +* @mirror: pointer to struct hmm_mirror +* +* This is called when the mm_struct is being released. +* The callback should make sure no references to the mirror occur +* after the callback returns. +*/ +void (*release)(struct hmm_mirror *mirror); +/* sync_cpu_device_pagetables() - synchronize page tables * * @mirror: pointer to struct hmm_mirror
mm/hmm.c +28 -1
@@ -160,6 +160,32 @@ static void hmm_invalidate_range(struct hmm *hmm,up_read (&hmm->mirrors_sem); } +static void hmm_release (struct mmu_notifier *mn,struct mm_struct *mm) +{ +struct hmm_mirror *mirror; +struct hmm *hmm = mm->hmm; + +down_write (&hmm->mirrors_sem); + mirror = list_first_entry_or_null(&hmm->mirrors, struct hmm_mirror, + list); +while (mirror) { +list_del_init (&mirror->list); +if (mirror->ops->release) { +/* +* Drop mirrors_sem so callback can wait on any pending +* work that might itself trigger mmu_notifier callback +* and thus would deadlock with us. +*/ +up_write (&hmm->mirrors_sem); + mirror->ops->release (mirror); +down_write (&hmm->mirrors_sem); + } + mirror =list_first_entry_or_null (&hmm->mirrors, + struct hmm_mirror, list); + } +up_write (&hmm->mirrors_sem); +} + static void hmm_invalidate_range_start(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start,
@@ -185,6 +211,7 @@ static void hmm_invalidate_range_end(struct mmu_notifier *mn, }static const struct mmu_notifier_ops hmm_mmu_notifier_ops = { + .release = hmm_release, .invalidate_range_start = hmm_invalidate_range_start, .invalidate_range_end = hmm_invalidate_range_end, };
@@ -230,7 +257,7 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror)struct hmm *hmm = mirror->hmm;down_write (&hmm->mirrors_sem); -list_del (&mirror->list); +list_del_init (&mirror->list);up_write (&hmm->mirrors_sem); }EXPORT_SYMBOL (hmm_mirror_unregister);
Signature · 1 line
2.14.3
PATCH 4/15 mm/hmm: unregister mmu_notifier when last HMM client quit v3
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
This code was lost in translation at one point. This properly call mmu_notifier_unregister_no_release() once last user is gone. This fix the zombie mm_struct as without this patch we do not drop the refcount we have on it.
Changed since v1:
- close race window between a last mirror unregistering and a new mirror registering, which could have lead to use after free() kind of bug
Changed since v2:
- Avoid issue if there is multiple call to hmm_mirror_register() (which lead to use after free).
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
Cc John Hubbard <[email protected]>
--- mm/hmm.c | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-)
mm/hmm.c +35 -3
@@ -233,13 +233,24 @@ int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm)if (!mm || !mirror || !mirror->ops)return -EINVAL ; +again: mirror->hmm =hmm_register (mm);if (!mirror->hmm)return -ENOMEM ;down_write (&mirror->hmm->mirrors_sem); -list_add (&mirror->list, &mirror->hmm->mirrors); -up_write (&mirror->hmm->mirrors_sem); +if (mirror->hmm->mm ==NULL ) { +/* +* A racing hmm_mirror_unregister() is about to destroy the hmm +* struct. Try again to allocate a new one. +*/ +up_write (&mirror->hmm->mirrors_sem); + mirror->hmm =NULL ; + goto again; + }else { +list_add (&mirror->list, &mirror->hmm->mirrors); +up_write (&mirror->hmm->mirrors_sem); + }return 0 ; }
@@ -254,11 +265,32 @@ EXPORT_SYMBOL(hmm_mirror_register);*/ void hmm_mirror_unregister (struct hmm_mirror *mirror) { -struct hmm *hmm = mirror->hmm; +bool should_unregister = false; +struct mm_struct *mm; +struct hmm *hmm; + +if (mirror->hmm ==NULL ) +return ; + hmm = mirror->hmm;down_write (&hmm->mirrors_sem);list_del_init (&mirror->list); + should_unregister =list_empty (&hmm->mirrors); + mirror->hmm =NULL ; + mm = hmm->mm; + hmm->mm =NULL ;up_write (&hmm->mirrors_sem); + +if (!should_unregister || mm ==NULL ) +return ; + +spin_lock (&mm->page_table_lock); +if (mm->hmm == hmm) + mm->hmm =NULL ; +spin_unlock (&mm->page_table_lock); + +mmu_notifier_unregister_no_release (&hmm->mmu_notifier, mm); +kfree (hmm); }EXPORT_SYMBOL (hmm_mirror_unregister);
Signature · 1 line
2.14.3
PATCH 5/15 mm/hmm: hmm_pfns_bad() was accessing wrong struct
In reply to [email protected]From: Jérôme Glisse <[email protected]>
The private field of mm_walk struct point to an hmm_vma_walk struct and not to the hmm_range struct desired. Fix to get proper struct pointer.
Signed-off-by Jérôme Glisse <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
Cc John Hubbard <[email protected]>
--- mm/hmm.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/hmm.c +2 -1
@@ -336,7 +336,8 @@ static int hmm_pfns_bad(unsigned long addr,unsigned long end, struct mm_walk *walk) { - struct hmm_range *range = walk->private; + struct hmm_vma_walk *hmm_vma_walk = walk->private; +struct hmm_range *range = hmm_vma_walk->range;hmm_pfn_t *pfns = range->pfns;unsigned long i;
Signature · 1 line
2.14.3
PATCH 6/15 mm/hmm: use struct for hmm_vma_fault(), hmm_vma_get_pfns() parameters v2
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
Both hmm_vma_fault() and hmm_vma_get_pfns() were taking a hmm_range struct as parameter and were initializing that struct with others of their parameters. Have caller of those function do this as they are likely to already do and only pass this struct to both function this shorten function signature and make it easier in the future to add new parameters by simply adding them to the structure.
Changed since v1:
- Improved comments
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- include/linux/hmm.h | 18 ++++--------- mm/hmm.c | 78 +++++++++++++++++++---------------------------------- 2 files changed, 33 insertions(+), 63 deletions(-)
include/linux/hmm.h +5 -13
@@ -274,6 +274,7 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror);/* * struct hmm_range - track invalidation lock on virtual address range * +* @vma: the vm area struct for the range * @list: all range lock are on a list * @start: range virtual start address (inclusive) * @end: range virtual end address (exclusive)
@@ -281,6 +282,7 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror);* @valid: pfns array did not change since it has been fill by an HMM function */ struct hmm_range { +struct vm_area_struct *vma;struct list_head list;unsigned long start;unsigned long end;
@@ -301,12 +303,8 @@ struct hmm_range {* * IF YOU DO NOT FOLLOW THE ABOVE RULE THE SNAPSHOT CONTENT MIGHT BE INVALID ! */ -int hmm_vma_get_pfns (struct vm_area_struct *vma, -struct hmm_range *range, -unsigned long start, -unsigned long end, -hmm_pfn_t *pfns); -bool hmm_vma_range_done (struct vm_area_struct *vma,struct hmm_range *range); +int hmm_vma_get_pfns (struct hmm_range *range); +bool hmm_vma_range_done (struct hmm_range *range);/*
@@ -327,13 +325,7 @@ bool hmm_vma_range_done(struct vm_area_struct *vma, struct hmm_range *range);* * See the function description in mm/hmm.c for further documentation. */ -int hmm_vma_fault (struct vm_area_struct *vma, -struct hmm_range *range, -unsigned long start, -unsigned long end, -hmm_pfn_t *pfns, -bool write, -bool block); +int hmm_vma_fault (struct hmm_range *range,bool write,bool block);#endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */
mm/hmm.c +28 -50
205 lines not on this page · raw message
Signature · 1 line
2.14.3
PATCH 7/15 mm/hmm: remove HMM_PFN_READ flag and ignore peculiar architecture v2
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
Only peculiar architecture allow write without read thus assume that any valid pfn do allow for read. Note we do not care for write only because it does make sense with thing like atomic compare and exchange or any other operations that allow you to get the memory value through them.
Changed since v1:
- Fail early on vma without read permission and return an error
- Improve comments
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- include/linux/hmm.h | 16 +++++++--------- mm/hmm.c | 44 ++++++++++++++++++++++++++++++++++---------- 2 files changed, 41 insertions(+), 19 deletions(-)
include/linux/hmm.h +7 -9
@@ -83,8 +83,7 @@ struct hmm;* hmm_pfn_t - HMM uses its own pfn type to keep several flags per page * * Flags: -* HMM_PFN_VALID: pfn is valid -* HMM_PFN_READ: CPU page table has read permission set +* HMM_PFN_VALID: pfn is valid. It has, at least, read permission. * HMM_PFN_WRITE: CPU page table has write permission set * HMM_PFN_ERROR: corresponding CPU page table entry points to poisoned memory * HMM_PFN_EMPTY: corresponding CPU page table entry is pte_none()
@@ -97,13 +96,12 @@ struct hmm;typedef unsigned long hmm_pfn_t ;#define HMM_PFN_VALID (1 << 0) -#define HMM_PFN_READ (1 << 1) -#define HMM_PFN_WRITE (1 << 2) -#define HMM_PFN_ERROR (1 << 3) -#define HMM_PFN_EMPTY (1 << 4) -#define HMM_PFN_SPECIAL (1 << 5) -#define HMM_PFN_DEVICE_UNADDRESSABLE (1 << 6) -#define HMM_PFN_SHIFT 7 +#define HMM_PFN_WRITE (1 << 1) +#define HMM_PFN_ERROR (1 << 2) +#define HMM_PFN_EMPTY (1 << 3) +#define HMM_PFN_SPECIAL (1 << 4) +#define HMM_PFN_DEVICE_UNADDRESSABLE (1 << 5) +#define HMM_PFN_SHIFT 6/* * hmm_pfn_t_to_page() - return struct page pointed to by a valid hmm_pfn_t
mm/hmm.c +34 -10
101 lines not on this page · raw message
Signature · 1 line
2.14.3
PATCH 8/15 mm/hmm: use uint64_t for HMM pfn instead of defining hmm_pfn_t to ulong v2
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
All device driver we care about are using 64bits page table entry. In order to match this and to avoid useless define convert all HMM pfn to directly use uint64_t. It is a first step on the road to allow driver to directly use pfn value return by HMM (saving memory and CPU cycles use for conversion between the two).
Changed since v1:
- Fix comments typos
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- include/linux/hmm.h | 46 +++++++++++++++++++++------------------------- mm/hmm.c | 26 +++++++++++++------------- 2 files changed, 34 insertions(+), 38 deletions(-)
include/linux/hmm.h +21 -25
@@ -80,8 +80,6 @@struct hmm ;/* -* hmm_pfn_t - HMM uses its own pfn type to keep several flags per page -* * Flags: * HMM_PFN_VALID: pfn is valid. It has, at least, read permission. * HMM_PFN_WRITE: CPU page table has write permission set
@@ -93,8 +91,6 @@ struct hmm;* set and the pfn value is undefined. * HMM_PFN_DEVICE_UNADDRESSABLE: unaddressable device memory (ZONE_DEVICE) */ -typedef unsigned long hmm_pfn_t ; -#define HMM_PFN_VALID (1 << 0)#define HMM_PFN_WRITE (1 << 1)#define HMM_PFN_ERROR (1 << 2)
@@ -104,14 +100,14 @@ typedef unsigned long hmm_pfn_t;#define HMM_PFN_SHIFT 6/* -* hmm_pfn_t_to_page() - return struct page pointed to by a valid hmm_pfn_t -* @pfn: hmm_pfn_t to convert to struct page -* Returns: struct page pointer if pfn is a valid hmm_pfn_t, NULL otherwise +* hmm_pfn_to_page() - return struct page pointed to by a valid HMM pfn +* @pfn: HMM pfn value to get corresponding struct page from +* Returns: struct page pointer if pfn is a valid HMM pfn, NULL otherwise * -* If the hmm_pfn_t is valid (ie valid flag set) then return the struct page -* matching the pfn value stored in the hmm_pfn_t. Otherwise return NULL. +* If the HMM pfn is valid (ie valid flag set) then return the struct page +* matching the pfn value stored in the HMM pfn. Otherwise return NULL. */ -static inline struct page *hmm_pfn_t_to_page (hmm_pfn_t pfn) +static inline struct page *hmm_pfn_to_page (uint64_t pfn) {if (!(pfn &HMM_PFN_VALID ))return NULL ;
@@ -119,11 +115,11 @@ static inline struct page *hmm_pfn_t_to_page(hmm_pfn_t pfn) }/* -* hmm_pfn_t_to_pfn() - return pfn value store in a hmm_pfn_t -* @pfn: hmm_pfn_t to extract pfn from -* Returns: pfn value if hmm_pfn_t is valid, -1UL otherwise +* hmm_pfn_to_pfn() - return pfn value store in a HMM pfn +* @pfn: HMM pfn value to extract pfn from +* Returns: pfn value if HMM pfn is valid, -1UL otherwise */ -static inline unsigned long hmm_pfn_t_to_pfn (hmm_pfn_t pfn) +static inline unsigned long hmm_pfn_to_pfn (uint64_t pfn) {if (!(pfn &HMM_PFN_VALID ))return -1UL ;
@@ -131,21 +127,21 @@ static inline unsigned long hmm_pfn_t_to_pfn(hmm_pfn_t pfn) }/* -* hmm_pfn_t_from_page() - create a valid hmm_pfn_t value from struct page -* @page: struct page pointer for which to create the hmm_pfn_t -* Returns: valid hmm_pfn_t for the page +* hmm_pfn_from_page() - create a valid HMM pfn value from struct page +* @page: struct page pointer for which to create the HMM pfn +* Returns: valid HMM pfn for the page */ -static inline hmm_pfn_t hmm_pfn_t_from_page (struct page *page) +static inline uint64_t hmm_pfn_from_page (struct page *page) {return (page_to_pfn (page) <<HMM_PFN_SHIFT ) |HMM_PFN_VALID ; }/* -* hmm_pfn_t_from_pfn() - create a valid hmm_pfn_t value from pfn -* @pfn: pfn value for which to create the hmm_pfn_t -* Returns: valid hmm_pfn_t for the pfn +* hmm_pfn_from_pfn() - create a valid HMM pfn value from pfn +* @pfn: pfn value for which to create the HMM pfn +* Returns: valid HMM pfn for the pfn */ -static inline hmm_pfn_t hmm_pfn_t_from_pfn (unsigned long pfn) +static inline uint64_t hmm_pfn_from_pfn (unsigned long pfn) {return (pfn <<HMM_PFN_SHIFT ) |HMM_PFN_VALID ; }
@@ -284,7 +280,7 @@ struct hmm_range {struct list_head list;unsigned long start;unsigned long end; -hmm_pfn_t *pfns; +uint64_t *pfns;bool valid; };
@@ -307,7 +303,7 @@ bool hmm_vma_range_done(struct hmm_range *range);/* * Fault memory on behalf of device driver. Unlike handle_mm_fault(), this will -* not migrate any device memory back to system memory. The hmm_pfn_t array will +* not migrate any device memory back to system memory. The HMM pfn array will * be updated with the fault result and current snapshot of the CPU page table * for the range. *
@@ -316,7 +312,7 @@ bool hmm_vma_range_done(struct hmm_range *range);* function returns -EAGAIN. * * Return value does not reflect if the fault was successful for every single -* address or not. Therefore, the caller must to inspect the hmm_pfn_t array to +* address or not. Therefore, the caller must to inspect the HMM pfn array to * determine fault status for each address. * * Trying to fault inside an invalid vma will result in -EINVAL.
mm/hmm.c +13 -13
@@ -304,7 +304,7 @@ struct hmm_vma_walk {static int hmm_vma_do_fault (struct mm_walk *walk,unsigned long addr, -hmm_pfn_t *pfn) +uint64_t *pfn) {unsigned int flags =FAULT_FLAG_ALLOW_RETRY |FAULT_FLAG_REMOTE ;struct hmm_vma_walk *hmm_vma_walk = walk->private;
@@ -324,7 +324,7 @@ static int hmm_vma_do_fault(struct mm_walk *walk,return -EAGAIN ; } -static void hmm_pfns_special (hmm_pfn_t *pfns, +static void hmm_pfns_special (uint64_t *pfns,unsigned long addr,unsigned long end) {
@@ -338,7 +338,7 @@ static int hmm_pfns_bad(unsigned long addr, {struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range; -hmm_pfn_t *pfns = range->pfns; +uint64_t *pfns = range->pfns;unsigned long i; i = (addr - range->start) >>PAGE_SHIFT ;
@@ -348,7 +348,7 @@ static int hmm_pfns_bad(unsigned long addr,return 0 ; } -static void hmm_pfns_clear (hmm_pfn_t *pfns, +static void hmm_pfns_clear (uint64_t *pfns,unsigned long addr,unsigned long end) {
@@ -362,7 +362,7 @@ static int hmm_vma_walk_hole(unsigned long addr, {struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range; -hmm_pfn_t *pfns = range->pfns; +uint64_t *pfns = range->pfns;unsigned long i; hmm_vma_walk->last = addr;
@@ -387,7 +387,7 @@ static int hmm_vma_walk_clear(unsigned long addr, {struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range; -hmm_pfn_t *pfns = range->pfns; +uint64_t *pfns = range->pfns;unsigned long i; hmm_vma_walk->last = addr;
@@ -414,7 +414,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range;struct vm_area_struct *vma = walk->vma; -hmm_pfn_t *pfns = range->pfns; +uint64_t *pfns = range->pfns;unsigned long addr = start, i;bool write_fault;pte_t *ptep;
@@ -431,7 +431,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,if (pmd_devmap (*pmdp) ||pmd_trans_huge (*pmdp)) {unsigned long pfn; -hmm_pfn_t flag =0 ; +uint64_t flag =0 ;pmd_t pmd;/*
@@ -456,7 +456,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, pfn =pmd_pfn (pmd) +pte_index (addr); flag |=pmd_write (pmd) ?HMM_PFN_WRITE :0 ;for (; addr < end; addr +=PAGE_SIZE , i++, pfn++) - pfns[i] =hmm_pfn_t_from_pfn (pfn) | flag; + pfns[i] =hmm_pfn_from_pfn (pfn) | flag;return 0 ; }
@@ -490,7 +490,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,* device and report anything else as error. */ if (is_device_private_entry (entry)) { - pfns[i] =hmm_pfn_t_from_pfn (swp_offset (entry)); + pfns[i] =hmm_pfn_from_pfn (swp_offset (entry));if (is_write_device_private_entry (entry)) { pfns[i] |=HMM_PFN_WRITE ; }else if (write_fault)
@@ -515,7 +515,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,if (write_fault && !pte_write (pte)) goto fault; - pfns[i] =hmm_pfn_t_from_pfn (pte_pfn (pte)); + pfns[i] =hmm_pfn_from_pfn (pte_pfn (pte)); pfns[i] |=pte_write (pte) ?HMM_PFN_WRITE :0 ;continue ;
@@ -678,8 +678,8 @@ EXPORT_SYMBOL(hmm_vma_range_done);* This is similar to a regular CPU page fault except that it will not trigger * any memory migration if the memory being faulted is not accessible by CPUs. * -* On error, for one virtual address in the range, the function will set the -* hmm_pfn_t error flag for the corresponding pfn entry. +* On error, for one virtual address in the range, the function will mark the +* corresponding HMM pfn entry with an error flag. * * Expected use pattern: * retry:
Signature · 1 line
2.14.3
PATCH 9/15 mm/hmm: cleanup special vma handling (VM_SPECIAL)
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
Special vma (one with any of the VM_SPECIAL flags) can not be access by device because there is no consistent model across device drivers on those vma and their backing memory.
This patch directly use hmm_range struct for hmm_pfns_special() argument as it is always affecting the whole vma and thus the whole range.
It also make behavior consistent after this patch both hmm_vma_fault() and hmm_vma_get_pfns() returns -EINVAL when facing such vma. Previously hmm_vma_fault() returned 0 and hmm_vma_get_pfns() return -EINVAL but both were filling the HMM pfn array with special entry.
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- mm/hmm.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-)
mm/hmm.c +20 -20
@@ -324,14 +324,6 @@ static int hmm_vma_do_fault(struct mm_walk *walk,return -EAGAIN ; } -static void hmm_pfns_special (uint64_t *pfns, -unsigned long addr, -unsigned long end) -{ -for (; addr < end; addr +=PAGE_SIZE , pfns++) - *pfns =HMM_PFN_SPECIAL ; -} -static int hmm_pfns_bad (unsigned long addr,unsigned long end,struct mm_walk *walk)
@@ -529,6 +521,14 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,return 0 ; } +static void hmm_pfns_special (struct hmm_range *range) +{ +unsigned long addr = range->start, i =0 ; + +for (; addr < range->end; addr +=PAGE_SIZE , i++) + range->pfns[i] =HMM_PFN_SPECIAL ; +} +/* * hmm_vma_get_pfns() - snapshot CPU page table for a range of virtual addresses * @range: range being snapshotted
@@ -553,12 +553,6 @@ int hmm_vma_get_pfns(struct hmm_range *range)struct mm_walk mm_walk;struct hmm *hmm; -/* FIXME support hugetlb fs */ -if (is_vm_hugetlb_page (vma) || (vma->vm_flags &VM_SPECIAL )) { -hmm_pfns_special (range->pfns, range->start, range->end); -return -EINVAL ; - } -/* Sanity check, this really should not happen ! */ if (range->start < vma->vm_start || range->start >= vma->vm_end)return -EINVAL ;
@@ -572,6 +566,12 @@ int hmm_vma_get_pfns(struct hmm_range *range)if (!hmm->mmu_notifier.ops)return -EINVAL ; +/* FIXME support hugetlb fs */ +if (is_vm_hugetlb_page (vma) || (vma->vm_flags &VM_SPECIAL )) { +hmm_pfns_special (range); +return -EINVAL ; + } +if (!(vma->vm_flags &VM_READ )) {/* * If vma do not allow read access, then assume that it does
@@ -740,6 +740,12 @@ int hmm_vma_fault(struct hmm_range *range, bool write, bool block)if (!hmm->mmu_notifier.ops)return -EINVAL ; +/* FIXME support hugetlb fs */ +if (is_vm_hugetlb_page (vma) || (vma->vm_flags &VM_SPECIAL )) { +hmm_pfns_special (range); +return -EINVAL ; + } +if (!(vma->vm_flags &VM_READ )) {/* * If vma do not allow read access, then assume that it does
@@ -751,12 +757,6 @@ int hmm_vma_fault(struct hmm_range *range, bool write, bool block)return -EPERM ; } -/* FIXME support hugetlb fs */ -if (is_vm_hugetlb_page (vma) || (vma->vm_flags &VM_SPECIAL )) { -hmm_pfns_special (range->pfns, range->start, range->end); -return 0 ; - } -/* Initialize range to track CPU page table update */ spin_lock (&hmm->lock); range->valid = true;
Signature · 1 line
2.14.3
PATCH 10/15 mm/hmm: do not differentiate between empty entry or missing directory v3
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
There is no point in differentiating between a range for which there is not even a directory (and thus entries) and empty entry (pte_none() or pmd_none() returns true).
Simply drop the distinction ie remove HMM_PFN_EMPTY flag and merge now duplicate hmm_vma_walk_hole() and hmm_vma_walk_clear() functions.
Changed since v1:
- Improved comments
Changed since v2:
- Typo in comments
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- include/linux/hmm.h | 8 +++----- mm/hmm.c | 45 +++++++++++++++------------------------------ 2 files changed, 18 insertions(+), 35 deletions(-)
include/linux/hmm.h +3 -5
@@ -84,7 +84,6 @@ struct hmm;* HMM_PFN_VALID: pfn is valid. It has, at least, read permission. * HMM_PFN_WRITE: CPU page table has write permission set * HMM_PFN_ERROR: corresponding CPU page table entry points to poisoned memory -* HMM_PFN_EMPTY: corresponding CPU page table entry is pte_none() * HMM_PFN_SPECIAL: corresponding CPU page table entry is special; i.e., the * result of vm_insert_pfn() or vm_insert_page(). Therefore, it should not * be mirrored by a device, because the entry will never have HMM_PFN_VALID
@@ -94,10 +93,9 @@ struct hmm;#define HMM_PFN_VALID (1 << 0)#define HMM_PFN_WRITE (1 << 1)#define HMM_PFN_ERROR (1 << 2) -#define HMM_PFN_EMPTY (1 << 3) -#define HMM_PFN_SPECIAL (1 << 4) -#define HMM_PFN_DEVICE_UNADDRESSABLE (1 << 5) -#define HMM_PFN_SHIFT 6 +#define HMM_PFN_SPECIAL (1 << 3) +#define HMM_PFN_DEVICE_UNADDRESSABLE (1 << 4) +#define HMM_PFN_SHIFT 5/* * hmm_pfn_to_page() - return struct page pointed to by a valid HMM pfn
mm/hmm.c +15 -30
77 lines not on this page · raw message
Signature · 1 line
2.14.3
PATCH 11/15 mm/hmm: rename HMM_PFN_DEVICE_UNADDRESSABLE to HMM_PFN_DEVICE_PRIVATE
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
Make naming consistent across code, DEVICE_PRIVATE is the name use outside HMM code so use that one.
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- include/linux/hmm.h | 4 ++-- mm/hmm.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
include/linux/hmm.h +2 -2
@@ -88,13 +88,13 @@ struct hmm;* result of vm_insert_pfn() or vm_insert_page(). Therefore, it should not * be mirrored by a device, because the entry will never have HMM_PFN_VALID * set and the pfn value is undefined. -* HMM_PFN_DEVICE_UNADDRESSABLE: unaddressable device memory (ZONE_DEVICE) +* HMM_PFN_DEVICE_PRIVATE: unaddressable device memory (ZONE_DEVICE) */ #define HMM_PFN_VALID (1 << 0)#define HMM_PFN_WRITE (1 << 1)#define HMM_PFN_ERROR (1 << 2)#define HMM_PFN_SPECIAL (1 << 3) -#define HMM_PFN_DEVICE_UNADDRESSABLE (1 << 4) +#define HMM_PFN_DEVICE_PRIVATE (1 << 4)#define HMM_PFN_SHIFT 5/*
mm/hmm.c +1 -1
@@ -472,7 +472,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, pfns[i] |=HMM_PFN_WRITE ; }else if (write_fault) goto fault; - pfns[i] |=HMM_PFN_DEVICE_UNADDRESSABLE ; + pfns[i] |=HMM_PFN_DEVICE_PRIVATE ; }else if (is_migration_entry (entry)) {if (hmm_vma_walk->fault) {pte_unmap (ptep);
Signature · 1 line
2.14.3
PATCH 12/15 mm/hmm: move hmm_pfns_clear() closer to where it is use
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
Move hmm_pfns_clear() closer to where it is use to make it clear it is not use by page table walkers.
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- mm/hmm.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
mm/hmm.c +8 -8
@@ -340,14 +340,6 @@ static int hmm_pfns_bad(unsigned long addr,return 0 ; } -static void hmm_pfns_clear (uint64_t *pfns, -unsigned long addr, -unsigned long end) -{ -for (; addr < end; addr +=PAGE_SIZE , pfns++) - *pfns =0 ; -} -/* * hmm_vma_walk_hole() - handle a range lacking valid pmd or pte(s) * @start: range virtual start address (inclusive)
@@ -506,6 +498,14 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,return 0 ; } +static void hmm_pfns_clear (uint64_t *pfns, +unsigned long addr, +unsigned long end) +{ +for (; addr < end; addr +=PAGE_SIZE , pfns++) + *pfns =0 ; +} +static void hmm_pfns_special (struct hmm_range *range) {unsigned long addr = range->start, i =0 ;
Signature · 1 line
2.14.3
PATCH 13/15 mm/hmm: factor out pte and pmd handling to simplify hmm_vma_walk_pmd() v2
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
No functional change, just create one function to handle pmd and one to handle pte (hmm_vma_handle_pmd() and hmm_vma_handle_pte()).
Changed since v1:
- s/pfns/pfn for pte as in that case we are dealing with a single pfn
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Reviewed-by John Hubbard <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
--- mm/hmm.c | 174 +++++++++++++++++++++++++++++++++++++-------------------------- 1 file changed, 102 insertions(+), 72 deletions(-)
mm/hmm.c +102 -72
@@ -375,6 +375,99 @@ static int hmm_vma_walk_hole(unsigned long addr,return hmm_vma_walk->fault ? -EAGAIN :0 ; } +static int hmm_vma_handle_pmd (struct mm_walk *walk, +unsigned long addr, +unsigned long end, +uint64_t *pfns, +pmd_t pmd) +{ +struct hmm_vma_walk *hmm_vma_walk = walk->private; +unsigned long pfn, i; +uint64_t flag =0 ; + +if (pmd_protnone (pmd)) +return hmm_vma_walk_hole (addr, end, walk); + +if ((hmm_vma_walk->fault & hmm_vma_walk->write) && !pmd_write (pmd)) +return hmm_vma_walk_hole (addr, end, walk); + + pfn =pmd_pfn (pmd) +pte_index (addr); + flag |=pmd_write (pmd) ?HMM_PFN_WRITE :0 ; +for (i =0 ; addr < end; addr +=PAGE_SIZE , i++, pfn++) + pfns[i] =hmm_pfn_from_pfn (pfn) | flag; + hmm_vma_walk->last = end; +return 0 ; +} + +static int hmm_vma_handle_pte (struct mm_walk *walk,unsigned long addr, +unsigned long end,pmd_t *pmdp,pte_t *ptep, +uint64_t *pfn) +{ +struct hmm_vma_walk *hmm_vma_walk = walk->private; +struct vm_area_struct *vma = walk->vma; +pte_t pte = *ptep; + + *pfn =0 ; + +if (pte_none (pte)) { + *pfn =0 ; +if (hmm_vma_walk->fault) + goto fault; +return 0 ; + } + +if (!pte_present (pte)) { +swp_entry_t entry =pte_to_swp_entry (pte); + +if (!non_swap_entry (entry)) { +if (hmm_vma_walk->fault) + goto fault; +return 0 ; + } + +/* +* This is a special swap entry, ignore migration, use +* device and report anything else as error. +*/ +if (is_device_private_entry (entry)) { + *pfn =hmm_pfn_from_pfn (swp_offset (entry)); +if (is_write_device_private_entry (entry)) { + *pfn |=HMM_PFN_WRITE ; + }else if ((hmm_vma_walk->fault & hmm_vma_walk->write)) + goto fault; + *pfn |=HMM_PFN_DEVICE_PRIVATE ; +return 0 ; + } + +if (is_migration_entry (entry)) { +if (hmm_vma_walk->fault) { +pte_unmap (ptep); + hmm_vma_walk->last = addr; +migration_entry_wait (vma->vm_mm, + pmdp, addr); +return -EAGAIN ; + } +return 0 ; + } + +/* Report error for everything else */ + *pfn =HMM_PFN_ERROR ; +return -EFAULT ; + } + +if ((hmm_vma_walk->fault & hmm_vma_walk->write) && !pte_write (pte)) + goto fault; + + *pfn =hmm_pfn_from_pfn (pte_pfn (pte)); + *pfn |=pte_write (pte) ?HMM_PFN_WRITE :0 ; +return 0 ; + +fault: +pte_unmap (ptep); +/* Fault any virtual address we were asked to fault */ +return hmm_vma_walk_hole (addr, end, walk); +} + static int hmm_vma_walk_pmd(pmd_t *pmdp, unsigned long start, unsigned long end,
@@ -382,25 +475,20 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, {struct hmm_vma_walk *hmm_vma_walk = walk->private;struct hmm_range *range = hmm_vma_walk->range; -struct vm_area_struct *vma = walk->vma;uint64_t *pfns = range->pfns;unsigned long addr = start, i; -bool write_fault;pte_t *ptep; i = (addr - range->start) >>PAGE_SHIFT ; - write_fault = hmm_vma_walk->fault & hmm_vma_walk->write; again:if (pmd_none (*pmdp))return hmm_vma_walk_hole (start, end, walk); -if (pmd_huge (*pmdp) && vma->vm_flags &VM_HUGETLB ) +if (pmd_huge (*pmdp) && (range->vma->vm_flags &VM_HUGETLB ))return hmm_pfns_bad (start, end, walk);if (pmd_devmap (*pmdp) ||pmd_trans_huge (*pmdp)) { -unsigned long pfn; -uint64_t flag =0 ;pmd_t pmd;/*
@@ -416,17 +504,8 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,barrier ();if (!pmd_devmap (pmd) && !pmd_trans_huge (pmd)) goto again; -if (pmd_protnone (pmd)) -return hmm_vma_walk_hole (start, end, walk); - -if (write_fault && !pmd_write (pmd)) -return hmm_vma_walk_hole (start, end, walk); - pfn =pmd_pfn (pmd) +pte_index (addr); - flag |=pmd_write (pmd) ?HMM_PFN_WRITE :0 ; -for (; addr < end; addr +=PAGE_SIZE , i++, pfn++) - pfns[i] =hmm_pfn_from_pfn (pfn) | flag; -return 0 ; +return hmm_vma_handle_pmd (walk, addr, end, &pfns[i], pmd); }if (pmd_bad (*pmdp))
@@ -434,67 +513,18 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, ptep =pte_offset_map (pmdp, addr);for (; addr < end; addr +=PAGE_SIZE , ptep++, i++) { -pte_t pte = *ptep; - - pfns[i] =0 ; - -if (pte_none (pte)) { - pfns[i] =0 ; -if (hmm_vma_walk->fault) - goto fault; -continue ; - } - -if (!pte_present (pte)) { -swp_entry_t entry =pte_to_swp_entry (pte); - -if (!non_swap_entry (entry)) { -if (hmm_vma_walk->fault) - goto fault; -continue ; - } +int r; -/* -* This is a special swap entry, ignore migration, use -* device and report anything else as error. -*/ -if (is_device_private_entry (entry)) { - pfns[i] =hmm_pfn_from_pfn (swp_offset (entry)); -if (is_write_device_private_entry (entry)) { - pfns[i] |=HMM_PFN_WRITE ; - }else if (write_fault) - goto fault; - pfns[i] |=HMM_PFN_DEVICE_PRIVATE ; - }else if (is_migration_entry (entry)) { -if (hmm_vma_walk->fault) { -pte_unmap (ptep); - hmm_vma_walk->last = addr; -migration_entry_wait (vma->vm_mm, - pmdp, addr); -return -EAGAIN ; - } -continue ; - }else { -/* Report error for everything else */ - pfns[i] =HMM_PFN_ERROR ; - } -continue ; + r =hmm_vma_handle_pte (walk, addr, end, pmdp, ptep, &pfns[i]); +if (r) { +/* hmm_vma_handle_pte() did unmap pte directory */ + hmm_vma_walk->last = addr; +return r; } - -if (write_fault && !pte_write (pte)) - goto fault; - - pfns[i] =hmm_pfn_from_pfn (pte_pfn (pte)); - pfns[i] |=pte_write (pte) ?HMM_PFN_WRITE :0 ; -continue ; - -fault: -pte_unmap (ptep); -/* Fault any virtual address we were asked to fault */ -return hmm_vma_walk_hole (start, end, walk); }pte_unmap (ptep -1 ); + hmm_vma_walk->last = addr;return 0 ; }
Signature · 1 line
2.14.3
PATCH 14/15 mm/hmm: change hmm_vma_fault() to allow write fault on page basis
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
This change hmm_vma_fault() to not take a global write fault flag for a range but instead rely on caller to populate HMM pfns array with proper fault flag ie HMM_PFN_VALID if driver want read fault for that address or HMM_PFN_VALID and HMM_PFN_WRITE for write.
Moreover by setting HMM_PFN_DEVICE_PRIVATE the device driver can ask for device private memory to be migrated back to system memory through page fault.
This is more flexible API and it better reflects how device handles and reports fault.
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
Cc John Hubbard <[email protected]>
--- include/linux/hmm.h | 2 +- mm/hmm.c | 151 ++++++++++++++++++++++++++++++++++++++++------------ 2 files changed, 119 insertions(+), 34 deletions(-)
include/linux/hmm.h +1 -1
@@ -317,7 +317,7 @@ bool hmm_vma_range_done(struct hmm_range *range);* * See the function description in mm/hmm.c for further documentation. */ -int hmm_vma_fault (struct hmm_range *range,bool write,bool block); +int hmm_vma_fault (struct hmm_range *range,bool block);#endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */
mm/hmm.c +118 -33
267 lines not on this page · raw message
Signature · 1 line
2.14.3
PATCH 15/15 mm/hmm: use device driver encoding for HMM pfn v2
In reply to [email protected]From: JA(C)rA'me Glisse <[email protected]>
User of hmm_vma_fault() and hmm_vma_get_pfns() provide a flags array and pfn shift value allowing them to define their own encoding for HMM pfn that are fill inside the pfns array of the hmm_range struct. With this device driver can get pfn that match their own private encoding out of HMM without having to do any conversion.
Changed since v1:
- Split flags and special values for clarification
- Improved comments and provide examples
Signed-off-by JA(C)rA'me Glisse <[email protected]>
Cc Evgeny Baskakov <[email protected]>
Cc Ralph Campbell <[email protected]>
Cc Mark Hairgrove <[email protected]>
Cc John Hubbard <[email protected]>
--- include/linux/hmm.h | 130 +++++++++++++++++++++++++++++++++++++--------------- mm/hmm.c | 84 ++++++++++++++++++--------------- 2 files changed, 142 insertions(+), 72 deletions(-)
include/linux/hmm.h +94 -36
@@ -80,68 +80,145 @@struct hmm ;/* +* hmm_pfn_flag_e - HMM flag enums +* * Flags: * HMM_PFN_VALID: pfn is valid. It has, at least, read permission. * HMM_PFN_WRITE: CPU page table has write permission set +* HMM_PFN_DEVICE_PRIVATE: private device memory (ZONE_DEVICE) +* +* The driver provide a flags array, if driver valid bit for an entry is bit +* 3 ie (entry & (1 << 3)) is true if entry is valid then driver must provide +* an array in hmm_range.flags with hmm_range.flags[HMM_PFN_VALID] == 1 << 3. +* Same logic apply to all flags. This is same idea as vm_page_prot in vma +* except that this is per device driver rather than per architecture. +*/ +enum hmm_pfn_flag_e { +HMM_PFN_VALID =0 , +HMM_PFN_WRITE , +HMM_PFN_DEVICE_PRIVATE , +HMM_PFN_FLAG_MAX +}; + +/* +* hmm_pfn_value_e - HMM pfn special value +* +* Flags: * HMM_PFN_ERROR: corresponding CPU page table entry points to poisoned memory +* HMM_PFN_NONE: corresponding CPU page table entry is pte_none() * HMM_PFN_SPECIAL: corresponding CPU page table entry is special; i.e., the * result of vm_insert_pfn() or vm_insert_page(). Therefore, it should not * be mirrored by a device, because the entry will never have HMM_PFN_VALID * set and the pfn value is undefined. -* HMM_PFN_DEVICE_PRIVATE: unaddressable device memory (ZONE_DEVICE) +* +* Driver provide entry value for none entry, error entry and special entry, +* driver can alias (ie use same value for error and special for instance). It +* should not alias none and error or special. +* +* HMM pfn value returned by hmm_vma_get_pfns() or hmm_vma_fault() will be: +* hmm_range.values[HMM_PFN_ERROR] if CPU page table entry is poisonous, +* hmm_range.values[HMM_PFN_NONE] if there is no CPU page table +* hmm_range.values[HMM_PFN_SPECIAL] if CPU page table entry is a special one */ -#define HMM_PFN_VALID (1 << 0) -#define HMM_PFN_WRITE (1 << 1) -#define HMM_PFN_ERROR (1 << 2) -#define HMM_PFN_SPECIAL (1 << 3) -#define HMM_PFN_DEVICE_PRIVATE (1 << 4) -#define HMM_PFN_SHIFT 5 +enum hmm_pfn_value_e { +HMM_PFN_ERROR , +HMM_PFN_NONE , +HMM_PFN_SPECIAL , +HMM_PFN_VALUE_MAX +}; + +/* +* struct hmm_range - track invalidation lock on virtual address range +* +* @vma: the vm area struct for the range +* @list: all range lock are on a list +* @start: range virtual start address (inclusive) +* @end: range virtual end address (exclusive) +* @pfns: array of pfns (big enough for the range) +* @flags: pfn flags to match device driver page table +* @values: pfn value for some special case (none, special, error, ...) +* @pfn_shifts: pfn shift value (should be <= PAGE_SHIFT) +* @valid: pfns array did not change since it has been fill by an HMM function +*/ +struct hmm_range { +struct vm_area_struct *vma; +struct list_head list; +unsigned long start; +unsigned long end; +uint64_t *pfns; +const uint64_t *flags; +const uint64_t *values; +uint8_t pfn_shift; +bool valid; +};/* * hmm_pfn_to_page() - return struct page pointed to by a valid HMM pfn +* @range: range use to decode HMM pfn value * @pfn: HMM pfn value to get corresponding struct page from * Returns: struct page pointer if pfn is a valid HMM pfn, NULL otherwise * * If the HMM pfn is valid (ie valid flag set) then return the struct page * matching the pfn value stored in the HMM pfn. Otherwise return NULL. */ -static inline struct page *hmm_pfn_to_page (uint64_t pfn) +static inline struct page *hmm_pfn_to_page (const struct hmm_range *range, +uint64_t pfn) { -if (!(pfn &HMM_PFN_VALID )) +if (pfn == range->values[HMM_PFN_NONE ]) +return NULL ; +if (pfn == range->values[HMM_PFN_ERROR ]) +return NULL ; +if (pfn == range->values[HMM_PFN_SPECIAL ])return NULL ; -return pfn_to_page (pfn >>HMM_PFN_SHIFT ); +if (!(pfn & range->flags[HMM_PFN_VALID ])) +return NULL ; +return pfn_to_page (pfn >> range->pfn_shift); }/* * hmm_pfn_to_pfn() - return pfn value store in a HMM pfn +* @range: range use to decode HMM pfn value * @pfn: HMM pfn value to extract pfn from * Returns: pfn value if HMM pfn is valid, -1UL otherwise */ -static inline unsigned long hmm_pfn_to_pfn (uint64_t pfn) +static inline unsigned long hmm_pfn_to_pfn (const struct hmm_range *range, +uint64_t pfn) { -if (!(pfn &HMM_PFN_VALID )) +if (pfn == range->values[HMM_PFN_NONE ]) +return -1UL ; +if (pfn == range->values[HMM_PFN_ERROR ]) +return -1UL ; +if (pfn == range->values[HMM_PFN_SPECIAL ])return -1UL ; -return (pfn >>HMM_PFN_SHIFT ); +if (!(pfn & range->flags[HMM_PFN_VALID ])) +return -1UL ; +return (pfn >> range->pfn_shift); }/* * hmm_pfn_from_page() - create a valid HMM pfn value from struct page +* @range: range use to encode HMM pfn value * @page: struct page pointer for which to create the HMM pfn * Returns: valid HMM pfn for the page */ -static inline uint64_t hmm_pfn_from_page (struct page *page) +static inline uint64_t hmm_pfn_from_page (const struct hmm_range *range, +struct page *page) { -return (page_to_pfn (page) <<HMM_PFN_SHIFT ) |HMM_PFN_VALID ; +return (page_to_pfn (page) << range->pfn_shift) | + range->flags[HMM_PFN_VALID ]; }/* * hmm_pfn_from_pfn() - create a valid HMM pfn value from pfn +* @range: range use to encode HMM pfn value * @pfn: pfn value for which to create the HMM pfn * Returns: valid HMM pfn for the pfn */ -static inline uint64_t hmm_pfn_from_pfn (unsigned long pfn) +static inline uint64_t hmm_pfn_from_pfn (const struct hmm_range *range, +unsigned long pfn) { -return (pfn <<HMM_PFN_SHIFT ) |HMM_PFN_VALID ; +return (pfn << range->pfn_shift) | + range->flags[HMM_PFN_VALID ]; }
@@ -263,25 +340,6 @@ int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm);void hmm_mirror_unregister (struct hmm_mirror *mirror); -/* -* struct hmm_range - track invalidation lock on virtual address range -* -* @vma: the vm area struct for the range -* @list: all range lock are on a list -* @start: range virtual start address (inclusive) -* @end: range virtual end address (exclusive) -* @pfns: array of pfns (big enough for the range) -* @valid: pfns array did not change since it has been fill by an HMM function -*/ -struct hmm_range { -struct vm_area_struct *vma; -struct list_head list; -unsigned long start; -unsigned long end; -uint64_t *pfns; -bool valid; -}; -/* * To snapshot the CPU page table, call hmm_vma_get_pfns(), then take a device * driver lock that serializes device page table updates, then call
mm/hmm.c +48 -36
229 lines not on this page · raw message
Signature · 1 line
2.14.3