powerpc/pseries: CMM: Implement balloon compaction and remove isolate notifier
Subsystems (6)
PATCH 1/12 powerpc/pseries: CMM: Implement release() function for sysfs device
In reply to David HildenbrandWhen unloading the module, one gets
[ 548.188594] ------------[ cut here ]------------ [ 548.188596] Device 'cmm0' does not have a release() function, it is broken and must be fixed. See Documentation/kobject.txt. [ 548.188622] WARNING: CPU: 0 PID: 19308 at drivers/base/core.c:1244 .device_release+0xcc/0xf0
...
We only have on static fake device. There is nothing to do when releasing the device (via cmm_exit).
--- arch/powerpc/platforms/pseries/cmm.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/powerpc/platforms/pseries/cmm.c +5 -0
@@ -411,6 +411,10 @@ static struct bus_type cmm_subsys = { .dev_name = "cmm", }; +static void cmm_release_device (struct device *dev) +{ +} +/** * cmm_sysfs_register - Register with sysfs *
@@ -426,6 +430,7 @@ static int cmm_sysfs_register(struct device *dev) dev->id =0 ; dev->bus = &cmm_subsys; + dev->release = cmm_release_device;if ((rc =device_register (dev))) goto subsys_unregister;
Signature · 1 line
2.21.0
PATCH 2/12 powerpc/pseries: CMM: Report errors when registering notifiers fails
In reply to David HildenbrandIf we don't set the rc, we will return "0", making it look like we succeeded.
--- arch/powerpc/platforms/pseries/cmm.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +6 -2
@@ -683,8 +683,12 @@ static int cmm_init(void)if ((rc =cmm_sysfs_register (&cmm_dev))) goto out_reboot_notifier; -if (register_memory_notifier (&cmm_mem_nb) || -register_memory_isolate_notifier (&cmm_mem_isolate_nb)) + rc =register_memory_notifier (&cmm_mem_nb); +if (rc) + goto out_unregister_notifier; + + rc =register_memory_isolate_notifier (&cmm_mem_isolate_nb); +if (rc) goto out_unregister_notifier;if (cmm_disabled)
Signature · 1 line
2.21.0
cover powerpc/pseries: CMM: Implement balloon compaction and remove isolate notifier
This is the follow-up of:
https://lkml.org/lkml/2019/9/10/711
We can get rid of the memory isolate notifier by switching to balloon compaction in powerpc's CMM (Collaborative Memory Management). The memory isolate notifier was only necessary to allow to offline memory blocks that contain inflated/"loaned" pages - which also possible when the inflated pages are movable (via balloon compaction). Having movable pages implies that memory will also no longer get fragmented when CMM is active.
While I do have access to a LPAR, it does not have CMM active and I have np clue how to enable it. Instead, I implemented a simple simulation mode. I did some tests and the whole infrastructure, including page migration, seems to work fine (e.g., I can still offline memory blocks that contain inflated pages). Of course, I cannot tell if HW will like the changes, especially:
1. I now use page_to_phys() to come up with the addresses to report to the hypervisor. Hope that's correct.
2. When migrating a page, I first inflate/"loan" the new page and then deflate the old page. I have no idea if HW accepts to set pages loaned if it didn't request a loan. I assume it does.
3. We might now inflate/deflate pages in parallel (of course, not the same page). I have no idea if HW likes that.
It would be good if somebody could either point me at the spec of the hypervisor interface or verify directly. Also, it would be good if somebody could test with actual HW that has this feature.
David Hildenbrand (12):
powerpc/pseries: CMM: Implement release() function for sysfs device powerpc/pseries: CMM: Report errors when registering notifiers fails powerpc/pseries: CMM: Cleanup rc handling in cmm_init() powerpc/pseries: CMM: Drop page array
powerpc/pseries: CMM: Use adjust_managed_page_count() insted of
totalram_pages_*
powerpc/pseries: CMM: Rip out memory isolate notifier powerpc/pseries: CMM: Convert loaned_pages to an atomic_long_t powerpc/pseries: CMM: Implement balloon compaction powerpc/pseries: CMM: Switch to balloon_page_alloc() powerpc/pseries: CMM: Simulation mode
mm: remove the memory isolate notifier
mm: remove "count" parameter from has_unmovable_pages()
arch/powerpc/platforms/pseries/Kconfig | 1 + arch/powerpc/platforms/pseries/cmm.c | 430 +++++++++++-------------- drivers/base/memory.c | 19 -- include/linux/memory.h | 27 -- include/linux/page-isolation.h | 4 +- include/uapi/linux/magic.h | 1 + mm/memory_hotplug.c | 2 +- mm/page_alloc.c | 21 +- mm/page_isolation.c | 27 +- 9 files changed, 204 insertions(+), 328 deletions(-)
Signature · 1 line
2.21.0
PATCH 3/12 powerpc/pseries: CMM: Cleanup rc handling in cmm_init()
No need to initialize rc. Also, let's return 0 directly when succeeding.
--- arch/powerpc/platforms/pseries/cmm.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +3 -4
@@ -669,7 +669,7 @@ static struct notifier_block cmm_mem_nb = {**/ static int cmm_init (void ) { -int rc = -ENOMEM ; +int rc;if (!firmware_has_feature (FW_FEATURE_CMO ))return -EOPNOTSUPP ;
@@ -692,7 +692,7 @@ static int cmm_init(void) goto out_unregister_notifier;if (cmm_disabled) -return rc; +return 0 ; cmm_thread_ptr =kthread_run (cmm_thread,NULL ,"cmmthread" );if (IS_ERR (cmm_thread_ptr)) {
@@ -700,8 +700,7 @@ static int cmm_init(void) goto out_unregister_notifier; } -return rc; - +return 0 ; out_unregister_notifier:unregister_memory_notifier (&cmm_mem_nb);unregister_memory_isolate_notifier (&cmm_mem_isolate_nb);
Signature · 1 line
2.21.0
PATCH 4/12 powerpc/pseries: CMM: Drop page array
In reply to David HildenbrandWe can simply store the pages in a list (page->lru), no need for a separate data structure (+ complicated handling). This is how most other balloon drivers store allocated pages without additional tracking data.
For the notifiers, use page_to_pfn() to check if a page is in the applicable range. Use page_to_phys() in plpar_page_set_loaned() and plpar_page_set_active() (I assume due to the __pa() that's the right thing to do).
--- arch/powerpc/platforms/pseries/cmm.c | 163 ++++++--------------------- 1 file changed, 36 insertions(+), 127 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +36 -127
@@ -75,21 +75,13 @@ module_param_named(debug, cmm_debug, uint, 0644); MODULE_PARM_DESC(debug, "Enable module debugging logging. Set to 1 to enable. " "[Default=" __stringify(CMM_DEBUG) "]"); -#define CMM_NR_PAGES ((PAGE_SIZE - sizeof(void *) - sizeof(unsigned long)) / sizeof(unsigned long)) -#define cmm_dbg (...) if (cmm_debug) { printk(KERN_INFO "cmm: "__VA_ARGS__); } -struct cmm_page_array { -struct cmm_page_array *next; -unsigned long index; -unsigned long page[CMM_NR_PAGES ]; -}; -static unsigned long loaned_pages;static unsigned long loaned_pages_target;static unsigned long oom_freed_pages; -static struct cmm_page_array *cmm_page_list; +static LIST_HEAD (cmm_page_list);static DEFINE_SPINLOCK (cmm_lock);static DEFINE_MUTEX (hotplug_mutex);
@@ -97,8 +89,9 @@ static int hotplug_occurred; /* protected by the hotplug mutex */static struct task_struct *cmm_thread_ptr; -static long plpar_page_set_loaned (unsigned long vpa) +static long plpar_page_set_loaned (struct page *page) { +const unsigned long vpa =page_to_phys (page);unsigned long cmo_page_sz =cmo_get_page_size ();long rc =0 ;int i;
@@ -113,8 +106,9 @@ static long plpar_page_set_loaned(unsigned long vpa)return rc; } -static long plpar_page_set_active (unsigned long vpa) +static long plpar_page_set_active (struct page *page) { +const unsigned long vpa =page_to_phys (page);unsigned long cmo_page_sz =cmo_get_page_size ();long rc =0 ;int i;
@@ -138,8 +132,7 @@ static long plpar_page_set_active(unsigned long vpa)**/ static long cmm_alloc_pages (long nr) { -struct cmm_page_array *pa, *npa; -unsigned long addr; +struct page *page;long rc;cmm_dbg ("Begin request for %ld pages\n" , nr);
@@ -156,43 +149,20 @@ static long cmm_alloc_pages(long nr)break ; } - addr =__get_free_page (GFP_NOIO | __GFP_NOWARN | - __GFP_NORETRY | __GFP_NOMEMALLOC); -if (!addr) + page =alloc_page (GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC); +if (!page)break ;spin_lock (&cmm_lock); - pa = cmm_page_list; -if (!pa || pa->index >=CMM_NR_PAGES ) { -/* Need a new page for the page list. */ -spin_unlock (&cmm_lock); - npa = (struct cmm_page_array *)__get_free_page ( -GFP_NOIO | __GFP_NOWARN | - __GFP_NORETRY | __GFP_NOMEMALLOC); -if (!npa) { -pr_info ("%s: Can not allocate new page list\n" , __func__); -free_page (addr); -break ; - } -spin_lock (&cmm_lock); - pa = cmm_page_list; - -if (!pa || pa->index >=CMM_NR_PAGES ) { - npa->next = pa; - npa->index =0 ; - pa = npa; - cmm_page_list = pa; - }else -free_page ((unsigned long ) npa); - } - -if ((rc =plpar_page_set_loaned (__pa (addr)))) { + rc =plpar_page_set_loaned (page); +if (rc) {pr_err ("%s: Can not set page to loaned. rc=%ld\n" , __func__, rc);spin_unlock (&cmm_lock); -free_page (addr); +__free_page (page);break ; } - pa->page[pa->index++] = addr; +list_add (&page->lru, &cmm_page_list); loaned_pages++;totalram_pages_dec ();spin_unlock (&cmm_lock);
@@ -212,25 +182,16 @@ static long cmm_alloc_pages(long nr)**/ static long cmm_free_pages (long nr) { -struct cmm_page_array *pa; -unsigned long addr; +struct page *page, *tmp;cmm_dbg ("Begin free of %ld pages.\n" , nr);spin_lock (&cmm_lock); - pa = cmm_page_list; -while (nr) { -if (!pa || pa->index <=0 ) +list_for_each_entry_safe (page, tmp, &cmm_page_list, lru) { +if (!nr)break ; - addr = pa->page[--pa->index]; - -if (pa->index ==0 ) { - pa = pa->next; -free_page ((unsigned long ) cmm_page_list); - cmm_page_list = pa; - } - -plpar_page_set_active (__pa (addr)); -free_page (addr); +plpar_page_set_active (page); +list_del (&page->lru); +__free_page (page); loaned_pages--; nr--;totalram_pages_inc ();
@@ -496,20 +457,13 @@ static struct notifier_block cmm_reboot_nb = {static unsigned long cmm_count_pages (void *arg) {struct memory_isolate_notify *marg = arg; -struct cmm_page_array *pa; -unsigned long start = (unsigned long )pfn_to_kaddr (marg->start_pfn); -unsigned long end = start + (marg->nr_pages <<PAGE_SHIFT ); -unsigned long idx; +struct page *page;spin_lock (&cmm_lock); - pa = cmm_page_list; -while (pa) { -if ((unsigned long )pa >= start && (unsigned long )pa < end) +list_for_each_entry (page, &cmm_page_list, lru) { +if (page_to_pfn (page) >= marg->start_pfn && +page_to_pfn (page) < marg->start_pfn + marg->nr_pages) marg->pages_found++; -for (idx =0 ; idx < pa->index; idx++) -if (pa->page[idx] >= start && pa->page[idx] < end) - marg->pages_found++; - pa = pa->next; }spin_unlock (&cmm_lock);return 0 ;
@@ -550,69 +504,24 @@ static struct notifier_block cmm_mem_isolate_nb = {static int cmm_mem_going_offline (void *arg) {struct memory_notify *marg = arg; -unsigned long start_page = (unsigned long )pfn_to_kaddr (marg->start_pfn); -unsigned long end_page = start_page + (marg->nr_pages <<PAGE_SHIFT ); -struct cmm_page_array *pa_curr, *pa_last, *npa; -unsigned long idx; +struct page *page, *tmp;unsigned long freed =0 ; -cmm_dbg ("Memory going offline, searching 0x%lx (%ld pages).\n" , - start_page, marg->nr_pages); +cmm_dbg ("Memory going offline, searching PFN 0x%lx (%ld pages).\n" , + marg->start_pfn, marg->nr_pages);spin_lock (&cmm_lock);/* Search the page list for pages in the range to be offlined */ - pa_last = pa_curr = cmm_page_list; -while (pa_curr) { -for (idx = (pa_curr->index -1 ); (idx +1 ) >0 ; idx--) { -if ((pa_curr->page[idx] < start_page) || - (pa_curr->page[idx] >= end_page)) -continue ; - -plpar_page_set_active (__pa (pa_curr->page[idx])); -free_page (pa_curr->page[idx]); - freed++; - loaned_pages--; -totalram_pages_inc (); - pa_curr->page[idx] = pa_last->page[--pa_last->index]; -if (pa_last->index ==0 ) { -if (pa_curr == pa_last) - pa_curr = pa_last->next; - pa_last = pa_last->next; -free_page ((unsigned long )cmm_page_list); - cmm_page_list = pa_last; - } - } - pa_curr = pa_curr->next; - } - -/* Search for page list structures in the range to be offlined */ - pa_last =NULL ; - pa_curr = cmm_page_list; -while (pa_curr) { -if (((unsigned long )pa_curr >= start_page) && - ((unsigned long )pa_curr < end_page)) { - npa = (struct cmm_page_array *)__get_free_page ( -GFP_NOIO | __GFP_NOWARN | - __GFP_NORETRY | __GFP_NOMEMALLOC); -if (!npa) { -spin_unlock (&cmm_lock); -cmm_dbg ("Failed to allocate memory for list " -"management. Memory hotplug " -"failed.\n" ); -return -ENOMEM ; - } -memcpy (npa, pa_curr,PAGE_SIZE ); -if (pa_curr == cmm_page_list) - cmm_page_list = npa; -if (pa_last) - pa_last->next = npa; -free_page ((unsigned long ) pa_curr); - freed++; - pa_curr = npa; - } - - pa_last = pa_curr; - pa_curr = pa_curr->next; +list_for_each_entry_safe (page, tmp, &cmm_page_list, lru) { +if (page_to_pfn (page) < marg->start_pfn || +page_to_pfn (page) >= marg->start_pfn + marg->nr_pages) +continue ; +plpar_page_set_active (page); +list_del (&page->lru); +__free_page (page); + freed++; + loaned_pages--; +totalram_pages_inc (); }spin_unlock (&cmm_lock);
Signature · 1 line
2.21.0
PATCH 5/12 powerpc/pseries: CMM: Use adjust_managed_page_count() insted of totalram_pages_*
In reply to David Hildenbrandadjust_managed_page_count() performs a totalram_pages_add(), but also adjust the managed pages of the zone. Let's use that instead, similar to virtio-balloon. Use it before freeing a page.
--- arch/powerpc/platforms/pseries/cmm.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +3 -3
@@ -164,7 +164,7 @@ static long cmm_alloc_pages(long nr)list_add (&page->lru, &cmm_page_list); loaned_pages++; -totalram_pages_dec (); +adjust_managed_page_count (page,-1 );spin_unlock (&cmm_lock); nr--; }
@@ -191,10 +191,10 @@ static long cmm_free_pages(long nr)break ;plpar_page_set_active (page);list_del (&page->lru); +adjust_managed_page_count (page,1 );__free_page (page); loaned_pages--; nr--; -totalram_pages_inc (); }spin_unlock (&cmm_lock);cmm_dbg ("End request with %ld pages unfulfilled\n" , nr);
@@ -518,10 +518,10 @@ static int cmm_mem_going_offline(void *arg)continue ;plpar_page_set_active (page);list_del (&page->lru); +adjust_managed_page_count (page,1 );__free_page (page); freed++; loaned_pages--; -totalram_pages_inc (); }spin_unlock (&cmm_lock);
Signature · 1 line
2.21.0
PATCH 6/12 powerpc/pseries: CMM: Rip out memory isolate notifier
In reply to David HildenbrandThe memory isolate notifier was added to allow to offline memory blocks that contain inflated/"loaned" pages. We can achieve the same using the balloon compaction framework.
Get rid of the memory isolate notifier. Also, we can get rid of cmm_mem_going_offline(), as we will never reach that code path now when we have allocated memory in the balloon (allocated pages are unmovable and will no longer be special-cased using the memory isolation notifier).
Leave the memory notifier in place, so we can still back off in case memory gets offlined.
--- arch/powerpc/platforms/pseries/cmm.c | 97 +--------------------------- 1 file changed, 1 insertion(+), 96 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +1 -96
@@ -38,12 +38,8 @@#define CMM_MIN_MEM_MB 256#define KB2PAGES (_p) ((_p)>>(PAGE_SHIFT-10))#define PAGES2KB (_p) ((_p)<<(PAGE_SHIFT-10)) -/* -* The priority level tries to ensure that this notifier is called as -* late as possible to reduce thrashing in the shared memory pool. -*/ +#define CMM_MEM_HOTPLUG_PRI 1 -#define CMM_MEM_ISOLATE_PRI 15static unsigned int delay =CMM_DEFAULT_DELAY ;static unsigned int hotplug_delay =CMM_HOTPLUG_DELAY ;
@@ -446,90 +442,6 @@ static struct notifier_block cmm_reboot_nb = { .notifier_call = cmm_reboot_notifier, }; -/** -* cmm_count_pages - Count the number of pages loaned in a particular range. -* -* @arg: memory_isolate_notify structure with address range and count -* -* Return value: -* 0 on success -**/ -static unsigned long cmm_count_pages (void *arg) -{ -struct memory_isolate_notify *marg = arg; -struct page *page; - -spin_lock (&cmm_lock); -list_for_each_entry (page, &cmm_page_list, lru) { -if (page_to_pfn (page) >= marg->start_pfn && -page_to_pfn (page) < marg->start_pfn + marg->nr_pages) - marg->pages_found++; - } -spin_unlock (&cmm_lock); -return 0 ; -} - -/** -* cmm_memory_isolate_cb - Handle memory isolation notifier calls -* @self: notifier block struct -* @action: action to take -* @arg: struct memory_isolate_notify data for handler -* -* Return value: -* NOTIFY_OK or notifier error based on subfunction return value -**/ -static int cmm_memory_isolate_cb (struct notifier_block *self, -unsigned long action,void *arg) -{ -int ret =0 ; - -if (action ==MEM_ISOLATE_COUNT ) - ret =cmm_count_pages (arg); - -return notifier_from_errno (ret); -} - -static struct notifier_block cmm_mem_isolate_nb = { - .notifier_call = cmm_memory_isolate_cb, - .priority =CMM_MEM_ISOLATE_PRI -}; - -/** -* cmm_mem_going_offline - Unloan pages where memory is to be removed -* @arg: memory_notify structure with page range to be offlined -* -* Return value: -* 0 on success -**/ -static int cmm_mem_going_offline (void *arg) -{ -struct memory_notify *marg = arg; -struct page *page, *tmp; -unsigned long freed =0 ; - -cmm_dbg ("Memory going offline, searching PFN 0x%lx (%ld pages).\n" , - marg->start_pfn, marg->nr_pages); -spin_lock (&cmm_lock); - -/* Search the page list for pages in the range to be offlined */ -list_for_each_entry_safe (page, tmp, &cmm_page_list, lru) { -if (page_to_pfn (page) < marg->start_pfn || -page_to_pfn (page) >= marg->start_pfn + marg->nr_pages) -continue ; -plpar_page_set_active (page); -list_del (&page->lru); -adjust_managed_page_count (page,1 ); -__free_page (page); - freed++; - loaned_pages--; - } - -spin_unlock (&cmm_lock); -cmm_dbg ("Released %ld pages in the search range.\n" , freed); - -return 0 ; -} -/** * cmm_memory_cb - Handle memory hotplug notifier calls * @self: notifier block struct
@@ -549,7 +461,6 @@ static int cmm_memory_cb(struct notifier_block *self,case MEM_GOING_OFFLINE :mutex_lock (&hotplug_mutex); hotplug_occurred =1 ; - ret =cmm_mem_going_offline (arg);break ;case MEM_OFFLINE :case MEM_CANCEL_OFFLINE :
@@ -596,10 +507,6 @@ static int cmm_init(void)if (rc) goto out_unregister_notifier; - rc =register_memory_isolate_notifier (&cmm_mem_isolate_nb); -if (rc) - goto out_unregister_notifier; -if (cmm_disabled)return 0 ;
@@ -612,7 +519,6 @@ static int cmm_init(void)return 0 ; out_unregister_notifier:unregister_memory_notifier (&cmm_mem_nb); -unregister_memory_isolate_notifier (&cmm_mem_isolate_nb);cmm_unregister_sysfs (&cmm_dev); out_reboot_notifier:unregister_reboot_notifier (&cmm_reboot_nb);
@@ -634,7 +540,6 @@ static void cmm_exit(void)unregister_oom_notifier (&cmm_oom_nb);unregister_reboot_notifier (&cmm_reboot_nb);unregister_memory_notifier (&cmm_mem_nb); -unregister_memory_isolate_notifier (&cmm_mem_isolate_nb);cmm_free_pages (loaned_pages);cmm_unregister_sysfs (&cmm_dev); }
Signature · 1 line
2.21.0
PATCH 7/12 powerpc/pseries: CMM: Convert loaned_pages to an atomic_long_t
In reply to David HildenbrandWhen switching to balloon compaction, we want to drop the cmm_lock and completely rely on the balloon compaction list lock internally. loaned_pages is currently protected under the cmm_lock.
Note: Right now cmm_alloc_pages() and cmm_free_pages() can be called at the same time, e.g., via the thread and a concurrent OOM notifier.
--- arch/powerpc/platforms/pseries/cmm.c | 35 +++++++++++++++------------- 1 file changed, 19 insertions(+), 16 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +19 -16
@@ -73,7 +73,7 @@ MODULE_PARM_DESC(debug, "Enable module debugging logging. Set to 1 to enable. "#define cmm_dbg (...) if (cmm_debug) { printk(KERN_INFO "cmm: "__VA_ARGS__); } -static unsigned long loaned_pages; +static atomic_long_t loaned_pages;static unsigned long loaned_pages_target;static unsigned long oom_freed_pages;
@@ -159,7 +159,7 @@ static long cmm_alloc_pages(long nr) }list_add (&page->lru, &cmm_page_list); - loaned_pages++; +atomic_long_inc (&loaned_pages);adjust_managed_page_count (page,-1 );spin_unlock (&cmm_lock); nr--;
@@ -189,7 +189,7 @@ static long cmm_free_pages(long nr)list_del (&page->lru);adjust_managed_page_count (page,1 );__free_page (page); - loaned_pages--; +atomic_long_dec (&loaned_pages); nr--; }spin_unlock (&cmm_lock);
@@ -214,7 +214,7 @@ static int cmm_oom_notify(struct notifier_block *self,cmm_dbg ("OOM processing started\n" ); nr =cmm_free_pages (nr); - loaned_pages_target = loaned_pages; + loaned_pages_target =atomic_long_read (&loaned_pages); *freed +=KB2PAGES (oom_kb) - nr; oom_freed_pages +=KB2PAGES (oom_kb) - nr;cmm_dbg ("OOM processing complete\n" );
@@ -231,10 +231,11 @@ static int cmm_oom_notify(struct notifier_block *self,**/ static void cmm_get_mpp (void ) { +const long __loaned_pages =atomic_long_read (&loaned_pages); +const long total_pages =totalram_pages () + __loaned_pages;int rc;struct hvcall_mpp_data mpp_data;signed long active_pages_target, page_loan_request, target; -signed long total_pages =totalram_pages () + loaned_pages;signed long min_mem_pages = (min_mem_mb *1024 *1024 ) /PAGE_SIZE ; rc =h_get_mpp (&mpp_data);
@@ -243,7 +244,7 @@ static void cmm_get_mpp(void)return ; page_loan_request =div_s64 ((s64 )mpp_data.loan_request,PAGE_SIZE ); - target = page_loan_request + (signed long )loaned_pages; + target = page_loan_request + __loaned_pages;if (target <0 || total_pages < min_mem_pages) target =0 ;
@@ -264,7 +265,7 @@ static void cmm_get_mpp(void) loaned_pages_target = target;cmm_dbg ("delta = %ld, loaned = %lu, target = %lu, oom = %lu, totalram = %lu\n" , - page_loan_request, loaned_pages, loaned_pages_target, + page_loan_request, __loaned_pages, loaned_pages_target, oom_freed_pages,totalram_pages ()); }
@@ -282,6 +283,7 @@ static struct notifier_block cmm_oom_nb = {static int cmm_thread (void *dummy) {unsigned long timeleft; +long __loaned_pages;while (1 ) { timeleft =msleep_interruptible (delay *1000 );
@@ -312,11 +314,12 @@ static int cmm_thread(void *dummy)cmm_get_mpp (); -if (loaned_pages_target > loaned_pages) { -if (cmm_alloc_pages (loaned_pages_target - loaned_pages)) - loaned_pages_target = loaned_pages; - }else if (loaned_pages_target < loaned_pages) -cmm_free_pages (loaned_pages - loaned_pages_target); + __loaned_pages =atomic_long_read (&loaned_pages); +if (loaned_pages_target > __loaned_pages) { +if (cmm_alloc_pages (loaned_pages_target - __loaned_pages)) + loaned_pages_target = __loaned_pages; + }else if (loaned_pages_target < __loaned_pages) +cmm_free_pages (__loaned_pages - loaned_pages_target); }return 0 ; }
@@ -330,7 +333,7 @@ static int cmm_thread(void *dummy) } \ static DEVICE_ATTR(name, 0444, show_##name, NULL) -CMM_SHOW (loaned_kb,"%lu\n" ,PAGES2KB (loaned_pages)); +CMM_SHOW (loaned_kb,"%lu\n" ,PAGES2KB (atomic_long_read (&loaned_pages)));CMM_SHOW (loaned_target_kb,"%lu\n" ,PAGES2KB (loaned_pages_target)); static ssize_t show_oom_pages(struct device *dev,
@@ -433,7 +436,7 @@ static int cmm_reboot_notifier(struct notifier_block *nb,if (cmm_thread_ptr)kthread_stop (cmm_thread_ptr); cmm_thread_ptr =NULL ; -cmm_free_pages (loaned_pages); +cmm_free_pages (atomic_long_read (&loaned_pages)); }return NOTIFY_DONE ; }
@@ -540,7 +543,7 @@ static void cmm_exit(void)unregister_oom_notifier (&cmm_oom_nb);unregister_reboot_notifier (&cmm_reboot_nb);unregister_memory_notifier (&cmm_mem_nb); -cmm_free_pages (loaned_pages); +cmm_free_pages (atomic_long_read (&loaned_pages));cmm_unregister_sysfs (&cmm_dev); }
@@ -561,7 +564,7 @@ static int cmm_set_disable(const char *val, const struct kernel_param *kp)if (cmm_thread_ptr)kthread_stop (cmm_thread_ptr); cmm_thread_ptr =NULL ; -cmm_free_pages (loaned_pages); +cmm_free_pages (atomic_long_read (&loaned_pages)); }else if (!disable && cmm_disabled) { cmm_thread_ptr =kthread_run (cmm_thread,NULL ,"cmmthread" );if (IS_ERR (cmm_thread_ptr))
Signature · 1 line
2.21.0
PATCH 9/12 powerpc/pseries: CMM: Switch to balloon_page_alloc()
In reply to David Hildenbrandballoon_page_alloc() will use GFP_HIGHUSER_MOVABLE in case we have CONFIG_BALLOON_COMPACTION. This is now possible, as balloon pages are movable with CONFIG_BALLOON_COMPACTION. Without CONFIG_BALLOON_COMPACTION, GFP_HIGHUSER is used.
Note that apart from that, balloon_page_alloc() uses the following flags:
__GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN
And current code used:
GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC
GFP_HIGHUSER/GFP_HIGHUSER_MOVABLE include
__GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_HARDWALL | __GFP_HIGHMEM
GFP_NOIO is __GFP_RECLAIM.
With CONFIG_BALLOON_COMPACTION, we essentially add:
__GFP_IO | __GFP_FS | __GFP_HARDWALL | __GFP_HIGHMEM | __GFP_MOVABLE
Without CONFIG_BALLOON_COMPACTION, we essentially add:
__GFP_IO | __GFP_FS | __GFP_HARDWALL | __GFP_HIGHMEM
I assume this is fine, as this is what all other balloon compaction users use. If it turns out to be a problem, we could add __GFP_MOVABLE manually if we have CONFIG_BALLOON_COMPACTION.
--- arch/powerpc/platforms/pseries/cmm.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +1 -2
@@ -147,8 +147,7 @@ static long cmm_alloc_pages(long nr)break ; } - page =alloc_page (GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NOMEMALLOC); + page =balloon_page_alloc ();if (!page)break ; rc =plpar_page_set_loaned (page);
Signature · 1 line
2.21.0
PATCH 11/12 mm: remove the memory isolate notifier
In reply to David HildenbrandLuckily, we have no users left, so we can get rid of it.
--- drivers/base/memory.c | 19 ------------------- include/linux/memory.h | 27 --------------------------- mm/page_isolation.c | 27 ++------------------------- 3 files changed, 2 insertions(+), 71 deletions(-)
drivers/base/memory.c +0 -19
@@ -73,20 +73,6 @@ void unregister_memory_notifier(struct notifier_block *nb) }EXPORT_SYMBOL (unregister_memory_notifier); -static ATOMIC_NOTIFIER_HEAD (memory_isolate_chain); - -int register_memory_isolate_notifier (struct notifier_block *nb) -{ -return atomic_notifier_chain_register (&memory_isolate_chain, nb); -} -EXPORT_SYMBOL (register_memory_isolate_notifier); - -void unregister_memory_isolate_notifier (struct notifier_block *nb) -{ -atomic_notifier_chain_unregister (&memory_isolate_chain, nb); -} -EXPORT_SYMBOL (unregister_memory_isolate_notifier); -static void memory_block_release (struct device *dev) {struct memory_block *mem =to_memory_block (dev);
@@ -178,11 +164,6 @@ int memory_notify(unsigned long val, void *v)return blocking_notifier_call_chain (&memory_chain, val, v); } -int memory_isolate_notify (unsigned long val,void *v) -{ -return atomic_notifier_call_chain (&memory_isolate_chain, val, v); -} -/* * The probe routines leave the pages uninitialized, just as the bootmem code * does. Make sure we do not access them, but instead use only information from
include/linux/memory.h +0 -27
@@ -55,19 +55,6 @@ struct memory_notify {int status_change_nid; }; -/* -* During pageblock isolation, count the number of pages within the -* range [start_pfn, start_pfn + nr_pages) which are owned by code -* in the notifier chain. -*/ -#define MEM_ISOLATE_COUNT (1<<0) - -struct memory_isolate_notify { -unsigned long start_pfn;/* Start of range to check */ -unsigned int nr_pages;/* # pages in range to check */ -unsigned int pages_found;/* # pages owned found by callbacks */ -}; -struct notifier_block ;struct mem_section ;
@@ -94,27 +81,13 @@ static inline int memory_notify(unsigned long val, void *v) {return 0 ; } -static inline int register_memory_isolate_notifier (struct notifier_block *nb) -{ -return 0 ; -} -static inline void unregister_memory_isolate_notifier (struct notifier_block *nb) -{ -} -static inline int memory_isolate_notify (unsigned long val,void *v) -{ -return 0 ; -}#else extern int register_memory_notifier (struct notifier_block *nb);extern void unregister_memory_notifier (struct notifier_block *nb); -extern int register_memory_isolate_notifier (struct notifier_block *nb); -extern void unregister_memory_isolate_notifier (struct notifier_block *nb);int create_memory_block_devices (unsigned long start,unsigned long size);void remove_memory_block_devices (unsigned long start,unsigned long size);extern void memory_dev_init (void );extern int memory_notify (unsigned long val,void *v); -extern int memory_isolate_notify (unsigned long val,void *v);extern struct memory_block *find_memory_block (struct mem_section *);typedef int (*walk_memory_blocks_func_t )(struct memory_block *,void *); extern int walk_memory_blocks(unsigned long start, unsigned long size,
mm/page_isolation.c +2 -25
@@ -18,9 +18,7 @@static int set_migratetype_isolate (struct page *page,int migratetype,int isol_flags) {struct zone *zone; -unsigned long flags, pfn; -struct memory_isolate_notify arg; -int notifier_ret; +unsigned long flags;int ret = -EBUSY ; zone =page_zone (page);
@@ -35,32 +33,11 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_if (is_migrate_isolate_page (page)) goto out; - pfn =page_to_pfn (page); - arg.start_pfn = pfn; - arg.nr_pages = pageblock_nr_pages; - arg.pages_found =0 ; - -/* -* It may be possible to isolate a pageblock even if the -* migratetype is not MIGRATE_MOVABLE. The memory isolation -* notifier chain is used by balloon drivers to return the -* number of pages in a range that are held by the balloon -* driver to shrink memory. If all the pages are accounted for -* by balloons, are free, or on the LRU, isolation can continue. -* Later, for example, when memory hotplug notifier runs, these -* pages reported as "can be isolated" should be isolated(freed) -* by the balloon driver through the memory notifier chain. -*/ - notifier_ret =memory_isolate_notify (MEM_ISOLATE_COUNT , &arg); - notifier_ret =notifier_to_errno (notifier_ret); -if (notifier_ret) - goto out;/* * FIXME: Now, memory hotplug doesn't call shrink_slab() by itself. * We just check MOVABLE pages. */ -if (!has_unmovable_pages (zone, page, arg.pages_found, migratetype, - isol_flags)) +if (!has_unmovable_pages (zone, page,0 , migratetype, isol_flags)) ret =0 ;/*
Signature · 1 line
2.21.0
PATCH 10/12 powerpc/pseries: CMM: Simulation mode
In reply to David HildenbrandLet's allow to test the implementation without needing HW support. When "simulate=1" is specified when loading the module, we bypass all HW checks and HW calls. The sysfs file "simulate_loan_target_kb" can be used to simulate HW requests.
The simualtion mode can be activated using
modprobe cmm debug=1 simulate=1
And the requested loan target can be changed using
echo X > /sys/devices/system/cmm/cmm0/simulate_loan_target_kb
--- arch/powerpc/platforms/pseries/cmm.c | 38 ++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 8 deletions(-)
arch/powerpc/platforms/pseries/cmm.c +30 -8
@@ -51,6 +51,8 @@ static unsigned int oom_kb = CMM_OOM_KB;static unsigned int cmm_debug =CMM_DEBUG ;static unsigned int cmm_disabled =CMM_DISABLE ;static unsigned long min_mem_mb =CMM_MIN_MEM_MB ; +static bool __read_mostly simulate; +static unsigned long simulate_loan_target_kb;static struct device cmm_dev;MODULE_AUTHOR ("Brian King <[email protected]>" );
@@ -74,6 +76,8 @@ MODULE_PARM_DESC(min_mem_mb, "Minimum amount of memory (in MB) to not balloon. "module_param_named (debug, cmm_debug, uint,0644 ); MODULE_PARM_DESC(debug, "Enable module debugging logging. Set to 1 to enable. " "[Default=" __stringify(CMM_DEBUG) "]"); +module_param_named (simulate, simulate, bool,0444 ); +MODULE_PARM_DESC (simulate,"Enable simulation mode (no communication with hw)." );#define cmm_dbg (...) if (cmm_debug) { printk(KERN_INFO "cmm: "__VA_ARGS__); }
@@ -94,6 +98,9 @@ static long plpar_page_set_loaned(struct page *page)long rc =0 ;int i; +if (unlikely (simulate)) +return 0 ; +for (i =0 ; !rc && i <PAGE_SIZE ; i += cmo_page_sz) rc =plpar_hcall_norets (H_PAGE_INIT ,H_PAGE_SET_LOANED , vpa + i,0 );
@@ -111,6 +118,9 @@ static long plpar_page_set_active(struct page *page)long rc =0 ;int i; +if (unlikely (simulate)) +return 0 ; +for (i =0 ; !rc && i <PAGE_SIZE ; i += cmo_page_sz) rc =plpar_hcall_norets (H_PAGE_INIT ,H_PAGE_SET_ACTIVE , vpa + i,0 );
@@ -234,13 +244,17 @@ static void cmm_get_mpp(void)signed long active_pages_target, page_loan_request, target;signed long min_mem_pages = (min_mem_mb *1024 *1024 ) /PAGE_SIZE ; - rc =h_get_mpp (&mpp_data); - -if (rc !=H_SUCCESS ) -return ; - - page_loan_request =div_s64 ((s64 )mpp_data.loan_request,PAGE_SIZE ); - target = page_loan_request + __loaned_pages; +if (likely (!simulate)) { + rc =h_get_mpp (&mpp_data); +if (rc !=H_SUCCESS ) +return ; + page_loan_request =div_s64 ((s64 )mpp_data.loan_request, +PAGE_SIZE ); + target = page_loan_request + __loaned_pages; + }else { + target =KB2PAGES (simulate_loan_target_kb); + page_loan_request = target - __loaned_pages; + }if (target <0 || total_pages < min_mem_pages) target =0 ;
@@ -362,6 +376,9 @@ static struct device_attribute *cmm_attrs[] = { &dev_attr_oom_freed_kb, }; +static DEVICE_ULONG_ATTR(simulate_loan_target_kb, 0644, + simulate_loan_target_kb); +static struct bus_type cmm_subsys = { .name ="cmm" , .dev_name ="cmm" ,
@@ -396,6 +413,11 @@ static int cmm_sysfs_register(struct device *dev) goto fail; } +if (!simulate) +return 0 ; + rc =device_create_file (dev, &dev_attr_simulate_loan_target_kb.attr); +if (rc) + goto fail;return 0 ; fail:
@@ -589,7 +611,7 @@ static int cmm_init(void) {int rc; -if (!firmware_has_feature (FW_FEATURE_CMO )) +if (!firmware_has_feature (FW_FEATURE_CMO ) && !simulate)return -EOPNOTSUPP ; rc =cmm_balloon_compaction_init ();
Signature · 1 line
2.21.0
PATCH 12/12 mm: remove "count" parameter from has_unmovable_pages()
In reply to David HildenbrandNow that the memory isolate notifier is gone, the parameter is always 0. Drop it and cleanup has_unmovable_pages().
--- include/linux/page-isolation.h | 4 ++-- mm/memory_hotplug.c | 2 +- mm/page_alloc.c | 21 +++++++-------------- mm/page_isolation.c | 2 +- 4 files changed, 11 insertions(+), 18 deletions(-)
include/linux/page-isolation.h +2 -2
@@ -33,8 +33,8 @@ static inline bool is_migrate_isolate(int migratetype)#define MEMORY_OFFLINE 0x1#define REPORT_FAILURE 0x2 -bool has_unmovable_pages (struct zone *zone,struct page *page,int count, -int migratetype,int flags); +bool has_unmovable_pages (struct zone *zone,struct page *page,int migratetype, +int flags);void set_pageblock_migratetype (struct page *page,int migratetype);int move_freepages_block (struct zone *zone,struct page *page,int migratetype,int *num_movable);
mm/memory_hotplug.c +1 -1
@@ -1149,7 +1149,7 @@ static bool is_pageblock_removable_nolock(unsigned long pfn)if (!zone_spans_pfn (zone, pfn))return false; -return !has_unmovable_pages (zone, page,0 ,MIGRATE_MOVABLE , +return !has_unmovable_pages (zone, page,MIGRATE_MOVABLE ,MEMORY_OFFLINE ); }
mm/page_alloc.c +7 -14
@@ -8182,17 +8182,15 @@ void *__init alloc_large_system_hash(const char *tablename,/* * This function checks whether pageblock includes unmovable pages or not. -* If @count is not zero, it is okay to include less @count unmovable pages * * PageLRU check without isolation or lru_lock could race so that * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable * check without lock_page also may miss some movable non-lru pages at * race condition. So you can't expect this function should be exact. */ -bool has_unmovable_pages (struct zone *zone,struct page *page,int count, -int migratetype,int flags) +bool has_unmovable_pages (struct zone *zone,struct page *page,int migratetype, +int flags) { -unsigned long found;unsigned long iter =0 ;unsigned long pfn =page_to_pfn (page);const char *reason ="unmovable page" ;
@@ -8218,13 +8216,11 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, goto unmovable; } -for (found =0 ; iter < pageblock_nr_pages; iter++) { -unsigned long check = pfn + iter; - -if (!pfn_valid_within (check)) +for (; iter < pageblock_nr_pages; iter++) { +if (!pfn_valid_within (pfn + iter))continue ; - page =pfn_to_page (check); + page =pfn_to_page (pfn + iter);if (PageReserved (page)) goto unmovable;
@@ -8273,11 +8269,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count,if ((flags &MEMORY_OFFLINE ) &&PageHWPoison (page))continue ; -if (__PageMovable (page)) +if (__PageMovable (page) ||PageLRU (page))continue ; -if (!PageLRU (page)) - found++;/* * If there are RECLAIMABLE pages, we need to check * it. But now, memory offline itself doesn't call
@@ -8291,8 +8285,7 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count,* is set to both of a memory hole page and a _used_ kernel * page at boot. */ -if (found > count) - goto unmovable; + goto unmovable; }return false; unmovable:
mm/page_isolation.c +1 -1
@@ -37,7 +37,7 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_* FIXME: Now, memory hotplug doesn't call shrink_slab() by itself. * We just check MOVABLE pages. */ -if (!has_unmovable_pages (zone, page,0 , migratetype, isol_flags)) +if (!has_unmovable_pages (zone, page, migratetype, isol_flags)) ret =0 ;/*
Signature · 1 line
2.21.0
PATCH 8/12 powerpc/pseries: CMM: Implement balloon compaction
In reply to David HildenbrandWe can now get rid of the cmm_lock and completely rely on the balloon compaction internals, which now also manage the page list and the lock.
Inflated/"loaned" pages are now movable. Memory blocks that contain such apges can get offlined. Also, all such pages will be marked PageOffline() and can therefore be excluded in memory dumps using recent versions of makedumpfile.
Don't switch to balloon_page_alloc() yet (due to the GFP_NOIO). Will do that separately to discuss this change in detail.
--- arch/powerpc/platforms/pseries/Kconfig | 1 + arch/powerpc/platforms/pseries/cmm.c | 132 ++++++++++++++++++++++--- include/uapi/linux/magic.h | 1 + 3 files changed, 120 insertions(+), 14 deletions(-)
arch/powerpc/platforms/pseries/Kconfig +1 -0
@@ -108,6 +108,7 @@ config PPC_SMLPAR config CMM tristate "Collaborative memory management" depends on PPC_SMLPAR + select MEMORY_BALLOON default y help Select this option, if you want to enable the kernel interface
arch/powerpc/platforms/pseries/cmm.c +118 -14
204 lines not on this page · raw message
include/uapi/linux/magic.h +1 -0
@@ -94,5 +94,6 @@#define ZSMALLOC_MAGIC 0x58295829#define DMA_BUF_MAGIC 0x444d4142/* "DMAB" */ #define Z3FOLD_MAGIC 0x33 +#define PPC_CMM_MAGIC 0xc7571590#endif /* __LINUX_MAGIC_H__ */
Signature · 1 line
2.21.0
Re: [PATCH v1 08/12] powerpc/pseries: CMM: Implement balloon compaction
Quoted 160 lines · David Hildenbrand
We can now get rid of the
cmm_lockand completely rely on the balloon compaction internals, which now also manage the page list and the lock.Inflated/"loaned" pages are now movable. Memory blocks that contain such apges can get offlined. Also, all such pages will be marked
PageOffline()and can therefore be excluded in memory dumps using recent versions of makedumpfile.Don't switch to
balloon_page_alloc()yet (due to theGFP_NOIO). Will do that separately to discuss this change in detail.Cc Benjamin Herrenschmidt <[email protected]>Cc Paul Mackerras <[email protected]>Cc Michael Ellerman <[email protected]>Cc Thiago Jung Bauermann <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Anshuman Khandual <[email protected]>Cc "Oliver O'Halloran" <[email protected]>Cc Alexey Kardashevskiy <[email protected]>Cc "Enrico Weigelt, metux IT consult" <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Allison Randal <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Pavel Tatashin <[email protected]>Cc Arun KS <[email protected]>Cc Todd Kjos <[email protected]>Cc Christian Brauner <[email protected]>Cc Gao Xiang <[email protected]>Cc Greg Hackmann <[email protected]>Cc David Howells <[email protected]>Signed-off-by David Hildenbrand <[email protected]>--- arch/powerpc/platforms/pseries/Kconfig | 1 + arch/powerpc/platforms/pseries/cmm.c | 132 ++++++++++++++++++++++--- include/uapi/linux/magic.h | 1 + 3 files changed, 120 insertions(+), 14 deletions(-)arch/powerpc/platforms/pseries/Kconfig +1
-0@@ -108,6 +108,7 @@ config PPC_SMLPAR config CMM tristate "Collaborative memory management" depends on PPC_SMLPAR + select MEMORY_BALLOON default y help Select this option, if you want to enable the kernel interfacearch/powerpc/platforms/pseries/cmm.c +40
-13@@ -19,6 +19,10 @@#include <linux/stringify.h> #include <linux/swap.h> #include <linux/device.h> +#include <linux/mount.h> +#include <linux/pseudo_fs.h> +#include <linux/magic.h> +#include <linux/balloon_compaction.h> #include <asm/firmware.h> #include <asm/hvcall.h> #include <asm/mmu.h> @@ -77,13 +81,11 @@ static atomic_long_t loaned_pages;static unsigned long loaned_pages_target;static unsigned long oom_freed_pages; -static LIST_HEAD (cmm_page_list); -static DEFINE_SPINLOCK (cmm_lock); -static DEFINE_MUTEX (hotplug_mutex);static int hotplug_occurred;/* protected by the hotplug mutex */ static struct task_struct *cmm_thread_ptr; +static struct balloon_dev_info b_dev_info;static long plpar_page_set_loaned (struct page *page) {@@ -149,19 +151,16 @@ static long cmm_alloc_pages(long nr) __GFP_NOMEMALLOC);if (!page)break ; -spin_lock (&cmm_lock); rc =plpar_page_set_loaned (page);if (rc) {pr_err ("%s: Can not set page to loaned. rc=%ld\n" , __func__, rc); -spin_unlock (&cmm_lock);__free_page (page);break ; } -list_add (&page->lru, &cmm_page_list); +balloon_page_enqueue (&b_dev_info, page);atomic_long_inc (&loaned_pages);adjust_managed_page_count (page,-1 ); -spin_unlock (&cmm_lock); nr--; }@@ -178,21 +177,19 @@ static long cmm_alloc_pages(long nr)**/ static long cmm_free_pages (long nr) { -struct page *page, *tmp; +struct page *page;cmm_dbg ("Begin free of %ld pages.\n" , nr); -spin_lock (&cmm_lock); -list_for_each_entry_safe (page, tmp, &cmm_page_list, lru) { -if (!nr) +while (nr) { + page =balloon_page_dequeue (&b_dev_info); +if (!page)break ;plpar_page_set_active (page); -list_del (&page->lru);adjust_managed_page_count (page,1 );__free_page (page);atomic_long_dec (&loaned_pages); nr--; } -spin_unlock (&cmm_lock);cmm_dbg ("End request with %ld pages unfulfilled\n" , nr);return nr; }@@ -484,6 +481,105 @@ static struct notifier_block cmm_mem_nb = { .priority = CMM_MEM_HOTPLUG_PRI }; +#ifdef CONFIG_BALLOON_COMPACTION +static struct vfsmount *balloon_mnt; + +static int cmm_init_fs_context (struct fs_context *fc) +{ +return init_pseudo (fc,PPC_CMM_MAGIC ) ?0 : -ENOMEM ; +} + +static struct file_system_type balloon_fs = { + .name ="ppc-cmm" , + .init_fs_context = cmm_init_fs_context, + .kill_sb = kill_anon_super, +}; + +static int cmm_migratepage (struct balloon_dev_info *b_dev_info, +struct page *newpage,struct page *page, +enum migrate_mode mode) +{ +unsigned long flags; + +/* +* loan/"inflate" the newpage first. +* +* We might race against the cmm_thread who might discover after our +* loan request that another page is to be unloaned. However, once +* the cmm_thread runs again later, this error will automatically +* be corrected. +*/ +if (plpar_page_set_loaned (newpage)) { +/* Unlikely, but possible. Tell the caller not to retry now. */
arch/powerpc/platforms/pseries/cmm.c static struct notifier_block cmm_mem_nb = { · line 523
Quoted 10 lines
arch/powerpc/platforms/pseries/cmm.c +10
-0+pr_err_ratelimited ("%s: Cannot set page to loaned." , __func__); +return -EBUSY ; + } + +/* balloon page list reference */ +get_page (newpage); + +spin_lock_irqsave (&b_dev_info->pages_lock, flags); +balloon_page_insert (b_dev_info, newpage); +balloon_page_delete (page);
I think I am missing a b_dev_info->isolated_pages-- here.
Quoted 1 line
arch/powerpc/platforms/pseries/cmm.c +1
-0+spin_unlock_irqrestore (&b_dev_info->pages_lock, flags);
Signature · 3 lines
Thanks, David / dhildenb
Michael Ellerman
reviewerRe: [PATCH v1 08/12] powerpc/pseries: CMM: Implement balloon compaction
Quoted 3 lines · David Hildenbrand
Quoted 2 lines · David Hildenbrand
We can now get rid of the
cmm_lockand completely rely on the balloon compaction internals, which now also manage the page list and the lock.
...
Quoted 21 lines
Quoted 21 lines
arch/powerpc/platforms/pseries/cmm.c +21
-0+ +static int cmm_migratepage (struct balloon_dev_info *b_dev_info, +struct page *newpage,struct page *page, +enum migrate_mode mode) +{ +unsigned long flags; + +/* +* loan/"inflate" the newpage first. +* +* We might race against the cmm_thread who might discover after our +* loan request that another page is to be unloaned. However, once +* the cmm_thread runs again later, this error will automatically +* be corrected. +*/ +if (plpar_page_set_loaned (newpage)) { +/* Unlikely, but possible. Tell the caller not to retry now. */ +pr_err_ratelimited ("%s: Cannot set page to loaned." , __func__); +return -EBUSY ; + } +
Quoted 8 lines
arch/powerpc/platforms/pseries/cmm.c static struct notifier_block cmm_mem_nb = { · line 523
Quoted 6 lines
arch/powerpc/platforms/pseries/cmm.c +6
-0+/* balloon page list reference */ +get_page (newpage); + +spin_lock_irqsave (&b_dev_info->pages_lock, flags); +balloon_page_insert (b_dev_info, newpage); +balloon_page_delete (page);I think I am missing a b_dev_info->isolated_pages-- here.
I don't know this code at all, but looking at other balloon drivers they do seem to do that in roughly the same spot.
I'll add it, how can we test that it's correct?
cheers
Re: [PATCH v1 08/12] powerpc/pseries: CMM: Implement balloon compaction
Quoted 35 lines · Michael Ellerman
Quoted 3 lines · David Hildenbrand
Quoted 2 lines · David Hildenbrand
We can now get rid of the
cmm_lockand completely rely on the balloon compaction internals, which now also manage the page list and the lock....
Quoted 21 lines
Quoted 21 lines
arch/powerpc/platforms/pseries/cmm.c +21
-0+ +static int cmm_migratepage (struct balloon_dev_info *b_dev_info, +struct page *newpage,struct page *page, +enum migrate_mode mode) +{ +unsigned long flags; + +/* +* loan/"inflate" the newpage first. +* +* We might race against the cmm_thread who might discover after our +* loan request that another page is to be unloaned. However, once +* the cmm_thread runs again later, this error will automatically +* be corrected. +*/ +if (plpar_page_set_loaned (newpage)) { +/* Unlikely, but possible. Tell the caller not to retry now. */ +pr_err_ratelimited ("%s: Cannot set page to loaned." , __func__); +return -EBUSY ; + } +Quoted 8 lines
arch/powerpc/platforms/pseries/cmm.c static struct notifier_block cmm_mem_nb = { · line 523
Quoted 6 lines
arch/powerpc/platforms/pseries/cmm.c +6
-0+/* balloon page list reference */ +get_page (newpage); + +spin_lock_irqsave (&b_dev_info->pages_lock, flags); +balloon_page_insert (b_dev_info, newpage); +balloon_page_delete (page);I think I am missing a b_dev_info->isolated_pages-- here.
Quoted 4 lines
I don't know this code at all, but looking at other balloon drivers they do seem to do that in roughly the same spot.
I'll add it, how can we test that it's correct?
It's certainly correct. We increment when we isolate (balloon_page_isolate()) and decrement when we un-isolate.
Un-isolate happens when we putback a isolated page (balloon_page_putback() - migration aborted) or when we successfully migrate it (via balloon_page_migrate()).
The issue is that we cannot decrement in balloon_page_migrate(), as we have to hold the b_dev_info->pages_lock. That's why we have to do it in the registered callback under lock.
Please note that b_dev_info->isolated_pages is only needed for a sanity check in balloon_page_dequeue(). That's why I didn't notice during testing. I wonder if we should at some point rip out that sanity check ...
Thanks and cheers!
Quoted 3 lines
cheers
Signature · 3 lines
Thanks, David / dhildenb
Michael Ellerman
reviewerRe: [PATCH v1 08/12] powerpc/pseries: CMM: Implement balloon compaction
Quoted 44 lines · David Hildenbrand
Quoted 35 lines · Michael Ellerman
Quoted 3 lines · David Hildenbrand
Quoted 2 lines · David Hildenbrand
We can now get rid of the
cmm_lockand completely rely on the balloon compaction internals, which now also manage the page list and the lock....
Quoted 21 lines
Quoted 21 lines
arch/powerpc/platforms/pseries/cmm.c +21
-0+ +static int cmm_migratepage (struct balloon_dev_info *b_dev_info, +struct page *newpage,struct page *page, +enum migrate_mode mode) +{ +unsigned long flags; + +/* +* loan/"inflate" the newpage first. +* +* We might race against the cmm_thread who might discover after our +* loan request that another page is to be unloaned. However, once +* the cmm_thread runs again later, this error will automatically +* be corrected. +*/ +if (plpar_page_set_loaned (newpage)) { +/* Unlikely, but possible. Tell the caller not to retry now. */ +pr_err_ratelimited ("%s: Cannot set page to loaned." , __func__); +return -EBUSY ; + } +Quoted 8 lines
arch/powerpc/platforms/pseries/cmm.c static struct notifier_block cmm_mem_nb = { · line 523
Quoted 6 lines
arch/powerpc/platforms/pseries/cmm.c +6
-0+/* balloon page list reference */ +get_page (newpage); + +spin_lock_irqsave (&b_dev_info->pages_lock, flags); +balloon_page_insert (b_dev_info, newpage); +balloon_page_delete (page);I think I am missing a b_dev_info->isolated_pages-- here.
Quoted 4 lines
I don't know this code at all, but looking at other balloon drivers they do seem to do that in roughly the same spot.
I'll add it, how can we test that it's correct?
It's certainly correct. We increment when we isolate (
balloon_page_isolate()) and decrement when we un-isolate.
Quoted 7 lines
Un-isolate happens when we putback a isolated page (
balloon_page_putback()- migration aborted) or when we successfully migrate it (viaballoon_page_migrate()).The issue is that we cannot decrement in
balloon_page_migrate(), as we have to hold theb_dev_info->pages_lock. That's why we have to do it in the registered callback under lock.
OK, I get it now.
Quoted 3 lines
Please note that
b_dev_info->isolated_pagesis only needed for a sanity check inballoon_page_dequeue(). That's why I didn't notice during testing. I wonder if we should at some point rip out that sanity check ...
OK. Sanity checks can be good, though checks that call BUG() are less nice :) But I'm not an mm expert so I'll defer to you folks on the sanity check.
For now I've merged this series with the decrement added to cmm_migratepage().
cheers
Re: [PATCH v1 01/12] powerpc/pseries: CMM: Implement release() function for sysfs device
In reply to David HildenbrandQuoted 11 lines · David Hildenbrand
When unloading the module, one gets
[ 548.188594] ------------[ cut here ]------------ [ 548.188596] Device 'cmm0' does not have a release() function, it is brok= en and must be fixed. See Documentation/kobject.txt. [ 548.188622] WARNING: CPU: 0 PID: 19308 at drivers/base/core.c:1244 .devi=
ce_release+0xcc/0xf0
...We only have on static fake device. There is nothing to do when releasing the device (via
cmm_exit).
Quoted 11 lines
Cc Benjamin Herrenschmidt <[email protected]>Cc Paul Mackerras <[email protected]>Cc Michael Ellerman <[email protected]>Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Allison Randal <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Arun KS <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Patches 1-10 applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/7d8212747435c534c8d564fbef4541a463c976ff
cheers
Re: [PATCH v1 01/12] powerpc/pseries: CMM: Implement release() function for sysfs device
Quoted 24 lines · Michael Ellerman
Quoted 11 lines · David Hildenbrand
When unloading the module, one gets
[ 548.188594] ------------[ cut here ]------------ [ 548.188596] Device 'cmm0' does not have a release() function, it is brok= en and must be fixed. See Documentation/kobject.txt. [ 548.188622] WARNING: CPU: 0 PID: 19308 at drivers/base/core.c:1244 .devi=
ce_release+0xcc/0xf0
...We only have on static fake device. There is nothing to do when releasing the device (via
cmm_exit).Quoted 11 lines
Cc Benjamin Herrenschmidt <[email protected]>Cc Paul Mackerras <[email protected]>Cc Michael Ellerman <[email protected]>Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Allison Randal <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Arun KS <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 6 lines
Patches 1-10 applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/7d8212747435c534c8d564fbef4541a463c976ff
cheers
Thanks! I'll probably resend patch 11/12 to give it more attention and to fixup one comment leftover in patch 11. I guess if we get ACKs these two patch should also go via your tree to avoid collisions.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v1 01/12] powerpc/pseries: CMM: Implement release() function for sysfs device
In reply to Michael EllermanQuoted 24 lines · Michael Ellerman
Quoted 11 lines · David Hildenbrand
When unloading the module, one gets
[ 548.188594] ------------[ cut here ]------------ [ 548.188596] Device 'cmm0' does not have a release() function, it is brok= en and must be fixed. See Documentation/kobject.txt. [ 548.188622] WARNING: CPU: 0 PID: 19308 at drivers/base/core.c:1244 .devi=
ce_release+0xcc/0xf0
...We only have on static fake device. There is nothing to do when releasing the device (via
cmm_exit).Quoted 11 lines
Cc Benjamin Herrenschmidt <[email protected]>Cc Paul Mackerras <[email protected]>Cc Michael Ellerman <[email protected]>Cc Andrew Morton <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Konstantin Khlebnikov <[email protected]>Cc Allison Randal <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Arun KS <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 6 lines
Patches 1-10 applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/7d8212747435c534c8d564fbef4541a463c976ff
cheers
Hi Michael,
just to make sure you saw the two MM patches (and the ACKs from Michal)
https://lkml.org/lkml/2019/11/14/410
if you prefer that Andrew picks these up, please let me know.
Cheers!
Signature · 3 lines
Thanks, David / dhildenb