Skip to contentSkip to conversation

mm/page_idle: Add per-pid idle page tracking using virtual index

PATCH In review Joel Fernandes (Google) 6 participants 35 messages Aug 7, 2019 – Aug 19, 2019

Subsystems (12)
lore
Share & tools
lore
#0 lore raw

PATCH v5 1/6 mm/page_idle: Add per-pid idle page tracking using virtual index

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

---
v3->v4: Minor fixups (Minchan)
Add swap pte handling (Konstantin, Minchan)
v2->v3:
Fixed a bug where I was doing a kfree that is not needed due to not needing to do GFP_ATOMIC allocations.

v1->v2:
Mark swap ptes as idle (Minchan)
Avoid need for GFP_ATOMIC (Andrew)
Get rid of idle_page_list lock by moving list to stack

Internal review -> v1:
Fixes from Suren.
Corrections to change log, docs (Florian, Sandeep)

 fs/proc/base.c            |   3 +
 fs/proc/internal.h        |   1 +
 fs/proc/task_mmu.c        |  42 +++++
 include/linux/page_idle.h |   4 +
 mm/page_idle.c            | 337 +++++++++++++++++++++++++++++++++-----
 5 files changed, 342 insertions(+), 45 deletions(-)
fs/proc/base.c +3 -0
index ebea9501afb8..fd2f74bd4e35 100644
@@ -3039,6 +3039,9 @@ static const struct pid_entry tgid_base_stuff[] = {
 	REG("smaps",      S_IRUGO, proc_pid_smaps_operations),
 	REG("smaps_rollup", S_IRUGO, proc_pid_smaps_rollup_operations),
 	REG("pagemap",    S_IRUSR, proc_pagemap_operations),
+#ifdef CONFIG_IDLE_PAGE_TRACKING
+	REG("page_idle", S_IRUSR|S_IWUSR, proc_page_idle_operations),
+#endif
 #endif
 #ifdef CONFIG_SECURITY
 	DIR("attr",       S_IRUGO|S_IXUGO, proc_attr_dir_inode_operations, proc_attr_dir_operations),
fs/proc/internal.h +1 -0
index cd0c8d5ce9a1..bc9371880c63 100644
@@ -293,6 +293,7 @@ extern const struct file_operations proc_pid_smaps_operations;
 extern const struct file_operations proc_pid_smaps_rollup_operations;
 extern const struct file_operations proc_clear_refs_operations;
 extern const struct file_operations proc_pagemap_operations;
+extern const struct file_operations proc_page_idle_operations;
 
 extern unsigned long task_vsize(struct mm_struct *);
 extern unsigned long task_statm(struct mm_struct *,
fs/proc/task_mmu.c +42 -0

48 lines not on this page · raw message

include/linux/page_idle.h +4 -0
index 1e894d34bdce..a765a6d14e1a 100644
@@ -106,6 +106,10 @@ static inline void clear_page_idle(struct page *page)
 }
 #endif /* CONFIG_64BIT */
 
+ssize_t page_idle_proc_write(struct file *file,
+	char __user *buf, size_t count, loff_t *ppos);
+ssize_t page_idle_proc_read(struct file *file,
+	char __user *buf, size_t count, loff_t *ppos);
 #else /* !CONFIG_IDLE_PAGE_TRACKING */
 
 static inline bool page_is_young(struct page *page)
mm/page_idle.c +292 -45

403 lines not on this page · raw message

Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
#1 lore raw

PATCH v5 2/6 mm/page_idle: Add support for handling swapped PG_Idle pages

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

---
 arch/Kconfig                  |  3 +++
 include/asm-generic/pgtable.h |  6 ++++++
 mm/page_idle.c                | 26 ++++++++++++++++++++++++--
 mm/rmap.c                     |  2 ++
 4 files changed, 35 insertions(+), 2 deletions(-)
arch/Kconfig +3 -0
index a7b57dd42c26..3aa121ce824e 100644
@@ -575,6 +575,9 @@ config ARCH_WANT_HUGE_PMD_SHARE
 config HAVE_ARCH_SOFT_DIRTY
 	bool
 
+config HAVE_ARCH_PTE_SWP_PGIDLE
+	bool
+
 config HAVE_MOD_ARCH_SPECIFIC
 	bool
 	help
include/asm-generic/pgtable.h +6 -0
index 75d9d68a6de7..6d51d0a355a7 100644
@@ -712,6 +712,12 @@ static inline void ptep_modify_prot_commit(struct vm_area_struct *vma,
 #define arch_start_context_switch(prev)	do {} while (0)
 #endif
 
+#ifndef CONFIG_HAVE_ARCH_PTE_SWP_PGIDLE
+static inline pte_t pte_swp_mkpage_idle(pte_t pte) { return pte; }
+static inline int pte_swp_page_idle(pte_t pte) { return 0; }
+static inline pte_t pte_swp_clear_mkpage_idle(pte_t pte) { return pte; }
+#endif
+
 #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
 #ifndef CONFIG_ARCH_ENABLE_THP_MIGRATION
 static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)
mm/page_idle.c +24 -2
index 9de4f4c67a8c..2766d4ab348c 100644
@@ -276,7 +276,7 @@ struct page_idle_proc_priv {
 };
 
 /*
- * Add page to list to be set as idle later.
+ * Set a page as idle or add it to a list to be set as idle later.
  */
 static void pte_page_idle_proc_add(struct page *page,
 			       unsigned long addr, struct mm_walk *walk)
@@ -303,6 +303,13 @@ static void pte_page_idle_proc_add(struct page *page,
 		page_get = page_idle_get_page(page);
 		if (!page_get)
 			return;
+	} else {
+		/* For swapped pages, set output bit as idle */
+		frames = (addr - priv->start_addr) >> PAGE_SHIFT;
+		bit = frames % BITMAP_CHUNK_BITS;
+		chunk = &chunk[frames / BITMAP_CHUNK_BITS];
+		*chunk |= (1 << bit);
+		return;
 	}
 
 	/*
@@ -323,6 +330,7 @@ static int pte_page_idle_proc_range(pmd_t *pmd, unsigned long addr,
 	spinlock_t *ptl;
 	struct page *page;
 	struct vm_area_struct *vma = walk->vma;
+	struct page_idle_proc_priv *priv = walk->private;
 
 	ptl = pmd_trans_huge_lock(pmd, vma);
 	if (ptl) {
@@ -341,6 +349,19 @@ static int pte_page_idle_proc_range(pmd_t *pmd, unsigned long addr,
 
 	pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
 	for (; addr != end; pte++, addr += PAGE_SIZE) {
+		/* For swap_pte handling, we use an idle bit in the swap pte. */
+		if (is_swap_pte(*pte)) {
+			if (priv->write) {
+				set_pte_at(walk->mm, addr, pte,
+					   pte_swp_mkpage_idle(*pte));
+			} else {
+				/* If swap pte has idle bit set, report it as idle */
+				if (pte_swp_page_idle(*pte))
+					pte_page_idle_proc_add(NULL, addr, walk);
+			}
+			continue;
+		}
+
 		if (!pte_present(*pte))
 			continue;
 
@@ -432,7 +453,8 @@ ssize_t page_idle_proc_generic(struct file *file, char __user *ubuff,
 				set_page_idle(page);
 			}
 		} else {
-			if (page_idle_pte_check(page)) {
+			/* If page is NULL, it was swapped out */
+			if (!page || page_idle_pte_check(page)) {
 				off = ((cur->addr) >> PAGE_SHIFT) - start_frame;
 				bit = off % BITMAP_CHUNK_BITS;
 				index = off / BITMAP_CHUNK_BITS;
mm/rmap.c +2 -0
index e5dfe2ae6b0d..4bd618aab402 100644
@@ -1629,6 +1629,8 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 			swp_pte = swp_entry_to_pte(entry);
 			if (pte_soft_dirty(pteval))
 				swp_pte = pte_swp_mksoft_dirty(swp_pte);
+			if (page_is_idle(page))
+				swp_pte = pte_swp_mkpage_idle(swp_pte);
 			set_pte_at(mm, address, pvmw.pte, swp_pte);
 			/* Invalidate as we cleared the pte */
 			mmu_notifier_invalidate_range(mm, address,
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
#2 lore raw

Re: [PATCH v5 2/6] mm/page_idle: Add support for handling swapped PG_Idle pages

Quoted 9 lines · Joel Fernandes (Google)

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

Quoted 8 lines

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

This could be solved by checking the !present/swapped out pages right? Whoever decided to put the page out to the swap just made it idle effectively. So the monitor can make some educated guess for tracking. If that is fundamentally not possible then please describe why.

Signature · 2 lines
Michal Hocko
SUSE Labs
#3 lore raw

Re: [PATCH v5 2/6] mm/page_idle: Add support for handling swapped PG_Idle pages

Quoted 19 lines · Michal Hocko
Quoted 9 lines · Joel Fernandes (Google)

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

Quoted 8 lines

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

Quoted 5 lines

This could be solved by checking the !present/swapped out pages right? Whoever decided to put the page out to the swap just made it idle effectively. So the monitor can make some educated guess for tracking. If that is fundamentally not possible then please describe why.

But the monitoring process (profiler) does not have control over the 'whoever made it effectively idle' process.

As you said it will be a guess, it will not be accurate.

I am curious what is your concern with using a bit in the swap PTE?

(Adding Konstantin as well since we may be interested in this, since we also suggested this idea).

thanks,

- Joel

#4 lore raw

Re: [PATCH v5 2/6] mm/page_idle: Add support for handling swapped PG_Idle pages

Quoted 27 lines · Joel Fernandes
Quoted 19 lines · Michal Hocko
Quoted 9 lines · Joel Fernandes (Google)

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

Quoted 8 lines

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

Quoted 5 lines

This could be solved by checking the !present/swapped out pages right? Whoever decided to put the page out to the swap just made it idle effectively. So the monitor can make some educated guess for tracking. If that is fundamentally not possible then please describe why.

Quoted 4 lines

But the monitoring process (profiler) does not have control over the 'whoever made it effectively idle' process.

As you said it will be a guess, it will not be accurate.

Yep. Without saving idle bit in swap entry (and presuming that all swap is idle) profiler could miss access. This patch adds accurate tracking almost for free. After that profiler could work with any pace without races.

Quoted 11 lines

I am curious what is your concern with using a bit in the swap PTE?

(Adding Konstantin as well since we may be interested in this, since we also suggested this idea).

thanks,

- Joel

#5 lore raw

Re: [PATCH v5 2/6] mm/page_idle: Add support for handling swapped PG_Idle pages

In reply to Joel Fernandes
Quoted 20 lines · Joel Fernandes
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Joel Fernandes (Google)

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

Quoted 8 lines
Quoted 5 lines

This could be solved by checking the !present/swapped out pages right? Whoever decided to put the page out to the swap just made it idle effectively. So the monitor can make some educated guess for tracking. If that is fundamentally not possible then please describe why.

But the monitoring process (profiler) does not have control over the 'whoever made it effectively idle' process.

Why does that matter? Whether it is a global/memcg reclaim or somebody calling MADV_PAGEOUT or whatever it is a decision to make the page not hot. Sure you could argue that a missing idle bit on swap entries might mean that the swap out decision was pre-mature/sub-optimal/wrong but is this the aim of the interface?

Quoted 1 line

As you said it will be a guess, it will not be accurate.

Yes and the point I am trying to make is that having some space and not giving a guarantee sounds like a safer option for this interface because
...

Quoted 2 lines

I am curious what is your concern with using a bit in the swap PTE?

... It is a promiss of the semantic I find limiting for future. The bit in the pte might turn out insufficient (e.g. pte reclaim) so teaching the userspace to consider this a hard guarantee is a ticket to problems later on. Maybe I am overly paranoid because I have seen so many "nice to have" features turning into a maintenance burden in the past.

If this is really considered mostly debugging purpouse interface then a certain level of imprecision should be tolerateable. If there is a really strong real world usecase that simply has no other way to go then this might be added later. Adding an information is always safer than take it away.

That being said, if I am a minority voice here then I will not really stand in the way and won't nack the patch. I will not ack it neither though.

Signature · 2 lines
Michal Hocko
SUSE Labs
#6 lore raw

Re: [PATCH v5 2/6] mm/page_idle: Add support for handling swapped PG_Idle pages

Quoted 36 lines · Michal Hocko
Quoted 20 lines · Joel Fernandes
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Joel Fernandes (Google)

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

Quoted 8 lines
Quoted 5 lines

This could be solved by checking the !present/swapped out pages right? Whoever decided to put the page out to the swap just made it idle effectively. So the monitor can make some educated guess for tracking. If that is fundamentally not possible then please describe why.

But the monitoring process (profiler) does not have control over the 'whoever made it effectively idle' process.

Why does that matter? Whether it is a global/memcg reclaim or somebody calling MADV_PAGEOUT or whatever it is a decision to make the page not hot. Sure you could argue that a missing idle bit on swap entries might mean that the swap out decision was pre-mature/sub-optimal/wrong but is this the aim of the interface?

Quoted 4 lines
Quoted 1 line

As you said it will be a guess, it will not be accurate.

Yes and the point I am trying to make is that having some space and not giving a guarantee sounds like a safer option for this interface because

I do see your point of view, but jJust because a future (and possibly not going to happen) usecase which you mentioned as pte reclaim, makes you feel that userspace may be subject to inaccuracies anyway, doesn't mean we should make everything inaccurate.. We already know idle page tracking is not completely accurate. But that doesn't mean we miss out on the opportunity to make the "non pte-reclaim" usecase inaccurate as well.

IMO, we should do our best for today, and not hypothesize. How likely is pte reclaim and is there a thread to describe that direction?

Quoted 14 lines
Quoted 1 line

I am curious what is your concern with using a bit in the swap PTE?

... It is a promiss of the semantic I find limiting for future. The bit in the pte might turn out insufficient (e.g. pte reclaim) so teaching the userspace to consider this a hard guarantee is a ticket to problems later on. Maybe I am overly paranoid because I have seen so many "nice to have" features turning into a maintenance burden in the past.

If this is really considered mostly debugging purpouse interface then a certain level of imprecision should be tolerateable. If there is a really strong real world usecase that simply has no other way to go then this might be added later. Adding an information is always safer than take it away.

Quoted 3 lines

That being said, if I am a minority voice here then I will not really stand in the way and won't nack the patch. I will not ack it neither though.

Ok.

thanks,

- Joel

#7 lore raw

Re: [PATCH v5 2/6] mm/page_idle: Add support for handling swapped PG_Idle pages

Quoted 42 lines · Joel Fernandes
Quoted 36 lines · Michal Hocko
Quoted 20 lines · Joel Fernandes
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Joel Fernandes (Google)

Idle page tracking currently does not work well in the following scenario:
1. mark page-A idle which was present at that time.
2. run workload
3. page-A is not touched by workload
4. *sudden* memory pressure happen so finally page A is finally swapped out
5. now see the page A - it appears as if it was accessed (pte unmapped so idle bit not set in output) - but it's incorrect.

To fix this, we store the idle information into a new idle bit of the swap PTE during swapping of anonymous pages.

Also in the future, madvise extensions will allow a system process manager (like Android's ActivityManager) to swap pages out of a process that it knows will be cold. To an external process like a heap profiler that is doing idle tracking on another process, this procedure will interfere with the idle page tracking similar to the above steps.

Quoted 8 lines
Quoted 5 lines

This could be solved by checking the !present/swapped out pages right? Whoever decided to put the page out to the swap just made it idle effectively. So the monitor can make some educated guess for tracking. If that is fundamentally not possible then please describe why.

But the monitoring process (profiler) does not have control over the 'whoever made it effectively idle' process.

Why does that matter? Whether it is a global/memcg reclaim or somebody calling MADV_PAGEOUT or whatever it is a decision to make the page not hot. Sure you could argue that a missing idle bit on swap entries might mean that the swap out decision was pre-mature/sub-optimal/wrong but is this the aim of the interface?

Quoted 4 lines
Quoted 1 line

As you said it will be a guess, it will not be accurate.

Yes and the point I am trying to make is that having some space and not giving a guarantee sounds like a safer option for this interface because

Quoted 6 lines

I do see your point of view, but jJust because a future (and possibly not going to happen) usecase which you mentioned as pte reclaim, makes you feel that userspace may be subject to inaccuracies anyway, doesn't mean we should make everything inaccurate.. We already know idle page tracking is not completely accurate. But that doesn't mean we miss out on the opportunity to make the "non pte-reclaim" usecase inaccurate as well.

Just keep in mind that you will add more burden to future features because they would have to somehow overcome this user visible behavior and we will get to the usual question - Is this going to break something that relies on the idle bit being stable?

Quoted 2 lines

IMO, we should do our best for today, and not hypothesize. How likely is pte reclaim and is there a thread to describe that direction?

Not that I am aware of now but with large NVDIMM mapped files I can see that this will get more and more interesting.

Signature · 2 lines
Michal Hocko
SUSE Labs
#8 lore raw

RFC PATCH v5 3/6 x86: Add support for idle bit in swap PTE

In reply to Joel Fernandes (Google)

This bit will be used by idle page tracking code to correctly identify if a page that was swapped out was idle before it got swapped out. Without this PTE bit, we lose information about if a page is idle or not since the page frame gets unmapped and the page gets freed.

Bits 2-6 are unused in the swap PTE (see the comment in arch/x86/include/asm/pgtable_64.h). Bit 2 corresponds to _PAGE_USER. Use it for swap PTE purposes.

---
 arch/x86/Kconfig                     |  1 +
 arch/x86/include/asm/pgtable.h       | 15 +++++++++++++++
 arch/x86/include/asm/pgtable_types.h |  6 ++++++
 3 files changed, 22 insertions(+)
arch/x86/Kconfig +1 -0
index 222855cc0158..728f22370f17 100644
@@ -139,6 +139,7 @@ config X86
 	select HAVE_ARCH_MMAP_RND_COMPAT_BITS	if MMU && COMPAT
 	select HAVE_ARCH_COMPAT_MMAP_BASES	if MMU && COMPAT
 	select HAVE_ARCH_PREL32_RELOCATIONS
+	select HAVE_ARCH_PTE_SWP_PGIDLE
 	select HAVE_ARCH_SECCOMP_FILTER
 	select HAVE_ARCH_THREAD_STRUCT_WHITELIST
 	select HAVE_ARCH_STACKLEAK
arch/x86/include/asm/pgtable.h +15 -0
index 0bc530c4eb13..ef3e662cee4a 100644
@@ -1371,6 +1371,21 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
 #endif
 #endif
 
+static inline pte_t pte_swp_mkpage_idle(pte_t pte)
+{
+	return pte_set_flags(pte, _PAGE_SWP_PGIDLE);
+}
+
+static inline int pte_swp_page_idle(pte_t pte)
+{
+	return pte_flags(pte) & _PAGE_SWP_PGIDLE;
+}
+
+static inline pte_t pte_swp_clear_mkpage_idle(pte_t pte)
+{
+	return pte_clear_flags(pte, _PAGE_SWP_PGIDLE);
+}
+
 #define PKRU_AD_BIT 0x1
 #define PKRU_WD_BIT 0x2
 #define PKRU_BITS_PER_PKEY 2
arch/x86/include/asm/pgtable_types.h +6 -0
index b5e49e6bac63..6739cba4c900 100644
@@ -100,6 +100,12 @@
 #define _PAGE_SWP_SOFT_DIRTY	(_AT(pteval_t, 0))
 #endif
 
+#ifdef CONFIG_IDLE_PAGE_TRACKING
+#define _PAGE_SWP_PGIDLE	_PAGE_USER
+#else
+#define _PAGE_SWP_PGIDLE	(_AT(pteval_t, 0))
+#endif
+
 #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE)
 #define _PAGE_NX	(_AT(pteval_t, 1) << _PAGE_BIT_NX)
 #define _PAGE_DEVMAP	(_AT(u64, 1) << _PAGE_BIT_DEVMAP)
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
#9 lore raw

RFC PATCH v5 4/6 arm64: Add support for idle bit in swap PTE

In reply to Joel Fernandes (Google)

This bit will be used by idle page tracking code to correctly identify if a page that was swapped out was idle before it got swapped out. Without this PTE bit, we lose information about if a page is idle or not since the page frame gets unmapped.

In this patch we reuse PTE_DEVMAP bit since idle page tracking only works on user pages in the LRU. Device pages should not consitute those so it should be unused and safe to use.

---
 arch/arm64/Kconfig                    |  1 +
 arch/arm64/include/asm/pgtable-prot.h |  1 +
 arch/arm64/include/asm/pgtable.h      | 15 +++++++++++++++
 3 files changed, 17 insertions(+)
arch/arm64/Kconfig +1 -0
index 3adcec05b1f6..9d1412c693d7 100644
@@ -128,6 +128,7 @@ config ARM64
 	select HAVE_ARCH_MMAP_RND_BITS
 	select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT
 	select HAVE_ARCH_PREL32_RELOCATIONS
+	select HAVE_ARCH_PTE_SWP_PGIDLE
 	select HAVE_ARCH_SECCOMP_FILTER
 	select HAVE_ARCH_STACKLEAK
 	select HAVE_ARCH_THREAD_STRUCT_WHITELIST
arch/arm64/include/asm/pgtable-prot.h +1 -0
index 92d2e9f28f28..917b15c5d63a 100644
@@ -18,6 +18,7 @@
 #define PTE_SPECIAL		(_AT(pteval_t, 1) << 56)
 #define PTE_DEVMAP		(_AT(pteval_t, 1) << 57)
 #define PTE_PROT_NONE		(_AT(pteval_t, 1) << 58) /* only when !PTE_VALID */
+#define PTE_SWP_PGIDLE		PTE_DEVMAP		 /* for idle page tracking during swapout */
 
 #ifndef __ASSEMBLY__
 
arch/arm64/include/asm/pgtable.h +15 -0
index 3f5461f7b560..558f5ebd81ba 100644
@@ -212,6 +212,21 @@ static inline pte_t pte_mkdevmap(pte_t pte)
 	return set_pte_bit(pte, __pgprot(PTE_DEVMAP));
 }
 
+static inline int pte_swp_page_idle(pte_t pte)
+{
+	return 0;
+}
+
+static inline pte_t pte_swp_mkpage_idle(pte_t pte)
+{
+	return set_pte_bit(pte, __pgprot(PTE_SWP_PGIDLE));
+}
+
+static inline pte_t pte_swp_clear_page_idle(pte_t pte)
+{
+	return clear_pte_bit(pte, __pgprot(PTE_SWP_PGIDLE));
+}
+
 static inline void set_pte(pte_t *ptep, pte_t pte)
 {
 	WRITE_ONCE(*ptep, pte);
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
#10 lore raw

PATCH v5 5/6 page_idle: Drain all LRU pagevec before idle tracking

In reply to Joel Fernandes (Google)

During idle page tracking, we see that sometimes faulted anon pages are in pagevec but are not drained to LRU. Idle page tracking only considers pages on LRU.

I am able to find multiple issues involving this. One issue looks like idle tracking is completely broken. It shows up in my testing as if a page that is marked as idle is always "accessed" -- because it was never marked as idle (due to not draining of pagevec).

The other issue shows up as a failure during swapping (support for which this series adds), with the following sequence:
1. Allocate some pages
2. Write to them
3. Mark them as idle <--- fails
4. Introduce some memory pressure to induce swapping.
5. Check the swap bit I introduced in this series. <--- fails to set idle
bit in swap PTE.

To fix this, this patch drains all CPU's pagevec before starting idle tracking.

---
 mm/page_idle.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
mm/page_idle.c +21 -0
index 2766d4ab348c..26440a497609 100644
@@ -180,6 +180,13 @@ static ssize_t page_idle_bitmap_read(struct file *file, struct kobject *kobj,
 	unsigned long pfn, end_pfn;
 	int bit, ret;
 
+	/*
+	 * Idle page tracking currently works only on LRU pages, so drain
+	 * them. This can cause slowness, but in the future we could
+	 * remove this operation if we are tracking non-LRU pages too.
+	 */
+	lru_add_drain_all();
+
 	ret = page_idle_get_frames(pos, count, NULL, &pfn, &end_pfn);
 	if (ret == -ENXIO)
 		return 0;  /* Reads beyond max_pfn do nothing */
@@ -211,6 +218,13 @@ static ssize_t page_idle_bitmap_write(struct file *file, struct kobject *kobj,
 	unsigned long pfn, end_pfn;
 	int bit, ret;
 
+	/*
+	 * Idle page tracking currently works only on LRU pages, so drain
+	 * them. This can cause slowness, but in the future we could
+	 * remove this operation if we are tracking non-LRU pages too.
+	 */
+	lru_add_drain_all();
+
 	ret = page_idle_get_frames(pos, count, NULL, &pfn, &end_pfn);
 	if (ret)
 		return ret;
@@ -428,6 +442,13 @@ ssize_t page_idle_proc_generic(struct file *file, char __user *ubuff,
 	walk.private = &priv;
 	walk.mm = mm;
 
+	/*
+	 * Idle page tracking currently works only on LRU pages, so drain
+	 * them. This can cause slowness, but in the future we could
+	 * remove this operation if we are tracking non-LRU pages too.
+	 */
+	lru_add_drain_all();
+
 	down_read(&mm->mmap_sem);
 
 	/*
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
#11 lore raw

PATCH v5 6/6 doc: Update documentation for page_idle virtual address indexing

In reply to Joel Fernandes (Google)

This patch updates the documentation with the new page_idle tracking feature which uses virtual address indexing.

---
 .../admin-guide/mm/idle_page_tracking.rst     | 43 ++++++++++++++++---
 1 file changed, 36 insertions(+), 7 deletions(-)
Documentation/admin-guide/mm/idle_page_tracking.rst +36 -7
index df9394fb39c2..9eef32000f5e 100644
@@ -19,10 +19,14 @@ It is enabled by CONFIG_IDLE_PAGE_TRACKING=y.
 
 User API
 ========
+There are 2 ways to access the idle page tracking API. One uses physical
+address indexing, another uses a simpler virtual address indexing scheme.
 
-The idle page tracking API is located at ``/sys/kernel/mm/page_idle``.
-Currently, it consists of the only read-write file,
-``/sys/kernel/mm/page_idle/bitmap``.
+Physical address indexing
+-------------------------
+The idle page tracking API for physical address indexing using page frame
+numbers (PFN) is located at ``/sys/kernel/mm/page_idle``.  Currently, it
+consists of the only read-write file, ``/sys/kernel/mm/page_idle/bitmap``.
 
 The file implements a bitmap where each bit corresponds to a memory page. The
 bitmap is represented by an array of 8-byte integers, and the page at PFN #i is
@@ -74,6 +78,31 @@ See :ref:`Documentation/admin-guide/mm/pagemap.rst <pagemap>` for more
 information about ``/proc/pid/pagemap``, ``/proc/kpageflags``, and
 ``/proc/kpagecgroup``.
 
+Virtual address indexing
+------------------------
+The idle page tracking API for virtual address indexing using virtual frame
+numbers (VFN) for a process ``<pid>`` is located at ``/proc/<pid>/page_idle``.
+It is a bitmap that follows the same semantics as
+``/sys/kernel/mm/page_idle/bitmap`` except that it uses virtual instead of
+physical frame numbers.
+
+This idle page tracking API does not deal with PFN so it does not require prior
+lookups of ``pagemap``. This is an advantage on some systems where looking up
+PFN is considered a security issue.  Also in some cases, this interface could
+be slightly more reliable to use than physical address indexing, since in
+physical address indexing, address space changes can occur between reading the
+``pagemap`` and reading the ``bitmap``, while in virtual address indexing, the
+process's ``mmap_sem`` is held for the duration of the access.
+
+To estimate the amount of pages that are not used by a workload one should:
+
+ 1. Mark all the workload's pages as idle by setting corresponding bits in
+    ``/proc/<pid>/page_idle``.
+
+ 2. Wait until the workload accesses its working set.
+
+ 3. Read ``/proc/<pid>/page_idle`` and count the number of bits set.
+
 .. _impl_details:
 
 Implementation Details
@@ -99,10 +128,10 @@ When a dirty page is written to swap or disk as a result of memory reclaim or
 exceeding the dirty memory limit, it is not marked referenced.
 
 The idle memory tracking feature adds a new page flag, the Idle flag. This flag
-is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` (see the
-:ref:`User API <user_api>`
-section), and cleared automatically whenever a page is referenced as defined
-above.
+is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` for physical
+addressing or by writing to ``/proc/<pid>/page_idle`` for virtual
+addressing (see the :ref:`User API <user_api>` section), and cleared
+automatically whenever a page is referenced as defined above.
 
 When a page is marked idle, the Accessed bit must be cleared in all PTEs it is
 mapped to, otherwise we will not be able to detect accesses to the page coming
Signature · 1 line
2.22.0.770.g0f2c4a37fd-goog
maintainer
#12 lore raw
In reply to Joel Fernandes (Google)
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

#13 lore raw
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Or are you referring to something else that needs to be made configurable?

thanks,

- Joel

maintainer
#14 lore raw
Quoted 17 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Quoted 5 lines

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

#15 lore raw
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

thanks,

- Joel

#16 lore raw
Quoted 32 lines · Joel Fernandes
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Quoted 7 lines

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what

s/most like/most likely/

Quoted 7 lines

will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

thanks,

- Joel

#17 lore raw
In reply to Joel Fernandes
Quoted 32 lines · Joel Fernandes
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Quoted 9 lines

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

I would tend to agree with Joel here. The functionality falls into an existing IDLE_PAGE_TRACKING config option quite nicely. If there really are users who want to save some space and this is standing in the way then they can easily add a new config option with some justification so the savings are clear. Without that an additional config simply adds to the already existing configurability complexity and balkanization.

Signature · 2 lines
Michal Hocko
SUSE Labs
#18 lore raw
Quoted 43 lines · Michal Hocko
Quoted 32 lines · Joel Fernandes
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Quoted 9 lines

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

Quoted 6 lines

I would tend to agree with Joel here. The functionality falls into an existing IDLE_PAGE_TRACKING config option quite nicely. If there really are users who want to save some space and this is standing in the way then they can easily add a new config option with some justification so the savings are clear. Without that an additional config simply adds to the already existing configurability complexity and balkanization.

Michal, Andrew, Minchan,

Would you have any other review comments on the v5 series? This is just a new interface that does not disrupt existing users of the older page-idle tracking, so as such it is a safe change (as in, doesn't change existing functionality except for the draining bug fix).

thanks,

- Joel

#19 lore raw
Quoted 51 lines · Joel Fernandes
Quoted 43 lines · Michal Hocko
Quoted 32 lines · Joel Fernandes
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Quoted 9 lines

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

Quoted 6 lines

I would tend to agree with Joel here. The functionality falls into an existing IDLE_PAGE_TRACKING config option quite nicely. If there really are users who want to save some space and this is standing in the way then they can easily add a new config option with some justification so the savings are clear. Without that an additional config simply adds to the already existing configurability complexity and balkanization.

Quoted 6 lines

Michal, Andrew, Minchan,

Would you have any other review comments on the v5 series? This is just a new interface that does not disrupt existing users of the older page-idle tracking, so as such it is a safe change (as in, doesn't change existing functionality except for the draining bug fix).

I hope to find some more time to finish the review but let me point out that "it's new it is regression safe" is not really a great argument for a new user visible API. If the API is flawed then this is likely going to kick us later and will be hard to fix. I am still not convinced about the swap part of the thing TBH.

Signature · 2 lines
Michal Hocko
SUSE Labs
#20 lore raw
Quoted 54 lines · Michal Hocko
Quoted 53 lines · Joel Fernandes
Quoted 43 lines · Michal Hocko
Quoted 32 lines · Joel Fernandes
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Quoted 9 lines

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

Quoted 6 lines

I would tend to agree with Joel here. The functionality falls into an existing IDLE_PAGE_TRACKING config option quite nicely. If there really are users who want to save some space and this is standing in the way then they can easily add a new config option with some justification so the savings are clear. Without that an additional config simply adds to the already existing configurability complexity and balkanization.

Michal, Andrew, Minchan,

Quoted 8 lines
Quoted 4 lines

Would you have any other review comments on the v5 series? This is just a new interface that does not disrupt existing users of the older page-idle tracking, so as such it is a safe change (as in, doesn't change existing functionality except for the draining bug fix).

I hope to find some more time to finish the review but let me point out that "it's new it is regression safe" is not really a great argument for a new user visible API.

Actually, I think you misunderstood me and took it out of context. I never intended to say "it is regression safe". I meant to say it is "low risk", as in that in all likelihood should not be hurting *existing users* of the *old interface*. Also as you know, it has been tested.

Quoted 3 lines

If the API is flawed then this is likely going
to kick us later and will be hard to fix. I am still not convinced about the swap part of the thing TBH.

Ok, then let us discuss it. As I mentioned before, without this we lose the access information due to MADVISE or swapping. Minchan and Konstantin both suggested it that's why I also added it (other than me also realizing that it is neeed). For x86, it uses existing bits in pte so it is not adding any more bits. For arm64, it uses unused bits that the hardware cannot use. So I don't see why this is an issue to you.

thanks,

- Joel

#21 lore raw
Quoted 60 lines · Joel Fernandes
Quoted 59 lines · Michal Hocko
Quoted 57 lines · Joel Fernandes
Quoted 43 lines · Michal Hocko
Quoted 32 lines · Joel Fernandes
Quoted 23 lines · Andrew Morton
Quoted 21 lines · On Wed, 7 Aug 2019 16:45:30 -0400 Joel Fernandes <[email protected]> wrote:
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Wed, 7 Aug 2019 13:15:54 -0400 "Joel Fernandes (Google)"…

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

So is heapprofd a developer-only thing? Is heapprofd included in end-user android loads? If not then, again, wouldn't it be better to make the feature Kconfigurable so that Android developers can enable it during development then disable it for production kernels?

Almost all of this code is already configurable with CONFIG_IDLE_PAGE_TRACKING. If you disable it, then all of this code gets disabled.

Quoted 7 lines
Quoted 1 line

Or are you referring to something else that needs to be made configurable?

Yes - the 300+ lines of code which this patchset adds!

The impacted people will be those who use the existing idle-page-tracking feature but who will not use the new feature. I guess we can assume this set is small...

Quoted 9 lines

Yes, I think this set should be small. The code size increase of page_idle.o is from ~1KB to ~2KB. Most of the extra space is consumed by page_idle_proc_generic() function which this patch adds. I don't think adding another CONFIG option to disable this while keeping existing CONFIG_IDLE_PAGE_TRACKING enabled, is worthwhile but I am open to the addition of such an option if anyone feels strongly about it. I believe that once this patch is merged, most like this new interface being added is what will be used more than the old interface (for some of the usecases) so it makes sense to keep it alive with CONFIG_IDLE_PAGE_TRACKING.

Quoted 6 lines

I would tend to agree with Joel here. The functionality falls into an existing IDLE_PAGE_TRACKING config option quite nicely. If there really are users who want to save some space and this is standing in the way then they can easily add a new config option with some justification so the savings are clear. Without that an additional config simply adds to the already existing configurability complexity and balkanization.

Michal, Andrew, Minchan,

Would you have any other review comments on the v5 series? This is just a new interface that does not disrupt existing users of the older page-idle tracking, so as such it is a safe change (as in, doesn't change existing functionality except for the draining bug fix).

Quoted 8 lines
Quoted 3 lines

I hope to find some more time to finish the review but let me point out that "it's new it is regression safe" is not really a great argument for a new user visible API.

Actually, I think you misunderstood me and took it out of context. I never intended to say "it is regression safe". I meant to say it is "low risk", as in that in all likelihood should not be hurting *existing users* of the *old interface*. Also as you know, it has been tested.

Yeah, misreading on my end.

Quoted 8 lines
Quoted 3 lines

If the API is flawed then this is likely going
to kick us later and will be hard to fix. I am still not convinced about the swap part of the thing TBH.

Ok, then let us discuss it. As I mentioned before, without this we lose the access information due to MADVISE or swapping. Minchan and Konstantin both suggested it that's why I also added it (other than me also realizing that it is neeed).

I have described my concerns about the general idle bit behavior after unmapping pointing to discrepancy with !anon pages. And I believe those haven't been addressed yet. Besides that I am still not seeing any description of the usecase that would suffer from the lack of the functionality in changelogs.

Signature · 2 lines
Michal Hocko
SUSE Labs
#22 lore raw

On Tue, Aug 13, 2019 at 04:14:32PM +0200, Michal Hocko wrote:
[snip]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

If the API is flawed then this is likely going
to kick us later and will be hard to fix. I am still not convinced about the swap part of the thing TBH.

Quoted 8 lines
Quoted 4 lines

Ok, then let us discuss it. As I mentioned before, without this we lose the access information due to MADVISE or swapping. Minchan and Konstantin both suggested it that's why I also added it (other than me also realizing that it is neeed).

I have described my concerns about the general idle bit behavior after unmapping pointing to discrepancy with !anon pages. And I believe those haven't been addressed yet.

You are referring to this post right?
https://lkml.org/lkml/2019/8/6/637

Specifically your question was:
How are you going to handle situation when the page is unmapped and refaulted again (e.g. a normal reclaim of a pagecache)?

Currently I don't know how to implement that. Would it work if I stored the page-idle bit information in the pte of the file page (after the page is unmapped by reclaim?).

Also, this could be a future extension - the Android heap profiler does not need it right now. I know that's not a good argument but it is useful to say that it doesn't affect a real world usecase.. the swap issue on the other hand, is a real usecase. Since the profiler should not get affected by swapping or MADVISE_COLD hints.

Quoted 3 lines

Besides that I am still not seeing any
description of the usecase that would suffer from the lack of the functionality in changelogs.

You are talking about the swap usecase? The usecase is well layed out in v5 2/6. Did you see it? https://lore.kernel.org/patchwork/patch/1112283/

thanks,

- Joel

#23 lore raw
Quoted 18 lines · Joel Fernandes

On Tue, Aug 13, 2019 at 04:14:32PM +0200, Michal Hocko wrote:
[snip]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

If the API is flawed then this is likely going
to kick us later and will be hard to fix. I am still not convinced about the swap part of the thing TBH.

Quoted 8 lines
Quoted 4 lines

Ok, then let us discuss it. As I mentioned before, without this we lose the access information due to MADVISE or swapping. Minchan and Konstantin both suggested it that's why I also added it (other than me also realizing that it is neeed).

I have described my concerns about the general idle bit behavior after unmapping pointing to discrepancy with !anon pages. And I believe those haven't been addressed yet.

You are referring to this post right?
https://lkml.org/lkml/2019/8/6/637

Quoted 6 lines

Specifically your question was:
How are you going to handle situation when the page is unmapped and refaulted again (e.g. a normal reclaim of a pagecache)?

Currently I don't know how to implement that. Would it work if I stored the page-idle bit information in the pte of the file page (after the page is unmapped by reclaim?).

It would work as long as we keep page tables around after unmap. As they are easily reconstructable this is a good candidate for reclaim as well.

Quoted 6 lines

Also, this could be a future extension - the Android heap profiler does not need it right now. I know that's not a good argument but it is useful to say that it doesn't affect a real world usecase.. the swap issue on the other hand, is a real usecase. Since the profiler should not get affected by swapping or MADVISE_COLD hints.

Quoted 6 lines
Quoted 3 lines

Besides that I am still not seeing any
description of the usecase that would suffer from the lack of the functionality in changelogs.

You are talking about the swap usecase? The usecase is well layed out in v5 2/6. Did you see it? https://lore.kernel.org/patchwork/patch/1112283/

For some reason I've missed it. I will coment on that.

Signature · 2 lines
Michal Hocko
SUSE Labs
reviewer
#24 lore raw
In reply to Joel Fernandes (Google)
Quoted 12 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

Quoted 8 lines

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

If the heap profiler is only interested in anonymous, process-private memory, that might be an easy way out? Limit (unprivileged) use of this interface to pages that aren't shared with any other processes?

Quoted 11 lines
mm/page_idle.c +11 -0
+/* Helper to get the start and end frame given a pos and count */
+static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
+                               unsigned long *start, unsigned long *end)
+{
+       unsigned long max_frame;
+
+       /* If an mm is not given, assume we want physical frames */
+       max_frame = mm ? (mm->task_size >> PAGE_SHIFT) : max_pfn;
+
+       if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
+               return -EINVAL;

mm/page_idle.c static void page_idle_clear_pte_refs(struct page *page) · line 153

Quoted 10 lines
mm/page_idle.c +10 -0
+
+       *start = pos * BITS_PER_BYTE;
+       if (*start >= max_frame)
+               return -ENXIO;
+
+       *end = *start + count * BITS_PER_BYTE;
+       if (*end > max_frame)
+               *end = max_frame;
+       return 0;
+}

You could add some overflow checks for the multiplications. I haven't seen any place where it actually matters, but it seems unclean; and in particular, on a 32-bit architecture where the maximum user address is very high (like with a 4G:4G split), it looks like this function might theoretically return with *start > *end, which could be confusing to callers.

[...]

Quoted 11 lines
        for (; pfn < end_pfn; pfn++) {
                bit = pfn % BITMAP_CHUNK_BITS;
                if (!bit)
                        *out = 0ULL;
mm/page_idle.c +0 -7
-               page = page_idle_get_page(pfn);
-               if (page) {
-                       if (page_is_idle(page)) {
-                               /*
-                                * The page might have been referenced via a
-                                * pte, in which case it is not idle. Clear
-                                * refs and recheck.

mm/page_idle.c static ssize_t page_idle_bitmap_read(struct file *file, struct kobject *kobj, · line 197

Quoted 10 lines
mm/page_idle.c +3 -5
-                                */
-                               page_idle_clear_pte_refs(page);
-                               if (page_is_idle(page))
-                                       *out |= 1ULL << bit;
-                       }
+               page = page_idle_get_page_pfn(pfn);
+               if (page && page_idle_pte_check(page)) {
+                       *out |= 1ULL << bit;
                        put_page(page);
                }

The page && !page_idle_pte_check(page) case looks like it's missing a put_page(); you probably intended to write something like this?

    page = page_idle_get_page_pfn(pfn);
    if (page) {
        if (page_idle_pte_check(page))
            *out |= 1ULL << bit;
        put_page(page);
    }

[...]

Quoted 8 lines
mm/page_idle.c +8 -0
+/*  page_idle tracking for /proc/<pid>/page_idle */
+
+struct page_node {
+       struct page *page;
+       unsigned long addr;
+       struct list_head list;
+};
+

mm/page_idle.c struct page_ext_operations page_idle_ops = { · line 276

Quoted 10 lines
mm/page_idle.c +10 -0
+struct page_idle_proc_priv {
+       unsigned long start_addr;
+       char *buffer;
+       int write;
+
+       /* Pre-allocate and provide nodes to pte_page_idle_proc_add() */
+       struct page_node *page_nodes;
+       int cur_page_node;
+       struct list_head *idle_page_list;
+};

A linked list is a weird data structure to use if the list elements are just consecutive array elements.

Quoted 12 lines
mm/page_idle.c +12 -0
+/*
+ * Add page to list to be set as idle later.
+ */
+static void pte_page_idle_proc_add(struct page *page,
+                              unsigned long addr, struct mm_walk *walk)
+{
+       struct page *page_get = NULL;
+       struct page_node *pn;
+       int bit;
+       unsigned long frames;
+       struct page_idle_proc_priv *priv = walk->private;
+       u64 *chunk = (u64 *)priv->buffer;

mm/page_idle.c struct page_ext_operations page_idle_ops = { · line 299

Quoted 10 lines
mm/page_idle.c +10 -0
+
+       if (priv->write) {
+               VM_BUG_ON(!page);
+
+               /* Find whether this page was asked to be marked */
+               frames = (addr - priv->start_addr) >> PAGE_SHIFT;
+               bit = frames % BITMAP_CHUNK_BITS;
+               chunk = &chunk[frames / BITMAP_CHUNK_BITS];
+               if (((*chunk >> bit) & 1) == 0)
+                       return;

This means that BITMAP_CHUNK_SIZE is UAPI on big-endian systems, right? My opinion is that it would be slightly nicer to design the UAPI such that incrementing virtual addresses are mapped to incrementing offsets in the buffer (iow, either use bytewise access or use little-endian), but I'm not going to ask you to redesign the UAPI this late.

[...]

Quoted 3 lines
mm/page_idle.c +3 -0
+ssize_t page_idle_proc_generic(struct file *file, char __user *ubuff,
+                              size_t count, loff_t *pos, int write)
+{

[...]

Quoted 1 line

+ down_read(&mm->mmap_sem);

[...]

mm/page_idle.c struct page_ext_operations page_idle_ops = { · line 450

Quoted 5 lines
mm/page_idle.c +5 -0
+
+       if (!write && !walk_error)
+               ret = copy_to_user(ubuff, buffer, count);
+
+       up_read(&mm->mmap_sem);

I'd move the up_read() above the copy_to_user(); copy_to_user() can block, and there's no reason to hold the mmap_sem across copy_to_user().

Sorry about only chiming in at v5 with all this.

#25 lore raw
Quoted 23 lines · Jann Horn
Quoted 12 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

Quoted 8 lines

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

Signature · 2 lines
Michal Hocko
SUSE Labs
#26 lore raw
Quoted 24 lines · Michal Hocko
Quoted 23 lines · Jann Horn
Quoted 20 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 7 lines
Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Yes, that's right. This patch doesn't change that semantic. Idle page tracking at the core is a global procedure which is based on pages that can be shared.

One of the usecases of the heap profiler is to enable profiling of pages that are shared between zygote and any processes that are forked. In this case, I am told by our team working on the heap profiler, that the monitoring of shared pages will help.

Quoted 7 lines
Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

The heap profiler is currently unprivileged. Would it help the concern Jann raised, if the new interface was limited to only anonymous private/shared pages and not to file pages? Or, is this even a real concern?

thanks,

- Joel

reviewer
#27 lore raw
Quoted 56 lines · Joel Fernandes
Quoted 24 lines · Michal Hocko
Quoted 23 lines · Jann Horn
Quoted 20 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 7 lines
Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Yes, that's right. This patch doesn't change that semantic. Idle page tracking at the core is a global procedure which is based on pages that can be shared.

One of the usecases of the heap profiler is to enable profiling of pages that are shared between zygote and any processes that are forked. In this case, I am told by our team working on the heap profiler, that the monitoring of shared pages will help.

Quoted 7 lines
Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

Quoted 3 lines

The heap profiler is currently unprivileged. Would it help the concern Jann raised, if the new interface was limited to only anonymous private/shared pages and not to file pages? Or, is this even a real concern?

+Daniel Gruss in case he wants to provide some more detail; he has been involved in a lot of the public research around this topic.

It is a bit of a concern when code that wasn't hardened as rigorously as cryptographic library code operates on secret values.
A paper was published this year that abused mincore() in combination with tricks for flushing the page cache to obtain information about when shared read-only memory was accessed:
<https://arxiv.org/pdf/1901.01161.pdf>. In response to that, the semantics of mincore() were changed to prevent that information from leaking (see commit 134fca9063ad4851de767d1768180e5dede9a881).

On the other hand, an attacker could also use things like cache timing attacks instead of abusing operating system features; that is more hardware-specific, but it has a higher spatial granularity (typically 64 bytes instead of 4096 bytes). Timing-granularity-wise, I'm not sure whether the proposed interface would be more or less bad than existing cache side-channels on common architectures. There are papers that demonstrate things like being able to distinguish some classes of keyboard keys from others on an Android phone:
<https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_lipp.pdf>

I don't think limiting it to anonymous pages is necessarily enough to completely solve this; in a normal Linux environment, it might be good enough, but on Android, I'm worried about the CoW private memory from the zygote.

reviewer
#28 lore raw
In reply to Michal Hocko
Quoted 39 lines · Michal Hocko
Quoted 23 lines · Jann Horn
Quoted 12 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

Quoted 8 lines

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

I suppose read-only access isn't a real problem as long as the profiler isn't writing the idle state in a very tight loop... but I don't see a usecase where you'd actually want that? As far as I can tell, if you can't write the idle state, being able to read it is pretty much useless.

If the profiler only wants to profile process-private memory, then that should be implementable in a safe way in principle, I think, but since Joel said that they want to profile CoW memory as well, I think that's inherently somewhat dangerous.

#29 lore raw
Quoted 53 lines · Jann Horn
Quoted 39 lines · Michal Hocko
Quoted 23 lines · Jann Horn
Quoted 12 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

Quoted 8 lines

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

I suppose read-only access isn't a real problem as long as the profiler isn't writing the idle state in a very tight loop... but I don't see a usecase where you'd actually want that? As far as I can tell, if you can't write the idle state, being able to read it is pretty much useless.

Quoted 4 lines

If the profiler only wants to profile process-private memory, then that should be implementable in a safe way in principle, I think, but since Joel said that they want to profile CoW memory as well, I think that's inherently somewhat dangerous.

I agree that allowing profiling of shared pages would leak information. To me the use case is not entirely clear. This is not a feature that would normally be run in everyday computer usage, right?

#30 lore raw
Quoted 54 lines · Daniel Gruss
Quoted 53 lines · Jann Horn
Quoted 39 lines · Michal Hocko
Quoted 23 lines · Jann Horn
Quoted 12 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

Quoted 8 lines

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

I suppose read-only access isn't a real problem as long as the profiler isn't writing the idle state in a very tight loop... but I don't see a usecase where you'd actually want that? As far as I can tell, if you can't write the idle state, being able to read it is pretty much useless.

Quoted 6 lines
Quoted 4 lines

If the profiler only wants to profile process-private memory, then that should be implementable in a safe way in principle, I think, but since Joel said that they want to profile CoW memory as well, I think that's inherently somewhat dangerous.

I agree that allowing profiling of shared pages would leak information.

Will think more about it. If we limit it to private pages, then it could become useless. Consider a scenario where:
A process allocates a some memory, then forks a bunch of worker processes that read that memory and perform some work with them. Per-PID page idle tracking is now run on the parent processes. Now it should appear that the pages are actively accessed (not-idle). If we don't track shared pages, then we cannot detect if those pages are really due to memory leaking, or if they are there for a purpose and are actively used.

Quoted 2 lines

To me the use case is not entirely clear. This is not a feature that would normally be run in everyday computer usage, right?

Generally, this to be used as a debugging feature that helps developers detect memory leaks in their programs.

thanks,

- Joel

#31 lore raw
In reply to Jann Horn
Quoted 53 lines · Jann Horn
Quoted 39 lines · Michal Hocko
Quoted 23 lines · Jann Horn
Quoted 12 lines · Joel Fernandes (Google)

The page_idle tracking feature currently requires looking up the pagemap for a process followed by interacting with /sys/kernel/mm/page_idle. Looking up PFN from pagemap in Android devices is not supported by unprivileged process and requires SYS_ADMIN and gives 0 for the PFN.

This patch adds support to directly interact with page_idle tracking at the PID level by introducing a /proc/<pid>/page_idle file. It follows the exact same semantics as the global /sys/kernel/mm/page_idle, but now looking up PFN through pagemap is not needed since the interface uses virtual frame numbers, and at the same time also does not require SYS_ADMIN.

Quoted 8 lines

In Android, we are using this for the heap profiler (heapprofd) which profiles and pin points code paths which allocates and leaves memory idle for long periods of time. This method solves the security issue with userspace learning the PFN, and while at it is also shown to yield better results than the pagemap lookup, the theory being that the window where the address space can change is reduced by eliminating the intermediate pagemap look up stage. In virtual address indexing, the process's mmap_sem is held for the duration of the access.

Quoted 4 lines

What happens when you use this interface on shared pages, like memory inherited from the zygote, library file mappings and so on? If two profilers ran concurrently for two different processes that both map the same libraries, would they end up messing up each other's data?

Yup PageIdle state is shared. That is the page_idle semantic even now IIRC.

Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

I suppose read-only access isn't a real problem as long as the profiler isn't writing the idle state in a very tight loop... but I don't see a usecase where you'd actually want that? As far as I can tell, if you can't write the idle state, being able to read it is pretty much useless.

Quoted 4 lines

If the profiler only wants to profile process-private memory, then that should be implementable in a safe way in principle, I think, but since Joel said that they want to profile CoW memory as well, I think that's inherently somewhat dangerous.

I cannot really say how useful that would be but I can see that implementing ownership checks would be really non-trivial for shared pages. Reducing the interface to exclusive pages would make it easier as you noted but less helpful.

Besides that the attack vector shouldn't be really much different from the page cache access, right? So essentially can_do_mincore model.

I guess we want to document that page idle tracking should be used with care because it potentially opens a side channel opportunity if used on sensitive data.

Signature · 2 lines
Michal Hocko
SUSE Labs
#32 lore raw

On Wed, Aug 14, 2019 at 09:56:01AM +0200, Michal Hocko wrote:
[snip]

Quoted 30 lines
Quoted 20 lines
Quoted 7 lines
Quoted 6 lines

Can this be used to observe which library pages other processes are accessing, even if you don't have access to those processes, as long as you can map the same libraries? I realize that there are already a bunch of ways to do that with side channels and such; but if you're adding an interface that allows this by design, it seems to me like something that should be gated behind some sort of privilege check.

Quoted 6 lines

Hmm, you need to be priviledged to get the pfn now and without that you cannot get to any page so the new interface is weakening the rules. Maybe we should limit setting the idle state to processes with the write status. Or do you think that even observing idle status is useful for practical side channel attacks? If yes, is that a problem of the profiler which does potentially dangerous things?

I suppose read-only access isn't a real problem as long as the profiler isn't writing the idle state in a very tight loop... but I don't see a usecase where you'd actually want that? As far as I can tell, if you can't write the idle state, being able to read it is pretty much useless.

Quoted 4 lines

If the profiler only wants to profile process-private memory, then that should be implementable in a safe way in principle, I think, but since Joel said that they want to profile CoW memory as well, I think that's inherently somewhat dangerous.

I cannot really say how useful that would be but I can see that implementing ownership checks would be really non-trivial for shared pages. Reducing the interface to exclusive pages would make it easier as you noted but less helpful.

Quoted 6 lines

Besides that the attack vector shouldn't be really much different from the page cache access, right? So essentially can_do_mincore model.

I guess we want to document that page idle tracking should be used with care because it potentially opens a side channel opportunity if used on sensitive data.

I have been thinking of this, and discussing with our heap profiler folks. Not being able to track shared pages would be a limitation, but I don't see any way forward considering this security concern so maybe we have to limit what we can do.

I will look into implementing this without doing the rmap but still make it work on shared pages from the point of view of the process being tracked. It just would no longer through the PTEs of *other* processes sharing the page.

My current thought is to just rely on the PTE accessed bit, and not use the PageIdle flag at all. But we'd still set the PageYoung flag so that the reclaim code still sees the page as accessed. The reason I feel like avoiding the PageIdle flag is:

1. It looks like mark_page_accessed() can be called from other paths which can also result in some kind of side-channel issue if a page was shared.

2. I don't think I need the PageIdle flag since the access bit alone should let me know, although it could be a bit slower. Since previously, I did not need to check every PTE and if the PageIdle flag was already cleared, then the page was declared as idle.

At least this series resulted in a bug fix and a tonne of learning, so thank you everyone!

Any other thoughts?

thanks,

- Joel

#33 lore raw
In reply to Jann Horn

On Mon, Aug 12, 2019 at 08:14:38PM +0200, Jann Horn wrote:
[snip]

Quoted 20 lines
Quoted 20 lines
(quoted diff) +20 -0
+/* Helper to get the start and end frame given a pos and count */
+static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
+                               unsigned long *start, unsigned long *end)
+{
+       unsigned long max_frame;
+
+       /* If an mm is not given, assume we want physical frames */
+       max_frame = mm ? (mm->task_size >> PAGE_SHIFT) : max_pfn;
+
+       if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
+               return -EINVAL;
+
+       *start = pos * BITS_PER_BYTE;
+       if (*start >= max_frame)
+               return -ENXIO;
+
+       *end = *start + count * BITS_PER_BYTE;
+       if (*end > max_frame)
+               *end = max_frame;
+       return 0;
Quoted 8 lines
Quoted 1 line

+}

You could add some overflow checks for the multiplications. I haven't seen any place where it actually matters, but it seems unclean; and in particular, on a 32-bit architecture where the maximum user address is very high (like with a 4G:4G split), it looks like this function might theoretically return with *start > *end, which could be confusing to callers.

I could store the multiplication result in unsigned long long (since we are bounds checking with max_frame, start > end would not occur). Something like the following (with extraneous casts). But I'll think some more about the point you are raising.

mm/page_idle.c +8 -4
index 1ea21bbb36cb..8fd7a5559986 100644
@@ -135,6 +135,7 @@ static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
 				unsigned long *start, unsigned long *end)
 {
 	unsigned long max_frame;
+	unsigned long long tmp;
 
 	/* If an mm is not given, assume we want physical frames */
 	max_frame = mm ? (mm->task_size >> PAGE_SHIFT) : max_pfn;
@@ -142,13 +143,16 @@ static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
 	if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
 		return -EINVAL;
 
-	*start = pos * BITS_PER_BYTE;
-	if (*start >= max_frame)
+	tmp = pos * BITS_PER_BYTE;
+	if (tmp >= (unsigned long long)max_frame)
 		return -ENXIO;
+	*start = (unsigned long)tmp;
 
-	*end = *start + count * BITS_PER_BYTE;
-	if (*end > max_frame)
+	tmp = *start + count * BITS_PER_BYTE;
+	if (tmp > (unsigned long long)max_frame)
 		*end = max_frame;
+	else
+		*end = (unsigned long)tmp;
 	return 0;
 }
 
Quoted 26 lines

[...]

Quoted 11 lines
        for (; pfn < end_pfn; pfn++) {
                bit = pfn % BITMAP_CHUNK_BITS;
                if (!bit)
                        *out = 0ULL;
(quoted diff) +0 -7
-               page = page_idle_get_page(pfn);
-               if (page) {
-                       if (page_is_idle(page)) {
-                               /*
-                                * The page might have been referenced via a
-                                * pte, in which case it is not idle. Clear
-                                * refs and recheck.

Review

Quoted 10 lines
(quoted diff) +3 -5
-                                */
-                               page_idle_clear_pte_refs(page);
-                               if (page_is_idle(page))
-                                       *out |= 1ULL << bit;
-                       }
+               page = page_idle_get_page_pfn(pfn);
+               if (page && page_idle_pte_check(page)) {
+                       *out |= 1ULL << bit;
                        put_page(page);
                }

The page && !page_idle_pte_check(page) case looks like it's missing a put_page(); you probably intended to write something like this?

Quoted 6 lines
    page = page_idle_get_page_pfn(pfn);
    if (page) {
        if (page_idle_pte_check(page))
            *out |= 1ULL << bit;
        put_page(page);
    }

Ah, you're right. Will fix, good eyes and thank you!

Quoted 14 lines

[...]

Quoted 13 lines
(quoted diff) +13 -0
+/*  page_idle tracking for /proc/<pid>/page_idle */
+
+struct page_node {
+       struct page *page;
+       unsigned long addr;
+       struct list_head list;
+};
+
+struct page_idle_proc_priv {
+       unsigned long start_addr;
+       char *buffer;
+       int write;
+
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+       /* Pre-allocate and provide nodes to pte_page_idle_proc_add() */
+       struct page_node *page_nodes;
+       int cur_page_node;
+       struct list_head *idle_page_list;
+};

A linked list is a weird data structure to use if the list elements are just consecutive array elements.

Not all of the pages will be considered in the later parts of the code, some pages are skipped.

However, in v4 I added an array to allocate all the page_node structures, since Andrew did not want GFP_ATOMIC allocations of individual list nodes.

So I think I could get rid of the linked list and leave the unused page_node(s) as NULL, but I don't know I have to check if that is possible. I could be missing a corner case, regardless I'll make a note of this and try!

Quoted 21 lines
Quoted 21 lines
(quoted diff) +21 -0
+/*
+ * Add page to list to be set as idle later.
+ */
+static void pte_page_idle_proc_add(struct page *page,
+                              unsigned long addr, struct mm_walk *walk)
+{
+       struct page *page_get = NULL;
+       struct page_node *pn;
+       int bit;
+       unsigned long frames;
+       struct page_idle_proc_priv *priv = walk->private;
+       u64 *chunk = (u64 *)priv->buffer;
+
+       if (priv->write) {
+               VM_BUG_ON(!page);
+
+               /* Find whether this page was asked to be marked */
+               frames = (addr - priv->start_addr) >> PAGE_SHIFT;
+               bit = frames % BITMAP_CHUNK_BITS;
+               chunk = &chunk[frames / BITMAP_CHUNK_BITS];
+               if (((*chunk >> bit) & 1) == 0)
Quoted 8 lines
Quoted 1 line

+ return;

This means that BITMAP_CHUNK_SIZE is UAPI on big-endian systems, right? My opinion is that it would be slightly nicer to design the UAPI such that incrementing virtual addresses are mapped to incrementing offsets in the buffer (iow, either use bytewise access or use little-endian), but I'm not going to ask you to redesign the UAPI this late.

That would also be slow and consume more memory in userspace buffers. Currently, a 64-bit (8 byte) chunk accounts for 64 pages worth or 256KB.

Also I wanted to keep the interface consistent with the global /sys/kernel/mm/page_idle interface.

Quoted 7 lines

[...]

Quoted 3 lines
(quoted diff) +3 -0
+ssize_t page_idle_proc_generic(struct file *file, char __user *ubuff,
+                              size_t count, loff_t *pos, int write)
+{

[...]

Quoted 1 line

+ down_read(&mm->mmap_sem);

[...]

Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+       if (!write && !walk_error)
+               ret = copy_to_user(ubuff, buffer, count);
+
+       up_read(&mm->mmap_sem);

I'd move the up_read() above the copy_to_user(); copy_to_user() can block, and there's no reason to hold the mmap_sem across copy_to_user().

Will do.

Quoted 1 line

Sorry about only chiming in at v5 with all this.

No problem, I'm glad you did!

thanks,

- Joel

reviewer
#34 lore raw
Quoted 31 lines · Joel Fernandes

On Mon, Aug 12, 2019 at 08:14:38PM +0200, Jann Horn wrote:
[snip]

Quoted 20 lines
Quoted 20 lines
(quoted diff) +20 -0
+/* Helper to get the start and end frame given a pos and count */
+static int page_idle_get_frames(loff_t pos, size_t count, struct mm_struct *mm,
+                               unsigned long *start, unsigned long *end)
+{
+       unsigned long max_frame;
+
+       /* If an mm is not given, assume we want physical frames */
+       max_frame = mm ? (mm->task_size >> PAGE_SHIFT) : max_pfn;
+
+       if (pos % BITMAP_CHUNK_SIZE || count % BITMAP_CHUNK_SIZE)
+               return -EINVAL;
+
+       *start = pos * BITS_PER_BYTE;
+       if (*start >= max_frame)
+               return -ENXIO;
+
+       *end = *start + count * BITS_PER_BYTE;
+       if (*end > max_frame)
+               *end = max_frame;
+       return 0;
Quoted 8 lines
Quoted 1 line

+}

You could add some overflow checks for the multiplications. I haven't seen any place where it actually matters, but it seems unclean; and in particular, on a 32-bit architecture where the maximum user address is very high (like with a 4G:4G split), it looks like this function might theoretically return with *start > *end, which could be confusing to callers.

Quoted 4 lines

I could store the multiplication result in unsigned long long (since we are bounds checking with max_frame, start > end would not occur). Something like the following (with extraneous casts). But I'll think some more about the point you are raising.

check_mul_overflow() exists and could make that a bit cleaner.

Quoted 9 lines
Quoted 6 lines

This means that BITMAP_CHUNK_SIZE is UAPI on big-endian systems, right? My opinion is that it would be slightly nicer to design the UAPI such that incrementing virtual addresses are mapped to incrementing offsets in the buffer (iow, either use bytewise access or use little-endian), but I'm not going to ask you to redesign the UAPI this late.

That would also be slow and consume more memory in userspace buffers. Currently, a 64-bit (8 byte) chunk accounts for 64 pages worth or 256KB.

I still wanted to use one bit per page; I just wanted to rearrange the bits. So the first byte would always contain 8 bits corresponding to the first 8 pages, instead of corresponding to pages 56-63 on some systems depending on endianness. Anyway, this is a moot point, since as you said...

Quoted 2 lines

Also I wanted to keep the interface consistent with the global /sys/kernel/mm/page_idle interface.

Sorry, I missed that this is already UAPI in the global interface. I agree, using a different API for the per-process interface would be a bad idea.