reading proc/pid/maps under RCU
Subsystems (7)
Suren Baghdasaryan
reviewercover reading proc/pid/maps under RCU
The issue this patchset is trying to address is mmap_lock contention when a low priority task (monitoring, data collecting, etc.) blocks a higher priority task from making updated to the address space. The contention is due to the mmap_lock being held for read when reading proc/pid/maps. With maple_tree introduction, VMA tree traversals are RCU-safe and per-vma locks make VMA access RCU-safe. this provides an opportunity for lock-less reading of proc/pid/maps. We still need to overcome a couple obstacles:
1. Make all VMA pointer fields used for proc/pid/maps content generation RCU-safe;
2. Ensure that proc/pid/maps data tearing, which is currently possible at page boundaries only, does not get worse.
The patchset deals with these issues but there is a downside which I would like to get input on:
This change introduces unfairness towards the reader of proc/pid/maps, which can be blocked by an overly active/malicious address space modifyer. A couple of ways I though we can address this issue are:
1. After several lock-less retries (or some time limit) to fall back to taking mmap_lock.
2. Employ lock-less reading only if the reader has low priority, indicating that blocking it is not critical.
3. Introducing a separate procfs file which publishes the same data in lock-less manner.
I imagine a combination of these approaches can also be employed. I would like to get feedback on this from the Linux community.
Note: mmap_read_lock/mmap_read_unlock sequence inside validate_map() can be replaced with more efficiend rwsem_wait() proposed by Matthew in [1].
[1] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (3):
mm: make vm_area_struct anon_name field RCU-safe
seq_file: add validate() operation to seq_operations mm/maps: read proc/pid/maps under RCU
fs/proc/internal.h | 3 + fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++---- fs/seq_file.c | 24 ++++++- include/linux/mm_inline.h | 10 ++- include/linux/mm_types.h | 3 +- include/linux/seq_file.h | 1 + mm/madvise.c | 30 +++++++-- 7 files changed, 181 insertions(+), 20 deletions(-)
Signature · 1 line
2.43.0.381.gb435a96ce8-goog
Suren Baghdasaryan
reviewerRFC 1/3 mm: make vm_area_struct anon_name field RCU-safe
For lockless /proc/pid/maps reading we have to ensure all the fields used when generating the output are RCU-safe. The only pointer fields in vm_area_struct which are used to generate that file's output are vm_file and anon_name. vm_file is RCU-safe but anon_name is not. Make anon_name RCU-safe as well.
--- include/linux/mm_inline.h | 10 +++++++++- include/linux/mm_types.h | 3 ++- mm/madvise.c | 30 ++++++++++++++++++++++++++---- 3 files changed, 37 insertions(+), 6 deletions(-)
include/linux/mm_inline.h +9 -1
@@ -389,7 +389,7 @@ static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma,struct anon_vma_name *anon_name =anon_vma_name (orig_vma);if (anon_name) - new_vma->anon_name =anon_vma_name_reuse (anon_name); +rcu_assign_pointer (new_vma->anon_name,anon_vma_name_reuse (anon_name)); }static inline void free_anon_vma_name (struct vm_area_struct *vma)
@@ -411,6 +411,8 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, !strcmp(anon_name1->name, anon_name2->name); } +struct anon_vma_name *anon_vma_name_get_rcu (struct vm_area_struct *vma); +#else /* CONFIG_ANON_VMA_NAME */ static inline void anon_vma_name_get (struct anon_vma_name *anon_name) {}static inline void anon_vma_name_put (struct anon_vma_name *anon_name) {}
@@ -424,6 +426,12 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, return true; } +static inline +struct anon_vma_name *anon_vma_name_get_rcu (struct vm_area_struct *vma) +{ +return NULL ; +} +#endif /* CONFIG_ANON_VMA_NAME */ static inline void init_tlb_flush_pending (struct mm_struct *mm)
include/linux/mm_types.h +2 -1
@@ -545,6 +545,7 @@ struct vm_userfaultfd_ctx {};struct anon_vma_name {struct kref kref; +struct rcu_head rcu;/* The name needs to be at the end because it is dynamically sized. */ char name[]; };
@@ -699,7 +700,7 @@ struct vm_area_struct {* terminated string containing the name given to the vma, or NULL if * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. */ -struct anon_vma_name *anon_name; +struct anon_vma_name __rcu *anon_name;#endif #ifdef CONFIG_SWAP atomic_long_t swap_readahead_info;
mm/madvise.c +26 -4
@@ -88,14 +88,15 @@ void anon_vma_name_free(struct kref *kref) {struct anon_vma_name *anon_name = container_of(kref, struct anon_vma_name, kref); -kfree (anon_name); +kfree_rcu (anon_name, rcu); }struct anon_vma_name *anon_vma_name (struct vm_area_struct *vma) {mmap_assert_locked (vma->vm_mm); -return vma->anon_name; +return rcu_dereference_protected (vma->anon_name, +rwsem_is_locked (&vma->vm_mm->mmap_lock)); }/* mmap_lock should be write-locked */
@@ -105,7 +106,7 @@ static int replace_anon_vma_name(struct vm_area_struct *vma,struct anon_vma_name *orig_name =anon_vma_name (vma);if (!anon_name) { - vma->anon_name =NULL ; +rcu_assign_pointer (vma->anon_name,NULL );anon_vma_name_put (orig_name);return 0 ; }
@@ -113,11 +114,32 @@ static int replace_anon_vma_name(struct vm_area_struct *vma,if (anon_vma_name_eq (orig_name, anon_name))return 0 ; - vma->anon_name =anon_vma_name_reuse (anon_name); +rcu_assign_pointer (vma->anon_name,anon_vma_name_reuse (anon_name));anon_vma_name_put (orig_name); return 0; } + +/* +* Returned anon_vma_name is stable due to elevated refcount but not guaranteed +* to be assigned to the original VMA after the call. +*/ +struct anon_vma_name *anon_vma_name_get_rcu (struct vm_area_struct *vma) +{ +struct anon_vma_name __rcu *anon_name; + +WARN_ON_ONCE (!rcu_read_lock_held ()); + + anon_name =rcu_dereference (vma->anon_name); +if (!anon_name) +return NULL ; + +if (unlikely (!kref_get_unless_zero (&anon_name->kref))) +return NULL ; + +return anon_name; +} +#else /* CONFIG_ANON_VMA_NAME */ static int replace_anon_vma_name (struct vm_area_struct *vma,struct anon_vma_name *anon_name)
Signature · 1 line
2.43.0.381.gb435a96ce8-goog
Suren Baghdasaryan
reviewerRFC 2/3 seq_file: add validate() operation to seq_operations
In reply to Suren Baghdasaryanseq_file outputs data in chunks using seq_file.buf as the intermediate storage before outputting the generated data for the current chunk. It is possible for already buffered data to become stale before it gets reported. In certain situations it is desirable to regenerate that data instead of reporting the stale one. Provide a validate() operation called before outputting the buffered data to allow users to validate buffered data. To indicate valid data, user's validate callback should return 0, to request regeneration of the stale data it should return -EAGAIN, any other error will be considered fatal and read operation will be aborted.
--- fs/seq_file.c | 24 +++++++++++++++++++++++- include/linux/seq_file.h | 1 + 2 files changed, 24 insertions(+), 1 deletion(-)
fs/seq_file.c +23 -1
@@ -172,6 +172,8 @@ ssize_t seq_read_iter(struct kiocb *iocb, struct iov_iter *iter) {struct seq_file *m = iocb->ki_filp->private_data;size_t copied =0 ; +loff_t orig_index; +size_t orig_count;size_t n;void *p;int err =0 ;
@@ -220,6 +222,10 @@ ssize_t seq_read_iter(struct kiocb *iocb, struct iov_iter *iter)if (m->count)// hadn't managed to copy everything goto Done; } + + orig_index = m->index; + orig_count = m->count; +Again:// get a non-empty record in the buffer m->from =0 ; p = m->op->start (m, &m->index);
@@ -278,6 +284,22 @@ ssize_t seq_read_iter(struct kiocb *iocb, struct iov_iter *iter) } } m->op->stop (m, p); +/* Note: we validate even if err<0 to prevent publishing copied data */ +if (m->op->validate) { +int val_err = m->op->validate (m, p); + +if (val_err) { +if (val_err == -EAGAIN ) { + m->index = orig_index; + m->count = orig_count; +// data is stale, retry + goto Again; + } +// data is invalid, return the last error + err = val_err; + goto Done; + } + } n =copy_to_iter (m->buf, m->count, iter); copied += n; m->count -= n;
@@ -572,7 +594,7 @@ static void single_stop(struct seq_file *p, void *v)int single_open (struct file *file,int (*show)(struct seq_file *,void *),void *data) { -struct seq_operations *op =kmalloc (sizeof (*op),GFP_KERNEL_ACCOUNT ); +struct seq_operations *op =kzalloc (sizeof (*op),GFP_KERNEL_ACCOUNT );int res = -ENOMEM ;if (op) {
include/linux/seq_file.h +1 -0
@@ -34,6 +34,7 @@ struct seq_operations {void (*stop) (struct seq_file *m,void *v);void * (*next) (struct seq_file *m,void *v,loff_t *pos);int (*show) (struct seq_file *m,void *v); +int (*validate)(struct seq_file *m,void *v); };#define SEQ_SKIP 1
Signature · 1 line
2.43.0.381.gb435a96ce8-goog
Suren Baghdasaryan
reviewerRFC 3/3 mm/maps: read proc/pid/maps under RCU
In reply to Suren BaghdasaryanWith maple_tree supporting vma tree traversal under RCU and per-vma locks making vma access RCU-safe, /proc/pid/maps can be read under RCU and without the need to read-lock mmap_lock. However vma content can change from under us, therefore we need to pin pointer fields used when generating the output (currently only vm_file and anon_name). In addition, we validate data before publishing it to the user using new seq_file validate interface. This way we keep this mechanism consistent with the previous behavior where data tearing is possible only at page boundaries.
This change is designed to reduce mmap_lock contention and prevent a process reading /proc/pid/maps files (often a low priority task, such as monitoring/data collection services) from blocking address space updates.
--- fs/proc/internal.h | 3 ++ fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 120 insertions(+), 13 deletions(-)
fs/proc/internal.h +3 -0
@@ -290,6 +290,9 @@ struct proc_maps_private {struct task_struct *task;struct mm_struct *mm;struct vma_iterator iter; +int mm_lock_seq; +struct anon_vma_name *anon_name; +struct file *vm_file;#ifdef CONFIG_NUMA struct mempolicy *task_mempolicy;#endif
fs/proc/task_mmu.c +117 -13
168 lines not on this page · raw message
Signature · 1 line
2.43.0.381.gb435a96ce8-goog
kernel test robot
build botRe: [RFC 3/3] mm/maps: read proc/pid/maps under RCU
kernel test robot noticed the following build errors:
Hi Suren,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.7 next-20240118] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Suren-Baghdasaryan/mm-make-vm_area_struct-anon_name-field-RCU-safe/20240116-024359 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20240115183837.205694-4-surenb%40google.com patch subject: [RFC 3/3] mm/maps: read proc/pid/maps under RCU config: i386-randconfig-011-20240118 (https://download.01.org/0day-ci/archive/20240118/[email protected]/config) compiler: gcc-11 (Debian 11.3.0-12) 11.3.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240118/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from include/linux/list.h:11,
from include/linux/preempt.h:12,
from include/linux/spinlock.h:56,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from include/linux/pagewalk.h:5,
from fs/proc/task_mmu.c:2:
fs/proc/task_mmu.c: In function 'm_start':
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
arch/x86/include/asm/barrier.h:72:17: note: in definition of macro '__smp_load_acquire'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
In file included from <command-line>:
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:391:27: note: in definition of macro '__unqual_scalar_typeof'
391 | _Generic((x), \
| ^
include/asm-generic/rwonce.h:50:9: note: in expansion of macro '__READ_ONCE'
50 | __READ_ONCE(x); \
| ^~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
In file included from ./arch/x86/include/generated/asm/rwonce.h:1,
from include/linux/compiler.h:251,
from include/asm-generic/bug.h:5,
from arch/x86/include/asm/bug.h:87,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/mm.h:6,
from include/linux/pagewalk.h:5,
from fs/proc/task_mmu.c:2:
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/asm-generic/rwonce.h:44:73: note: in definition of macro '__READ_ONCE'
44 | #define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
| ^
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
In file included from <command-line>:
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:9: note: in expansion of macro 'compiletime_assert'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:28: note: in expansion of macro '__native_word'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:73:9: note: in expansion of macro 'compiletime_assert_atomic_type'
73 | compiletime_assert_atomic_type(*p); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:9: note: in expansion of macro 'compiletime_assert'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:28: note: in expansion of macro '__native_word'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:73:9: note: in expansion of macro 'compiletime_assert_atomic_type'
73 | compiletime_assert_atomic_type(*p); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:9: note: in expansion of macro 'compiletime_assert'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:28: note: in expansion of macro '__native_word'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:73:9: note: in expansion of macro 'compiletime_assert_atomic_type'
73 | compiletime_assert_atomic_type(*p); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
>> fs/proc/task_mmu.c:190:63: error: 'struct mm_struct' has no member named 'mm_lock_seq'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:9: note: in expansion of macro 'compiletime_assert'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:438:28: note: in expansion of macro '__native_word'
438 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:73:9: note: in expansion of macro 'compiletime_assert_atomic_type'
73 | compiletime_assert_atomic_type(*p); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:190:37: note: in expansion of macro 'smp_load_acquire'
190 | priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
fs/proc/task_mmu.c: In function 'get_vma_fields':
>> fs/proc/task_mmu.c:387:18: error: 'struct vm_area_struct' has no member named 'anon_name'; did you mean 'anon_vma'?
387 | if (vma->anon_name) {
| ^~~~~~~~~
| anon_vma
In file included from include/linux/list.h:11,
from include/linux/preempt.h:12,
from include/linux/spinlock.h:56,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from include/linux/pagewalk.h:5,
from fs/proc/task_mmu.c:2:
fs/proc/task_mmu.c: In function 'validate_map':
fs/proc/task_mmu.c:437:57: error: 'struct mm_struct' has no member named 'mm_lock_seq'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
arch/x86/include/asm/barrier.h:72:17: note: in definition of macro '__smp_load_acquire'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^
fs/proc/task_mmu.c:437:31: note: in expansion of macro 'smp_load_acquire'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
In file included from <command-line>:
fs/proc/task_mmu.c:437:57: error: 'struct mm_struct' has no member named 'mm_lock_seq'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:437:31: note: in expansion of macro 'smp_load_acquire'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:437:57: error: 'struct mm_struct' has no member named 'mm_lock_seq'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:437:31: note: in expansion of macro 'smp_load_acquire'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~~~~~~~~~~~~~~~
fs/proc/task_mmu.c:437:57: error: 'struct mm_struct' has no member named 'mm_lock_seq'
437 | mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
| ^~
include/linux/compiler_types.h:415:23: note: in definition of macro '__compiletime_assert'
415 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:435:9: note: in expansion of macro '_compiletime_assert'
435 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/barrier.h:72:28: note: in expansion of macro 'READ_ONCE'
72 | typeof(*p) ___p1 = READ_ONCE(*p); \
| ^~~~~~~~~
include/asm-generic/barrier.h:176:29: note: in expansion of macro '__smp_load_acquire'
176 | #define smp_load_acquire(p) __smp_load_acquire(p)
| ^~~~~~~~~~~~~~~~~~vim +190 fs/proc/task_mmu.c
> 2 #include <linux/pagewalk.h>
3 #include <linux/mm_inline.h>
4 #include <linux/hugetlb.h>
5 #include <linux/huge_mm.h>
6 #include <linux/mount.h>
7 #include <linux/ksm.h>
8 #include <linux/seq_file.h>
9 #include <linux/highmem.h>
10 #include <linux/ptrace.h>
11 #include <linux/slab.h>
12 #include <linux/pagemap.h>
13 #include <linux/mempolicy.h>
14 #include <linux/rmap.h>
15 #include <linux/swap.h>
16 #include <linux/sched/mm.h>
17 #include <linux/swapops.h>
18 #include <linux/mmu_notifier.h>
19 #include <linux/page_idle.h>
20 #include <linux/shmem_fs.h>
21 #include <linux/uaccess.h>
22 #include <linux/pkeys.h>
23 #include <linux/minmax.h>
24 #include <linux/overflow.h>
25
26 #include <asm/elf.h>
27 #include <asm/tlb.h>
28 #include <asm/tlbflush.h>
29 #include "internal.h"
30
31 #define SEQ_PUT_DEC(str, val) \
32 seq_put_decimal_ull_width(m, str, (val) << (PAGE_SHIFT-10), 8)
33 void task_mem(struct seq_file *m, struct mm_struct *mm)
34 {
35 unsigned long text, lib, swap, anon, file, shmem;
36 unsigned long hiwater_vm, total_vm, hiwater_rss, total_rss;
37
38 anon = get_mm_counter(mm, MM_ANONPAGES);
39 file = get_mm_counter(mm, MM_FILEPAGES);
40 shmem = get_mm_counter(mm, MM_SHMEMPAGES);
41
42 /*
43 * Note: to minimize their overhead, mm maintains hiwater_vm and
44 * hiwater_rss only when about to *lower* total_vm or rss. Any
45 * collector of these hiwater stats must therefore get total_vm
46 * and rss too, which will usually be the higher. Barriers? not
47 * worth the effort, such snapshots can always be inconsistent.
48 */
49 hiwater_vm = total_vm = mm->total_vm;
50 if (hiwater_vm < mm->hiwater_vm)
51 hiwater_vm = mm->hiwater_vm;
52 hiwater_rss = total_rss = anon + file + shmem;
53 if (hiwater_rss < mm->hiwater_rss)
54 hiwater_rss = mm->hiwater_rss;
55
56 /* split executable areas between text and lib */
57 text = PAGE_ALIGN(mm->end_code) - (mm->start_code & PAGE_MASK);
58 text = min(text, mm->exec_vm << PAGE_SHIFT);
59 lib = (mm->exec_vm << PAGE_SHIFT) - text;
60
61 swap = get_mm_counter(mm, MM_SWAPENTS);
62 SEQ_PUT_DEC("VmPeak:\t", hiwater_vm);
63 SEQ_PUT_DEC(" kB\nVmSize:\t", total_vm);
64 SEQ_PUT_DEC(" kB\nVmLck:\t", mm->locked_vm);
65 SEQ_PUT_DEC(" kB\nVmPin:\t", atomic64_read(&mm->pinned_vm));
66 SEQ_PUT_DEC(" kB\nVmHWM:\t", hiwater_rss);
67 SEQ_PUT_DEC(" kB\nVmRSS:\t", total_rss);
68 SEQ_PUT_DEC(" kB\nRssAnon:\t", anon);
69 SEQ_PUT_DEC(" kB\nRssFile:\t", file);
70 SEQ_PUT_DEC(" kB\nRssShmem:\t", shmem);
71 SEQ_PUT_DEC(" kB\nVmData:\t", mm->data_vm);
72 SEQ_PUT_DEC(" kB\nVmStk:\t", mm->stack_vm);
73 seq_put_decimal_ull_width(m,
74 " kB\nVmExe:\t", text >> 10, 8);
75 seq_put_decimal_ull_width(m,
76 " kB\nVmLib:\t", lib >> 10, 8);
77 seq_put_decimal_ull_width(m,
78 " kB\nVmPTE:\t", mm_pgtables_bytes(mm) >> 10, 8);
79 SEQ_PUT_DEC(" kB\nVmSwap:\t", swap);
80 seq_puts(m, " kB\n");
81 hugetlb_report_usage(m, mm);
82 }
83 #undef SEQ_PUT_DEC
84
85 unsigned long task_vsize(struct mm_struct *mm)
86 {
87 return PAGE_SIZE * mm->total_vm;
88 }
89
90 unsigned long task_statm(struct mm_struct *mm,
91 unsigned long *shared, unsigned long *text,
92 unsigned long *data, unsigned long *resident)
93 {
94 *shared = get_mm_counter(mm, MM_FILEPAGES) +
95 get_mm_counter(mm, MM_SHMEMPAGES);
96 *text = (PAGE_ALIGN(mm->end_code) - (mm->start_code & PAGE_MASK))
97 >> PAGE_SHIFT;
98 *data = mm->data_vm + mm->stack_vm;
99 *resident = *shared + get_mm_counter(mm, MM_ANONPAGES);
100 return mm->total_vm;
101 }
102
103 #ifdef CONFIG_NUMA
104 /*
105 * Save get_task_policy() for show_numa_map().
106 */
107 static void hold_task_mempolicy(struct proc_maps_private *priv)
108 {
109 struct task_struct *task = priv->task;
110
111 task_lock(task);
112 priv->task_mempolicy = get_task_policy(task);
113 mpol_get(priv->task_mempolicy);
114 task_unlock(task);
115 }
116 static void release_task_mempolicy(struct proc_maps_private *priv)
117 {
118 mpol_put(priv->task_mempolicy);
119 }
120 #else
121 static void hold_task_mempolicy(struct proc_maps_private *priv)
122 {
123 }
124 static void release_task_mempolicy(struct proc_maps_private *priv)
125 {
126 }
127 #endif
128
129 static struct vm_area_struct *proc_get_vma(struct proc_maps_private *priv,
130 loff_t *ppos)
131 {
132 struct vm_area_struct *vma = vma_next(&priv->iter);
133
134 if (vma) {
135 *ppos = vma->vm_start;
136 } else {
137 *ppos = -2UL;
138 vma = get_gate_vma(priv->mm);
139 }
140
141 return vma;
142 }
143
144 static const struct seq_operations proc_pid_maps_op;
145
146 static inline bool needs_mmap_lock(struct seq_file *m)
147 {
148 #ifdef CONFIG_PER_VMA_LOCK
149 /*
150 * smaps and numa_maps perform page table walk, therefore require
151 * mmap_lock but maps can be read under RCU.
152 */
153 return m->op != &proc_pid_maps_op;
154 #else
155 /* Without per-vma locks VMA access is not RCU-safe */
156 return true;
157 #endif
158 }
159
160 static void *m_start(struct seq_file *m, loff_t *ppos)
161 {
162 struct proc_maps_private *priv = m->private;
163 unsigned long last_addr = *ppos;
164 struct mm_struct *mm;
165
166 /* See m_next(). Zero at the start or after lseek. */
167 if (last_addr == -1UL)
168 return NULL;
169
170 priv->task = get_proc_task(priv->inode);
171 if (!priv->task)
172 return ERR_PTR(-ESRCH);
173
174 mm = priv->mm;
175 if (!mm || !mmget_not_zero(mm)) {
176 put_task_struct(priv->task);
177 priv->task = NULL;
178 return NULL;
179 }
180
181 if (needs_mmap_lock(m)) {
182 if (mmap_read_lock_killable(mm)) {
183 mmput(mm);
184 put_task_struct(priv->task);
185 priv->task = NULL;
186 return ERR_PTR(-EINTR);
187 }
188 } else {
189 /* For memory barrier see the comment for mm_lock_seq in mm_struct */
> 190 priv->mm_lock_seq = smp_load_acquire(&priv->mm->mm_lock_seq);
191 rcu_read_lock();
192 }
193
194 vma_iter_init(&priv->iter, mm, last_addr);
195 hold_task_mempolicy(priv);
196 if (last_addr == -2UL)
197 return get_gate_vma(mm);
198
199 return proc_get_vma(priv, ppos);
200 }
201
202 static void *m_next(struct seq_file *m, void *v, loff_t *ppos)
203 {
204 if (*ppos == -2UL) {
205 *ppos = -1UL;
206 return NULL;
207 }
208 return proc_get_vma(m->private, ppos);
209 }
210
211 static void m_stop(struct seq_file *m, void *v)
212 {
213 struct proc_maps_private *priv = m->private;
214 struct mm_struct *mm = priv->mm;
215
216 if (!priv->task)
217 return;
218
219 release_task_mempolicy(priv);
220 if (needs_mmap_lock(m))
221 mmap_read_unlock(mm);
222 else
223 rcu_read_unlock();
224 mmput(mm);
225 put_task_struct(priv->task);
226 priv->task = NULL;
227 }
228
229 static int proc_maps_open(struct inode *inode, struct file *file,
230 const struct seq_operations *ops, int psize)
231 {
232 struct proc_maps_private *priv = __seq_open_private(file, ops, psize);
233
234 if (!priv)
235 return -ENOMEM;
236
237 priv->inode = inode;
238 priv->mm = proc_mem_open(inode, PTRACE_MODE_READ);
239 if (IS_ERR(priv->mm)) {
240 int err = PTR_ERR(priv->mm);
241
242 seq_release_private(inode, file);
243 return err;
244 }
245
246 return 0;
247 }
248
249 static int proc_map_release(struct inode *inode, struct file *file)
250 {
251 struct seq_file *seq = file->private_data;
252 struct proc_maps_private *priv = seq->private;
253
254 if (priv->mm)
255 mmdrop(priv->mm);
256
257 return seq_release_private(inode, file);
258 }
259
260 static int do_maps_open(struct inode *inode, struct file *file,
261 const struct seq_operations *ops)
262 {
263 return proc_maps_open(inode, file, ops,
264 sizeof(struct proc_maps_private));
265 }
266
267 static void show_vma_header_prefix(struct seq_file *m,
268 unsigned long start, unsigned long end,
269 vm_flags_t flags, unsigned long long pgoff,
270 dev_t dev, unsigned long ino)
271 {
272 seq_setwidth(m, 25 + sizeof(void *) * 6 - 1);
273 seq_put_hex_ll(m, NULL, start, 8);
274 seq_put_hex_ll(m, "-", end, 8);
275 seq_putc(m, ' ');
276 seq_putc(m, flags & VM_READ ? 'r' : '-');
277 seq_putc(m, flags & VM_WRITE ? 'w' : '-');
278 seq_putc(m, flags & VM_EXEC ? 'x' : '-');
279 seq_putc(m, flags & VM_MAYSHARE ? 's' : 'p');
280 seq_put_hex_ll(m, " ", pgoff, 8);
281 seq_put_hex_ll(m, " ", MAJOR(dev), 2);
282 seq_put_hex_ll(m, ":", MINOR(dev), 2);
283 seq_put_decimal_ull(m, " ", ino);
284 seq_putc(m, ' ');
285 }
286
287 static void
288 show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
289 {
290 struct anon_vma_name *anon_name = NULL;
291 struct mm_struct *mm = vma->vm_mm;
292 struct file *file = vma->vm_file;
293 vm_flags_t flags = vma->vm_flags;
294 unsigned long ino = 0;
295 unsigned long long pgoff = 0;
296 unsigned long start, end;
297 dev_t dev = 0;
298 const char *name = NULL;
299
300 if (file) {
301 const struct inode *inode = file_user_inode(vma->vm_file);
302
303 dev = inode->i_sb->s_dev;
304 ino = inode->i_ino;
305 pgoff = ((loff_t)vma->vm_pgoff) << PAGE_SHIFT;
306 }
307
308 start = vma->vm_start;
309 end = vma->vm_end;
310 show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino);
311 if (mm) {
312 anon_name = needs_mmap_lock(m) ? anon_vma_name(vma) :
313 anon_vma_name_get_rcu(vma);
314 }
315
316 /*
317 * Print the dentry name for named mappings, and a
318 * special [heap] marker for the heap:
319 */
320 if (file) {
321 seq_pad(m, ' ');
322 /*
323 * If user named this anon shared memory via
324 * prctl(PR_SET_VMA ..., use the provided name.
325 */
326 if (anon_name)
327 seq_printf(m, "[anon_shmem:%s]", anon_name->name);
328 else
329 seq_path(m, file_user_path(file), "\n");
330 goto done;
331 }
332
333 if (vma->vm_ops && vma->vm_ops->name) {
334 name = vma->vm_ops->name(vma);
335 if (name)
336 goto done;
337 }
338
339 name = arch_vma_name(vma);
340 if (!name) {
341 if (!mm) {
342 name = "[vdso]";
343 goto done;
344 }
345
346 if (vma_is_initial_heap(vma)) {
347 name = "[heap]";
348 goto done;
349 }
350
351 if (vma_is_initial_stack(vma)) {
352 name = "[stack]";
353 goto done;
354 }
355
356 if (anon_name) {
357 seq_pad(m, ' ');
358 seq_printf(m, "[anon:%s]", anon_name->name);
359 }
360 }
361
362 done:
363 if (name) {
364 seq_pad(m, ' ');
365 seq_puts(m, name);
366 }
367 seq_putc(m, '\n');
368 if (anon_name && !needs_mmap_lock(m))
369 anon_vma_name_put(anon_name);
370 }
371
372 /*
373 * Pin vm_area_struct fields used by show_map_vma. We also copy pinned fields
374 * into proc_maps_private because by the time put_vma_fields() is called, VMA
375 * might have changed and these fields might be pointing to different objects.
376 */
377 static bool get_vma_fields(struct vm_area_struct *vma, struct proc_maps_private *priv)
378 {
379 if (vma->vm_file) {
380 priv->vm_file = get_file_rcu(&vma->vm_file);
381 if (!priv->vm_file)
382 return false;
383
384 } else
385 priv->vm_file = NULL;
386
> 387 if (vma->anon_name) { 388 priv->anon_name = anon_vma_name_get_rcu(vma);
389 if (!priv->anon_name) {
390 if (priv->vm_file) {
391 fput(priv->vm_file);
392 return false;
393 }
394 }
395 } else
396 priv->anon_name = NULL;
397
398 return true;
399 }
400
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
On 1/15/24 19:38, Suren Baghdasaryan wrote:
Hi,
Quoted 11 lines
The issue this patchset is trying to address is
mmap_lockcontention when a low priority task (monitoring, data collecting, etc.) blocks a higher priority task from making updated to the address space. The contention is due to themmap_lockbeing held for read when reading proc/pid/maps. Withmaple_treeintroduction, VMA tree traversals are RCU-safe and per-vma locks make VMA access RCU-safe. this provides an opportunity for lock-less reading of proc/pid/maps. We still need to overcome a couple obstacles:
1. Make all VMA pointer fields used for proc/pid/maps content generation RCU-safe;
2. Ensure that proc/pid/maps data tearing, which is currently possible at page boundaries only, does not get worse.
Hm I thought we were to only choose this more complicated in case additional tearing becomes a problem, and at first assume that if software can deal with page boundary tearing, it can deal with sub-page tearing too?
Quoted 4 lines
The patchset deals with these issues but there is a downside which I would like to get input on:
This change introduces unfairness towards the reader of proc/pid/maps, which can be blocked by an overly active/malicious address space modifyer.
So this is a consequence of the validate() operation, right? We could avoid this if we allowed sub-page tearing.
Quoted 31 lines
A couple of ways I though we can address this issue are:
1. After several lock-less retries (or some time limit) to fall back to takingmmap_lock.
2. Employ lock-less reading only if the reader has low priority, indicating that blocking it is not critical.
3. Introducing a separate procfs file which publishes the same data in lock-less manner.I imagine a combination of these approaches can also be employed. I would like to get feedback on this from the Linux community.
Note: mmap_read_lock/mmap_read_unlock sequence inside
validate_map()can be replaced with more efficiendrwsem_wait()proposed by Matthew in [1].[1] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (3):
mm: makevm_area_structanon_namefield RCU-safe
seq_file: addvalidate()operation toseq_operationsmm/maps: read proc/pid/maps under RCUfs/proc/internal.h | 3 + fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++---- fs/seq_file.c | 24 ++++++- include/linux/mm_inline.h | 10 ++- include/linux/mm_types.h | 3 +- include/linux/seq_file.h | 1 + mm/madvise.c | 30 +++++++-- 7 files changed, 181 insertions(+), 20 deletions(-)
Quoted 20 lines · Vlastimil Babka
On 1/15/24 19:38, Suren Baghdasaryan wrote:
Hi,
Quoted 11 lines
The issue this patchset is trying to address is
mmap_lockcontention when a low priority task (monitoring, data collecting, etc.) blocks a higher priority task from making updated to the address space. The contention is due to themmap_lockbeing held for read when reading proc/pid/maps. Withmaple_treeintroduction, VMA tree traversals are RCU-safe and per-vma locks make VMA access RCU-safe. this provides an opportunity for lock-less reading of proc/pid/maps. We still need to overcome a couple obstacles:
1. Make all VMA pointer fields used for proc/pid/maps content generation RCU-safe;
2. Ensure that proc/pid/maps data tearing, which is currently possible at page boundaries only, does not get worse.Hm I thought we were to only choose this more complicated in case additional tearing becomes a problem, and at first assume that if software can deal with page boundary tearing, it can deal with sub-page tearing too?
Quoted 15 lines
Quoted 4 lines
The patchset deals with these issues but there is a downside which I would like to get input on:
This change introduces unfairness towards the reader of proc/pid/maps, which can be blocked by an overly active/malicious address space modifyer.So this is a consequence of the
validate()operation, right? We could avoid this if we allowed sub-page tearing.Quoted 7 lines
A couple of ways I though we can address this issue are:
1. After several lock-less retries (or some time limit) to fall back to takingmmap_lock.
2. Employ lock-less reading only if the reader has low priority, indicating that blocking it is not critical.
3. Introducing a separate procfs file which publishes the same data in lock-less manner.
Oh and if this option 3 becomes necessary, then such new file shouldn't validate() either, and whoever wants to avoid the reader contention and converts their monitoring to the new file will have to account for this possible extra tearing from the start. So I would suggest trying to change the existing file with no validate() first, and if existing userspace gets broken, employ option 3. This would mean no validate() in either case?
Quoted 24 lines
Quoted 23 lines
I imagine a combination of these approaches can also be employed. I would like to get feedback on this from the Linux community.
Note: mmap_read_lock/mmap_read_unlock sequence inside
validate_map()can be replaced with more efficiendrwsem_wait()proposed by Matthew in [1].[1] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (3):
mm: makevm_area_structanon_namefield RCU-safe
seq_file: addvalidate()operation toseq_operationsmm/maps: read proc/pid/maps under RCUfs/proc/internal.h | 3 + fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++---- fs/seq_file.c | 24 ++++++- include/linux/mm_inline.h | 10 ++- include/linux/mm_types.h | 3 +- include/linux/seq_file.h | 1 + mm/madvise.c | 30 +++++++-- 7 files changed, 181 insertions(+), 20 deletions(-)
Suren Baghdasaryan
reviewerQuoted 18 lines · Vlastimil Babka
Quoted 16 lines · Vlastimil Babka
On 1/15/24 19:38, Suren Baghdasaryan wrote:
Hi,
Quoted 11 lines
The issue this patchset is trying to address is
mmap_lockcontention when a low priority task (monitoring, data collecting, etc.) blocks a higher priority task from making updated to the address space. The contention is due to themmap_lockbeing held for read when reading proc/pid/maps. Withmaple_treeintroduction, VMA tree traversals are RCU-safe and per-vma locks make VMA access RCU-safe. this provides an opportunity for lock-less reading of proc/pid/maps. We still need to overcome a couple obstacles:
1. Make all VMA pointer fields used for proc/pid/maps content generation RCU-safe;
2. Ensure that proc/pid/maps data tearing, which is currently possible at page boundaries only, does not get worse.
Quoted 3 lines
Quoted 3 lines
Hm I thought we were to only choose this more complicated in case additional tearing becomes a problem, and at first assume that if software can deal with page boundary tearing, it can deal with sub-page tearing too?
Hi Vlastimil,
Thanks for the feedback!
Yes, originally I thought we wouldn't be able to avoid additional tearing without a big change but then realized it's not that hard, so I tried to keep the change in behavior transparent to the userspace.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
The patchset deals with these issues but there is a downside which I would like to get input on:
This change introduces unfairness towards the reader of proc/pid/maps, which can be blocked by an overly active/malicious address space modifyer.So this is a consequence of the
validate()operation, right? We could avoid this if we allowed sub-page tearing.
Yes, if we don't care about sub-page tearing then we could get rid of validate step and this issue with updaters blocking the reader would go away. If we choose that direction there will be one more issue to fix, namely the maple_tree temporary inconsistent state when a VMA is replaced with another one and we might observe NULL there. We might be able to use Matthew's rwsem_wait() to deal with that issue.
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
A couple of ways I though we can address this issue are:
1. After several lock-less retries (or some time limit) to fall back to takingmmap_lock.
2. Employ lock-less reading only if the reader has low priority, indicating that blocking it is not critical.
3. Introducing a separate procfs file which publishes the same data in lock-less manner.
Quoted 6 lines
Oh and if this option 3 becomes necessary, then such new file shouldn't
validate()either, and whoever wants to avoid the reader contention and converts their monitoring to the new file will have to account for this possible extra tearing from the start. So I would suggest trying to change the existing file with novalidate()first, and if existing userspace gets broken, employ option 3. This would mean novalidate()in either case?
Yes but I was trying to avoid introducing additional file which publishes the same content in a slightly different way. We will have to explain when userspace should use one vs the other and that would require going into low level implementation details, I think. Don't know if that's acceptable/preferable.
Thanks,
Suren.
Quoted 26 lines
Quoted 25 lines
Quoted 24 lines
I imagine a combination of these approaches can also be employed. I would like to get feedback on this from the Linux community.
Note: mmap_read_lock/mmap_read_unlock sequence inside
validate_map()can be replaced with more efficiendrwsem_wait()proposed by Matthew in [1].[1] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (3):
mm: makevm_area_structanon_namefield RCU-safe
seq_file: addvalidate()operation toseq_operationsmm/maps: read proc/pid/maps under RCUfs/proc/internal.h | 3 + fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++---- fs/seq_file.c | 24 ++++++- include/linux/mm_inline.h | 10 ++- include/linux/mm_types.h | 3 +- include/linux/seq_file.h | 1 + mm/madvise.c | 30 +++++++-- 7 files changed, 181 insertions(+), 20 deletions(-)
Suren Baghdasaryan
reviewerQuoted 24 lines · Suren Baghdasaryan
Quoted 18 lines · Vlastimil Babka
Quoted 16 lines · Vlastimil Babka
On 1/15/24 19:38, Suren Baghdasaryan wrote:
Hi,
Quoted 11 lines
The issue this patchset is trying to address is
mmap_lockcontention when a low priority task (monitoring, data collecting, etc.) blocks a higher priority task from making updated to the address space. The contention is due to themmap_lockbeing held for read when reading proc/pid/maps. Withmaple_treeintroduction, VMA tree traversals are RCU-safe and per-vma locks make VMA access RCU-safe. this provides an opportunity for lock-less reading of proc/pid/maps. We still need to overcome a couple obstacles:
1. Make all VMA pointer fields used for proc/pid/maps content generation RCU-safe;
2. Ensure that proc/pid/maps data tearing, which is currently possible at page boundaries only, does not get worse.Quoted 3 lines
Quoted 3 lines
Hm I thought we were to only choose this more complicated in case additional tearing becomes a problem, and at first assume that if software can deal with page boundary tearing, it can deal with sub-page tearing too?
Quoted 5 lines
Hi Vlastimil,
Thanks for the feedback!
Yes, originally I thought we wouldn't be able to avoid additional tearing without a big change but then realized it's not that hard, so I tried to keep the change in behavior transparent to the userspace.
In the absence of other feedback I'm going to implement and post the originally envisioned approach: remove validation step and avoid any possibility of blocking but allowing for sub-page tearing. Will use Matthew's rwsem_wait() to deal with possible inconsistent maple_tree state.
Thanks,
Suren.
Quoted 67 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
The patchset deals with these issues but there is a downside which I would like to get input on:
This change introduces unfairness towards the reader of proc/pid/maps, which can be blocked by an overly active/malicious address space modifyer.So this is a consequence of the
validate()operation, right? We could avoid this if we allowed sub-page tearing.Yes, if we don't care about sub-page tearing then we could get rid of validate step and this issue with updaters blocking the reader would go away. If we choose that direction there will be one more issue to fix, namely the
maple_treetemporary inconsistent state when a VMA is replaced with another one and we might observe NULL there. We might be able to use Matthew'srwsem_wait()to deal with that issue.Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
A couple of ways I though we can address this issue are:
1. After several lock-less retries (or some time limit) to fall back to takingmmap_lock.
2. Employ lock-less reading only if the reader has low priority, indicating that blocking it is not critical.
3. Introducing a separate procfs file which publishes the same data in lock-less manner.Quoted 6 lines
Oh and if this option 3 becomes necessary, then such new file shouldn't
validate()either, and whoever wants to avoid the reader contention and converts their monitoring to the new file will have to account for this possible extra tearing from the start. So I would suggest trying to change the existing file with novalidate()first, and if existing userspace gets broken, employ option 3. This would mean novalidate()in either case?Yes but I was trying to avoid introducing additional file which publishes the same content in a slightly different way. We will have to explain when userspace should use one vs the other and that would require going into low level implementation details, I think. Don't know if that's acceptable/preferable.
Thanks,
Suren.Quoted 26 lines
Quoted 25 lines
Quoted 24 lines
I imagine a combination of these approaches can also be employed. I would like to get feedback on this from the Linux community.
Note: mmap_read_lock/mmap_read_unlock sequence inside
validate_map()can be replaced with more efficiendrwsem_wait()proposed by Matthew in [1].[1] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (3):
mm: makevm_area_structanon_namefield RCU-safe
seq_file: addvalidate()operation toseq_operationsmm/maps: read proc/pid/maps under RCUfs/proc/internal.h | 3 + fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++---- fs/seq_file.c | 24 ++++++- include/linux/mm_inline.h | 10 ++- include/linux/mm_types.h | 3 +- include/linux/seq_file.h | 1 + mm/madvise.c | 30 +++++++-- 7 files changed, 181 insertions(+), 20 deletions(-)
Suren Baghdasaryan
reviewerQuoted 32 lines · Suren Baghdasaryan
Quoted 24 lines · Suren Baghdasaryan
Quoted 18 lines · Vlastimil Babka
Quoted 16 lines · Vlastimil Babka
On 1/15/24 19:38, Suren Baghdasaryan wrote:
Hi,
Quoted 11 lines
The issue this patchset is trying to address is
mmap_lockcontention when a low priority task (monitoring, data collecting, etc.) blocks a higher priority task from making updated to the address space. The contention is due to themmap_lockbeing held for read when reading proc/pid/maps. Withmaple_treeintroduction, VMA tree traversals are RCU-safe and per-vma locks make VMA access RCU-safe. this provides an opportunity for lock-less reading of proc/pid/maps. We still need to overcome a couple obstacles:
1. Make all VMA pointer fields used for proc/pid/maps content generation RCU-safe;
2. Ensure that proc/pid/maps data tearing, which is currently possible at page boundaries only, does not get worse.Quoted 3 lines
Quoted 3 lines
Hm I thought we were to only choose this more complicated in case additional tearing becomes a problem, and at first assume that if software can deal with page boundary tearing, it can deal with sub-page tearing too?
Quoted 5 lines
Hi Vlastimil,
Thanks for the feedback!
Yes, originally I thought we wouldn't be able to avoid additional tearing without a big change but then realized it's not that hard, so I tried to keep the change in behavior transparent to the userspace.
Quoted 5 lines
In the absence of other feedback I'm going to implement and post the originally envisioned approach: remove validation step and avoid any possibility of blocking but allowing for sub-page tearing. Will use Matthew's
rwsem_wait()to deal with possible inconsistentmaple_treestate.
I posted v1 at
https://lore.kernel.org/all/[email protected]/
In the RFC I used mm_struct.mm_lock_seq to detect if mm is being changed but then I realized that won't work. mm_struct.mm_lock_seq is incremented after mm is changed and right before mmap_lock is write-unlocked. Instead I need a counter that changes once we write-lock mmap_lock and before any mm changes. So the new patchset introduces a separate counter to detect possible mm changes. In addition, I could not use rwsem_wait() and instead had to take mmap_lock for read to wait for the writer to finish and then record the new counter while holding mmap_lock for read. That prevents concurrent mm changes while we are recording the new counter value.
Quoted 70 lines
Thanks,
Suren.Quoted 67 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
The patchset deals with these issues but there is a downside which I would like to get input on:
This change introduces unfairness towards the reader of proc/pid/maps, which can be blocked by an overly active/malicious address space modifyer.So this is a consequence of the
validate()operation, right? We could avoid this if we allowed sub-page tearing.Yes, if we don't care about sub-page tearing then we could get rid of validate step and this issue with updaters blocking the reader would go away. If we choose that direction there will be one more issue to fix, namely the
maple_treetemporary inconsistent state when a VMA is replaced with another one and we might observe NULL there. We might be able to use Matthew'srwsem_wait()to deal with that issue.Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
A couple of ways I though we can address this issue are:
1. After several lock-less retries (or some time limit) to fall back to takingmmap_lock.
2. Employ lock-less reading only if the reader has low priority, indicating that blocking it is not critical.
3. Introducing a separate procfs file which publishes the same data in lock-less manner.Quoted 6 lines
Oh and if this option 3 becomes necessary, then such new file shouldn't
validate()either, and whoever wants to avoid the reader contention and converts their monitoring to the new file will have to account for this possible extra tearing from the start. So I would suggest trying to change the existing file with novalidate()first, and if existing userspace gets broken, employ option 3. This would mean novalidate()in either case?Yes but I was trying to avoid introducing additional file which publishes the same content in a slightly different way. We will have to explain when userspace should use one vs the other and that would require going into low level implementation details, I think. Don't know if that's acceptable/preferable.
Thanks,
Suren.Quoted 26 lines
Quoted 25 lines
Quoted 24 lines
I imagine a combination of these approaches can also be employed. I would like to get feedback on this from the Linux community.
Note: mmap_read_lock/mmap_read_unlock sequence inside
validate_map()can be replaced with more efficiendrwsem_wait()proposed by Matthew in [1].[1] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (3):
mm: makevm_area_structanon_namefield RCU-safe
seq_file: addvalidate()operation toseq_operationsmm/maps: read proc/pid/maps under RCUfs/proc/internal.h | 3 + fs/proc/task_mmu.c | 130 ++++++++++++++++++++++++++++++++++---- fs/seq_file.c | 24 ++++++- include/linux/mm_inline.h | 10 ++- include/linux/mm_types.h | 3 +- include/linux/seq_file.h | 1 + mm/madvise.c | 30 +++++++-- 7 files changed, 181 insertions(+), 20 deletions(-)