perf: Convert mmap() related reference counts to refcount_t
Peter Zijlstra
maintainerv3 cover perf: Convert mmap() related reference counts to refcount_t
Took over the series from Thomas; much thanks to him for cleaning this up.
The recently fixed reference count leaks could have been detected by using refcount_t and refcount_t would have mitigated the potential overflow at least.
It turned out that converting the code as is does not work as the allocation code ends up doing a refcount_inc() for the first allocation, which causes refcount_t sanity checks to emit a UAF warning.
The reason is that the code is sharing functionality at the wrong level and ends up being overly complicated for no reason. That's what inevitable led to the refcount leak problems.
Address this by splitting the ringbuffer and the AUX buffer mapping and allocation parts out into seperate functions, which handle the reference counts in a sane way.
That not only simplifies the code and makes it halfways comprehensible, but also allows to convert the mmap() related reference counts to refcount_t.
It survives lightweight testing with perf and passes the perf/mmap selftest.
Also available at:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git perf/core
---
v2: https://lkml.kernel.org/r/[email protected]
Changes vs v2:
- replaced patches 4,5 with fine grained steps
Peter Zijlstra
maintainerPATCH v3 1/15 perf: Remove redundant condition for AUX buffer size
From: Thomas Gleixner <[email protected]>
It is already checked whether the VMA size is the same as nr_pages * PAGE_SIZE, so later checking both:
aux_size == vma_size && aux_size == nr_pages * PAGE_SIZE
is redundant. Remove the vma_size check as nr_pages is what is actually used in the allocation function. That prepares for splitting out the buffer allocation into seperate functions, so that only nr_pages needs to be handed in.
No functional change.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -7043,7 +7043,7 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) goto aux_unlock; -if (aux_size != vma_size || aux_size != nr_pages *PAGE_SIZE ) +if (aux_size != nr_pages *PAGE_SIZE ) goto aux_unlock;/* already mapped with a different size */
Re: [PATCH v3 01/15] perf: Remove redundant condition for AUX buffer size
Quoted 10 lines · Peter Zijlstra
From: Thomas Gleixner <[email protected]>
It is already checked whether the VMA size is the same as
nr_pages*PAGE_SIZE, so later checking both:aux_size == vma_size && aux_size == nr_pages * PAGE_SIZEis redundant. Remove the
vma_sizecheck asnr_pagesis what is actually used in the allocation function. That prepares for splitting out the buffer allocation into seperate functions, so that onlynr_pagesneeds to be
NIT: still get seperate -> separate typo there. Not a big deal obviously :)
Quoted 25 lines
handed in.
No functional change.
Signed-off-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/events/core.c +1
-1@@ -7043,7 +7043,7 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) goto aux_unlock; -if (aux_size != vma_size || aux_size != nr_pages *PAGE_SIZE ) +if (aux_size != nr_pages *PAGE_SIZE ) goto aux_unlock;/* already mapped with a different size */
[tip: perf/core] perf: Remove redundant condition for AUX buffer size
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: e8c4f6ee8eeed8e02800bed6afb9aa22fc3476a1 Gitweb: https://git.kernel.org/tip/e8c4f6ee8eeed8e02800bed6afb9aa22fc3476a1 Author: Thomas Gleixner <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:38:59 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:12:58 +02:00
perf: Remove redundant condition for AUX buffer size
It is already checked whether the VMA size is the same as nr_pages * PAGE_SIZE, so later checking both:
aux_size == vma_size && aux_size == nr_pages * PAGE_SIZE
is redundant. Remove the vma_size check as nr_pages is what is actually used in the allocation function. That prepares for splitting out the buffer allocation into separate functions, so that only nr_pages needs to be handed in.
No functional change.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -7043,7 +7043,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) goto aux_unlock; -if (aux_size != vma_size || aux_size != nr_pages *PAGE_SIZE ) +if (aux_size != nr_pages *PAGE_SIZE ) goto aux_unlock;/* already mapped with a different size */
Peter Zijlstra
maintainerPATCH v3 2/15 perf: Split out mlock limit handling
In reply to Peter ZijlstraFrom: Thomas Gleixner <[email protected]>
To prepare for splitting the buffer allocation out into seperate functions for the ring buffer and the AUX buffer, split out mlock limit handling into a helper function, which can be called from both.
No functional change intended.
--- kernel/events/core.c | 77 +++++++++++++++++++++++++-------------------------- 1 file changed, 38 insertions(+), 39 deletions(-)
kernel/events/core.c +38 -37
@@ -6927,17 +6927,49 @@ static int map_range(struct perf_bufferreturn err; } +static bool perf_mmap_calc_limits (struct vm_area_struct *vma,long *user_extra,long *extra) +{ +unsigned long user_locked, user_lock_limit, locked, lock_limit; +struct user_struct *user =current_user (); + + user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 ); +/* Increase the limit linearly with more CPUs */ + user_lock_limit *=num_online_cpus (); + + user_locked =atomic_long_read (&user->locked_vm); + +/* +* sysctl_perf_event_mlock may have changed, so that +* user->locked_vm > user_lock_limit +*/ +if (user_locked > user_lock_limit) + user_locked = user_lock_limit; + user_locked += *user_extra; + +if (user_locked > user_lock_limit) { +/* +* charge locked_vm until it hits user_lock_limit; +* charge the rest from pinned_vm +*/ + *extra = user_locked - user_lock_limit; + *user_extra -= *extra; + } + + lock_limit =rlimit (RLIMIT_MEMLOCK ); + lock_limit >>=PAGE_SHIFT ; + locked =atomic64_read (&vma->vm_mm->pinned_vm) + *extra; + +return locked <= lock_limit || !perf_is_paranoid () ||capable (CAP_IPC_LOCK ); +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data; -unsigned long user_locked, user_lock_limit;struct user_struct *user =current_user (); +unsigned long vma_size, nr_pages; +long user_extra =0 , extra =0 ;struct mutex *aux_mutex =NULL ;struct perf_buffer *rb =NULL ; -unsigned long locked, lock_limit; -unsigned long vma_size; -unsigned long nr_pages; -long user_extra =0 , extra =0 ;int ret, flags =0 ;mapped_f mapped;
@@ -7063,38 +7093,7 @@ static int perf_mmap(struct file *file, } } - user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 ); - -/* -* Increase the limit linearly with more CPUs: -*/ - user_lock_limit *=num_online_cpus (); - - user_locked =atomic_long_read (&user->locked_vm); - -/* -* sysctl_perf_event_mlock may have changed, so that -* user->locked_vm > user_lock_limit -*/ -if (user_locked > user_lock_limit) - user_locked = user_lock_limit; - user_locked += user_extra; - -if (user_locked > user_lock_limit) { -/* -* charge locked_vm until it hits user_lock_limit; -* charge the rest from pinned_vm -*/ - extra = user_locked - user_lock_limit; - user_extra -= extra; - } - - lock_limit =rlimit (RLIMIT_MEMLOCK ); - lock_limit >>=PAGE_SHIFT ; - locked =atomic64_read (&vma->vm_mm->pinned_vm) + extra; - -if ((locked > lock_limit) &&perf_is_paranoid () && - !capable (CAP_IPC_LOCK )) { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; goto unlock; }
Re: [PATCH v3 02/15] perf: Split out mlock limit handling
Quoted 3 lines · Peter Zijlstra
From: Thomas Gleixner <[email protected]>
To prepare for splitting the buffer allocation out into seperate functions
NIT: Same comment as 1/2, seperate -> separate, again doesn't hugely matter but just FYI!
Quoted 112 lines
for the ring buffer and the AUX buffer, split out mlock limit handling into a helper function, which can be called from both.
No functional change intended.
Signed-off-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>--- kernel/events/core.c | 77 +++++++++++++++++++++++++-------------------------- 1 file changed, 38 insertions(+), 39 deletions(-)kernel/events/core.c +38
-37@@ -6927,17 +6927,49 @@ static int map_range(struct perf_bufferreturn err; } +static bool perf_mmap_calc_limits (struct vm_area_struct *vma,long *user_extra,long *extra) +{ +unsigned long user_locked, user_lock_limit, locked, lock_limit; +struct user_struct *user =current_user (); + + user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 ); +/* Increase the limit linearly with more CPUs */ + user_lock_limit *=num_online_cpus (); + + user_locked =atomic_long_read (&user->locked_vm); + +/* +* sysctl_perf_event_mlock may have changed, so that +* user->locked_vm > user_lock_limit +*/ +if (user_locked > user_lock_limit) + user_locked = user_lock_limit; + user_locked += *user_extra; + +if (user_locked > user_lock_limit) { +/* +* charge locked_vm until it hits user_lock_limit; +* charge the rest from pinned_vm +*/ + *extra = user_locked - user_lock_limit; + *user_extra -= *extra; + } + + lock_limit =rlimit (RLIMIT_MEMLOCK ); + lock_limit >>=PAGE_SHIFT ; + locked =atomic64_read (&vma->vm_mm->pinned_vm) + *extra; + +return locked <= lock_limit || !perf_is_paranoid () ||capable (CAP_IPC_LOCK ); +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data; -unsigned long user_locked, user_lock_limit;struct user_struct *user =current_user (); +unsigned long vma_size, nr_pages; +long user_extra =0 , extra =0 ;struct mutex *aux_mutex =NULL ;struct perf_buffer *rb =NULL ; -unsigned long locked, lock_limit; -unsigned long vma_size; -unsigned long nr_pages; -long user_extra =0 , extra =0 ;int ret, flags =0 ;mapped_f mapped;@@ -7063,38 +7093,7 @@ static int perf_mmap(struct file *file, } } - user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 ); - -/* -* Increase the limit linearly with more CPUs: -*/ - user_lock_limit *=num_online_cpus (); - - user_locked =atomic_long_read (&user->locked_vm); - -/* -* sysctl_perf_event_mlock may have changed, so that -* user->locked_vm > user_lock_limit -*/ -if (user_locked > user_lock_limit) - user_locked = user_lock_limit; - user_locked += user_extra; - -if (user_locked > user_lock_limit) { -/* -* charge locked_vm until it hits user_lock_limit; -* charge the rest from pinned_vm -*/ - extra = user_locked - user_lock_limit; - user_extra -= extra; - } - - lock_limit =rlimit (RLIMIT_MEMLOCK ); - lock_limit >>=PAGE_SHIFT ; - locked =atomic64_read (&vma->vm_mm->pinned_vm) + extra; - -if ((locked > lock_limit) &&perf_is_paranoid () && - !capable (CAP_IPC_LOCK )) { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; goto unlock; }
[tip: perf/core] perf: Split out mlock limit handling
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 81e026ca47b386e4213c1beff069038a3ba8bb76 Gitweb: https://git.kernel.org/tip/81e026ca47b386e4213c1beff069038a3ba8bb76 Author: Thomas Gleixner <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:00 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:12:58 +02:00
perf: Split out mlock limit handling
To prepare for splitting the buffer allocation out into separate functions for the ring buffer and the AUX buffer, split out mlock limit handling into a helper function, which can be called from both.
No functional change intended.
--- kernel/events/core.c | 75 +++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 37 deletions(-)
kernel/events/core.c +38 -37
@@ -6927,17 +6927,49 @@ static int map_range(struct perf_buffer *rb, struct vm_area_struct *vma)return err; } +static bool perf_mmap_calc_limits (struct vm_area_struct *vma,long *user_extra,long *extra) +{ +unsigned long user_locked, user_lock_limit, locked, lock_limit; +struct user_struct *user =current_user (); + + user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 ); +/* Increase the limit linearly with more CPUs */ + user_lock_limit *=num_online_cpus (); + + user_locked =atomic_long_read (&user->locked_vm); + +/* +* sysctl_perf_event_mlock may have changed, so that +* user->locked_vm > user_lock_limit +*/ +if (user_locked > user_lock_limit) + user_locked = user_lock_limit; + user_locked += *user_extra; + +if (user_locked > user_lock_limit) { +/* +* charge locked_vm until it hits user_lock_limit; +* charge the rest from pinned_vm +*/ + *extra = user_locked - user_lock_limit; + *user_extra -= *extra; + } + + lock_limit =rlimit (RLIMIT_MEMLOCK ); + lock_limit >>=PAGE_SHIFT ; + locked =atomic64_read (&vma->vm_mm->pinned_vm) + *extra; + +return locked <= lock_limit || !perf_is_paranoid () ||capable (CAP_IPC_LOCK ); +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data; -unsigned long user_locked, user_lock_limit;struct user_struct *user =current_user (); +unsigned long vma_size, nr_pages; +long user_extra =0 , extra =0 ;struct mutex *aux_mutex =NULL ;struct perf_buffer *rb =NULL ; -unsigned long locked, lock_limit; -unsigned long vma_size; -unsigned long nr_pages; -long user_extra =0 , extra =0 ;int ret, flags =0 ;mapped_f mapped;
@@ -7063,38 +7095,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) } } - user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 ); - -/* -* Increase the limit linearly with more CPUs: -*/ - user_lock_limit *=num_online_cpus (); - - user_locked =atomic_long_read (&user->locked_vm); - -/* -* sysctl_perf_event_mlock may have changed, so that -* user->locked_vm > user_lock_limit -*/ -if (user_locked > user_lock_limit) - user_locked = user_lock_limit; - user_locked += user_extra; - -if (user_locked > user_lock_limit) { -/* -* charge locked_vm until it hits user_lock_limit; -* charge the rest from pinned_vm -*/ - extra = user_locked - user_lock_limit; - user_extra -= extra; - } - - lock_limit =rlimit (RLIMIT_MEMLOCK ); - lock_limit >>=PAGE_SHIFT ; - locked =atomic64_read (&vma->vm_mm->pinned_vm) + extra; - -if ((locked > lock_limit) &&perf_is_paranoid () && - !capable (CAP_IPC_LOCK )) { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; goto unlock; }
Peter Zijlstra
maintainerPATCH v3 3/15 perf: Split out VM accounting
In reply to Peter ZijlstraFrom: Thomas Gleixner <[email protected]>
Similary to the mlock limit calculation the VM accounting is required for both the ringbuffer and the AUX buffer allocations.
To prepare for splitting them out into seperate functions, move the accounting into a helper function.
--- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/events/core.c +9 -4
@@ -6962,10 +6962,17 @@ static bool perf_mmap_calc_limits(structreturn locked <= lock_limit || !perf_is_paranoid () ||capable (CAP_IPC_LOCK ); } +static void perf_mmap_account (struct vm_area_struct *vma,long user_extra,long extra) +{ +struct user_struct *user =current_user (); + +atomic_long_add (user_extra, &user->locked_vm); +atomic64_add (extra, &vma->vm_mm->pinned_vm); +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data; -struct user_struct *user =current_user ();unsigned long vma_size, nr_pages;long user_extra =0 , extra =0 ;struct mutex *aux_mutex =NULL ;
@@ -7136,9 +7143,7 @@ static int perf_mmap(struct file *file, unlock:if (!ret) { -atomic_long_add (user_extra, &user->locked_vm); -atomic64_add (extra, &vma->vm_mm->pinned_vm); - +perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }else if (rb) {/* AUX allocation failed */
Re: [PATCH v3 03/15] perf: Split out VM accounting
Quoted 3 lines · Peter Zijlstra
From: Thomas Gleixner <[email protected]>
Similary to the mlock limit calculation the VM accounting is required for
NIT: Similary -> Similarly
Quoted 3 lines
both the ringbuffer and the AUX buffer allocations.
To prepare for splitting them out into seperate functions, move the
NIT: Seperate -> Separate
And that's the end of the typo nits ;)
Quoted 44 lines
accounting into a helper function.
Signed-off-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>--- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)kernel/events/core.c +9
-4@@ -6962,10 +6962,17 @@ static bool perf_mmap_calc_limits(structreturn locked <= lock_limit || !perf_is_paranoid () ||capable (CAP_IPC_LOCK ); } +static void perf_mmap_account (struct vm_area_struct *vma,long user_extra,long extra) +{ +struct user_struct *user =current_user (); + +atomic_long_add (user_extra, &user->locked_vm); +atomic64_add (extra, &vma->vm_mm->pinned_vm); +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data; -struct user_struct *user =current_user ();unsigned long vma_size, nr_pages;long user_extra =0 , extra =0 ;struct mutex *aux_mutex =NULL ;@@ -7136,9 +7143,7 @@ static int perf_mmap(struct file *file, unlock:if (!ret) { -atomic_long_add (user_extra, &user->locked_vm); -atomic64_add (extra, &vma->vm_mm->pinned_vm); - +perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }else if (rb) {/* AUX allocation failed */
[tip: perf/core] perf: Split out VM accounting
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 1ea3e3b0dadc06c5e6c1bdf5312e70ee861b1ba0 Gitweb: https://git.kernel.org/tip/1ea3e3b0dadc06c5e6c1bdf5312e70ee861b1ba0 Author: Thomas Gleixner <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:01 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:12:59 +02:00
perf: Split out VM accounting
Similarly to the mlock limit calculation the VM accounting is required for both the ringbuffer and the AUX buffer allocations.
To prepare for splitting them out into separate functions, move the accounting into a helper function.
--- kernel/events/core.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/events/core.c +9 -4
@@ -6962,10 +6962,17 @@ static bool perf_mmap_calc_limits(struct vm_area_struct *vma, long *user_extra,return locked <= lock_limit || !perf_is_paranoid () ||capable (CAP_IPC_LOCK ); } +static void perf_mmap_account (struct vm_area_struct *vma,long user_extra,long extra) +{ +struct user_struct *user =current_user (); + +atomic_long_add (user_extra, &user->locked_vm); +atomic64_add (extra, &vma->vm_mm->pinned_vm); +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data; -struct user_struct *user =current_user ();unsigned long vma_size, nr_pages;long user_extra =0 , extra =0 ;struct mutex *aux_mutex =NULL ;
@@ -7136,9 +7143,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) unlock:if (!ret) { -atomic_long_add (user_extra, &user->locked_vm); -atomic64_add (extra, &vma->vm_mm->pinned_vm); - +perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }else if (rb) {/* AUX allocation failed */
Peter Zijlstra
maintainerPATCH v3 4/15 perf: Move perf_mmap_calc_limits() into both rb and aux branches
In reply to Peter Zijlstra if (cond) {
A;
} else {
B;
}
C;into
if (cond) {
A;
C;
} else {
B;
C;
}--- kernel/events/core.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
kernel/events/core.c +18 -8
@@ -7054,6 +7054,16 @@ static int perf_mmap(struct file *file,ring_buffer_attach (event,NULL ); } +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; + goto unlock; + } + +WARN_ON (!rb && event->rb); + +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; + }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -7100,17 +7110,17 @@ static int perf_mmap(struct file *file, ret =0 ; goto unlock; } - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; - goto unlock; - } +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; + }if (!rb) { rb = rb_alloc(nr_pages,
Re: [PATCH v3 04/15] perf: Move perf_mmap_calc_limits() into both rb and aux branches
Quoted 17 lines · Peter Zijlstra
if (cond) { A; } else { B; } C;into
if (cond) { A; C; } else { B; C; }
Quoted 2 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Seems sensible enough, guessing trade off on duplication vs. complexity of having to keep track of 'C' being in place for both and I assume makes later refactorings easier :)
Quoted 51 lines
--- kernel/events/core.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)kernel/events/core.c +18
-8@@ -7054,6 +7054,16 @@ static int perf_mmap(struct file *file,ring_buffer_attach (event,NULL ); } +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; + goto unlock; + } + +WARN_ON (!rb && event->rb); + +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; + }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already @@ -7100,17 +7110,17 @@ static int perf_mmap(struct file *file, ret =0 ; goto unlock; } - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; - goto unlock; - } +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; + }if (!rb) { rb = rb_alloc(nr_pages,
[tip: perf/core] perf: Move perf_mmap_calc_limits() into both rb and aux branches
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 86a0a7c59845e7093c9c73a7115c9d86349499d1 Gitweb: https://git.kernel.org/tip/86a0a7c59845e7093c9c73a7115c9d86349499d1 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:02 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:12:59 +02:00
perf: Move perf_mmap_calc_limits() into both rb and aux branches
if (cond) {
A;
} else {
B;
}
C;into
if (cond) {
A;
C;
} else {
B;
C;
}--- kernel/events/core.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
kernel/events/core.c +18 -8
@@ -7054,6 +7054,16 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)ring_buffer_attach (event,NULL ); } +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; + goto unlock; + } + +WARN_ON (!rb && event->rb); + +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; + }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -7100,17 +7110,17 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) ret =0 ; goto unlock; } - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; - goto unlock; - } +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; + }if (!rb) { rb = rb_alloc(nr_pages,
Peter Zijlstra
maintainerPATCH v3 5/15 perf: Merge consecutive conditionals in perf_mmap()
In reply to Peter Zijlstra if (cond) {
A;
} else {
B;
}
if (cond) {
C;
} else {
D;
}into:
if (cond) {
A;
C;
} else {
B;
D;
}Notably the conditions are not identical in form, but are equivalent.
--- kernel/events/core.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-)
kernel/events/core.c +19 -22
@@ -7064,6 +7064,25 @@ static int perf_mmap(struct file *file,if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; + rb =rb_alloc (nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark :0 , + event->cpu, flags); + +if (!rb) { + ret = -ENOMEM ; + goto unlock; + } + +atomic_set (&rb->mmap_count,1 ); + rb->mmap_user =get_current_user (); + rb->mmap_locked = extra; + +ring_buffer_attach (event, rb); + +perf_event_update_time (event); +perf_event_init_userpage (event); +perf_event_update_userpage (event); + ret =0 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -7120,29 +7139,7 @@ static int perf_mmap(struct file *file,if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; - } -if (!rb) { - rb =rb_alloc (nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); - -if (!rb) { - ret = -ENOMEM ; - goto unlock; - } - -atomic_set (&rb->mmap_count,1 ); - rb->mmap_user =get_current_user (); - rb->mmap_locked = extra; - -ring_buffer_attach (event, rb); - -perf_event_update_time (event); -perf_event_init_userpage (event); -perf_event_update_userpage (event); - ret =0 ; - }else { ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags);if (!ret) {
Re: [PATCH v3 05/15] perf: Merge consecutive conditionals in perf_mmap()
Quoted 22 lines · Peter Zijlstra
if (cond) { A; } else { B; } if (cond) { C; } else { D; }into:
if (cond) { A; C; } else { B; D; }
Quoted 1 line
Notably the conditions are not identical in form, but are equivalent.
Good to point out.
Quoted 3 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Seems logical and reduces complexity, so LGTM and:
Quoted 64 lines
--- kernel/events/core.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-)kernel/events/core.c +19
-22@@ -7064,6 +7064,25 @@ static int perf_mmap(struct file *file,if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; + rb =rb_alloc (nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark :0 , + event->cpu, flags); + +if (!rb) { + ret = -ENOMEM ; + goto unlock; + } + +atomic_set (&rb->mmap_count,1 ); + rb->mmap_user =get_current_user (); + rb->mmap_locked = extra; + +ring_buffer_attach (event, rb); + +perf_event_update_time (event); +perf_event_init_userpage (event); +perf_event_update_userpage (event); + ret =0 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already @@ -7120,29 +7139,7 @@ static int perf_mmap(struct file *file,if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; - } -if (!rb) { - rb =rb_alloc (nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); - -if (!rb) { - ret = -ENOMEM ; - goto unlock; - } - -atomic_set (&rb->mmap_count,1 ); - rb->mmap_user =get_current_user (); - rb->mmap_locked = extra; - -ring_buffer_attach (event, rb); - -perf_event_update_time (event); -perf_event_init_userpage (event); -perf_event_update_userpage (event); - ret =0 ; - }else { ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags);if (!ret) {
[tip: perf/core] perf: Merge consecutive conditionals in perf_mmap()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 3821f258686691cf12bbfc636ab22fa2b049dc86 Gitweb: https://git.kernel.org/tip/3821f258686691cf12bbfc636ab22fa2b049dc86 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:03 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:12:59 +02:00
perf: Merge consecutive conditionals in perf_mmap()
if (cond) {
A;
} else {
B;
}
if (cond) {
C;
} else {
D;
}into:
if (cond) {
A;
C;
} else {
B;
D;
}Notably the conditions are not identical in form, but are equivalent.
--- kernel/events/core.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-)
kernel/events/core.c +19 -22
@@ -7064,6 +7064,25 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; + rb =rb_alloc (nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark :0 , + event->cpu, flags); + +if (!rb) { + ret = -ENOMEM ; + goto unlock; + } + +atomic_set (&rb->mmap_count,1 ); + rb->mmap_user =get_current_user (); + rb->mmap_locked = extra; + +ring_buffer_attach (event, rb); + +perf_event_update_time (event); +perf_event_init_userpage (event); +perf_event_update_userpage (event); + ret =0 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -7120,29 +7139,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ; - } -if (!rb) { - rb =rb_alloc (nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); - -if (!rb) { - ret = -ENOMEM ; - goto unlock; - } - -atomic_set (&rb->mmap_count,1 ); - rb->mmap_user =get_current_user (); - rb->mmap_locked = extra; - -ring_buffer_attach (event, rb); - -perf_event_update_time (event); -perf_event_init_userpage (event); -perf_event_update_userpage (event); - ret =0 ; - }else { ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags);if (!ret) {
Peter Zijlstra
maintainerPATCH v3 6/15 perf: Move common code into both rb and aux branches
In reply to Peter Zijlstra if (cond) {
A;
} else {
B;
}
C;into
if (cond) {
A;
C;
} else {
B;
C;
}--- kernel/events/core.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)
kernel/events/core.c +15 -10
@@ -7043,6 +7043,8 @@ static int perf_mmap(struct file *file, ret =0 ;/* We need the rb to map pages. */ rb = event->rb; +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); goto unlock; }
@@ -7083,6 +7085,9 @@ static int perf_mmap(struct file *file,perf_event_init_userpage (event);perf_event_update_userpage (event); ret =0 ; + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -7127,11 +7132,12 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto unlock; + goto aux_success; }if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; +atomic_dec (&rb->mmap_count); goto unlock; }
@@ -7142,20 +7148,19 @@ static int perf_mmap(struct file *file, ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); -if (!ret) { -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; +if (ret) { +atomic_dec (&rb->mmap_count); + goto unlock; } - } -unlock: -if (!ret) { +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; +aux_success:perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); - }else if (rb) { -/* AUX allocation failed */ -atomic_dec (&rb->mmap_count); } + +unlock: aux_unlock:if (aux_mutex)mutex_unlock (aux_mutex);
Re: [PATCH v3 06/15] perf: Move common code into both rb and aux branches
Quoted 9 lines · Peter Zijlstra
if (cond) { A; } else { B; } C;into
Quoted 8 lines
if (cond) {
A;
C;
} else {
B;
C;
}Hm we're doing more than that here though, we're refactoring other stuff at the same time.
I guess you're speaking broad strokes here, but maybe worth mentioniing the tricksy hobbitses around the rb_has_aux() bit.
Anyway LGTM so:
Quoted 42 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/events/core.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)kernel/events/core.c +7
-1@@ -7043,6 +7043,8 @@ static int perf_mmap(struct file *file, ret =0 ;/* We need the rb to map pages. */ rb = event->rb; +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); goto unlock; }@@ -7083,6 +7085,9 @@ static int perf_mmap(struct file *file,perf_event_init_userpage (event);perf_event_update_userpage (event); ret =0 ; + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already @@ -7127,11 +7132,12 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto unlock; + goto aux_success; }if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; +atomic_dec (&rb->mmap_count); goto unlock; }
kernel/events/core.c static int perf_mmap(struct file *file, · line 7151
Quoted 7 lines
kernel/events/core.c +1
-3@@ -7142,20 +7148,19 @@ static int perf_mmap(struct file *file, ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); -if (!ret) { -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; +if (ret) {
You dropped the 'AUX allocation failed' comment here, but honestly I think that's fine, it's pretty obvious that's the case given the literal previous line is you trying the AUX allocation... :)
Quoted 9 lines
kernel/events/core.c +4
-3+atomic_dec (&rb->mmap_count); + goto unlock; } - } -unlock: -if (!ret) { +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra;
kernel/events/core.c static int perf_mmap(struct file *file, · line 7164
Quoted 10 lines
kernel/events/core.c +3
-3+aux_success:perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); - }else if (rb) { -/* AUX allocation failed */ -atomic_dec (&rb->mmap_count); } + +unlock: aux_unlock:
Hm, this sucks, I assume this is a temporary thing to reduce churn above and effective prove that the code paths are equivalently going to the same place?
Which, given the complexity of the code, and the enormous ease with which you can miss stuff (from personal experience...!), this is probably a sensible way of doing it.
Quoted 4 lines
kernel/events/core.c +0
-0if (aux_mutex)mutex_unlock (aux_mutex);
Peter Zijlstra
maintainerRe: [PATCH v3 06/15] perf: Move common code into both rb and aux branches
Quoted 19 lines · Lorenzo Stoakes
Quoted 9 lines · Peter Zijlstra
if (cond) { A; } else { B; } C;into
Quoted 8 lines
if (cond) { A; C; } else { B; C; }
Quoted 5 lines
Hm we're doing more than that here though, we're refactoring other stuff at the same time.
I guess you're speaking broad strokes here, but maybe worth mentioniing the tricksy hobbitses around the
rb_has_aux()bit.
Does something like so clarify:
Notably C has a success branch and both A and B have two places for success. For A (rb case), duplicate the success case because later patches will result in them no longer being identical. For B (aux case), share using goto (cleaned up later).
Quoted 47 lines
Anyway LGTM so:
Reviewed-by Lorenzo Stoakes <[email protected]>Quoted 43 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/events/core.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)kernel/events/core.c +7
-1@@ -7043,6 +7043,8 @@ static int perf_mmap(struct file *file, ret =0 ;/* We need the rb to map pages. */ rb = event->rb; +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); goto unlock; }@@ -7083,6 +7085,9 @@ static int perf_mmap(struct file *file,perf_event_init_userpage (event);perf_event_update_userpage (event); ret =0 ; + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already @@ -7127,11 +7132,12 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto unlock; + goto aux_success; }if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; +atomic_dec (&rb->mmap_count); goto unlock; }@@ -7142,20 +7148,19 @@ static int perf_mmap(struct file *file,
Quoted 10 lines
kernel/events/core.c static int perf_mmap(struct file *file, · line 7151
Quoted 6 lines
kernel/events/core.c +1
-3ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); -if (!ret) { -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; +if (ret) {You dropped the 'AUX allocation failed' comment here, but honestly I think that's fine, it's pretty obvious that's the case given the literal previous line is you trying the AUX allocation... :)
Yes that :-)
Re: [PATCH v3 06/15] perf: Move common code into both rb and aux branches
Quoted 26 lines · Peter Zijlstra
Quoted 19 lines · Lorenzo Stoakes
Quoted 9 lines · Peter Zijlstra
if (cond) { A; } else { B; } C;into
Quoted 8 lines
if (cond) { A; C; } else { B; C; }Quoted 5 lines
Hm we're doing more than that here though, we're refactoring other stuff at the same time.
I guess you're speaking broad strokes here, but maybe worth mentioniing the tricksy hobbitses around the
rb_has_aux()bit.
Quoted 6 lines
Does something like so clarify:
Notably C has a success branch and both A and B have two places for success. For A (rb case), duplicate the success case because later patches will result in them no longer being identical. For B (aux case), share using goto (cleaned up later).
Sounds great, thanks!
Quoted 10 lines
Quoted 10 lines
Anyway LGTM so:
Reviewed-by Lorenzo Stoakes <[email protected]>Quoted 5 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/events/core.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)
[snip]
Quoted 13 lines
Quoted 12 lines
kernel/events/core.c static int perf_mmap(struct file *file, · line 7151
Quoted 8 lines
kernel/events/core.c +1
-3@@ -7142,20 +7148,19 @@ static int perf_mmap(struct file *file, ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); -if (!ret) { -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; +if (ret) {You dropped the 'AUX allocation failed' comment here, but honestly I think that's fine, it's pretty obvious that's the case given the literal previous line is you trying the AUX allocation... :)
Quoted 1 line
Yes that :-)
:)
[tip: perf/core] perf: Move common code into both rb and aux branches
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 4118994b33bb628dd9aeb941c5af6f950f1dea90 Gitweb: https://git.kernel.org/tip/4118994b33bb628dd9aeb941c5af6f950f1dea90 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:04 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:12:59 +02:00
perf: Move common code into both rb and aux branches
if (cond) {
A;
} else {
B;
}
C;into
if (cond) {
A;
C;
} else {
B;
C;
}Notably C has a success branch and both A and B have two places for success. For A (rb case), duplicate the success case because later patches will result in them no longer being identical. For B (aux case), share using goto (cleaned up later).
--- kernel/events/core.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)
kernel/events/core.c +15 -10
@@ -7043,6 +7043,8 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) ret =0 ;/* We need the rb to map pages. */ rb = event->rb; +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); goto unlock; }
@@ -7083,6 +7085,9 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)perf_event_init_userpage (event);perf_event_update_userpage (event); ret =0 ; + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -7127,11 +7132,12 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto unlock; + goto aux_success; }if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { ret = -EPERM ; +atomic_dec (&rb->mmap_count); goto unlock; }
@@ -7142,20 +7148,19 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, flags); -if (!ret) { -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; +if (ret) { +atomic_dec (&rb->mmap_count); + goto unlock; } - } -unlock: -if (!ret) { +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; +aux_success:perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); - }else if (rb) { -/* AUX allocation failed */ -atomic_dec (&rb->mmap_count); } + +unlock: aux_unlock:if (aux_mutex)mutex_unlock (aux_mutex);
Peter Zijlstra
maintainerPATCH v3 7/15 perf: Remove redundant aux_unlock label
In reply to Peter Zijlstraunlock and aux_unlock are now identical, remove the aux_unlock one.
--- kernel/events/core.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
kernel/events/core.c +8 -9
@@ -7095,7 +7095,7 @@ static int perf_mmap(struct file *file, rb = event->rb;if (!rb) - goto aux_unlock; + goto unlock; aux_mutex = &rb->aux_mutex;mutex_lock (aux_mutex);
@@ -7104,27 +7104,27 @@ static int perf_mmap(struct file *file, aux_size =READ_ONCE (rb->user_page->aux_size);if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) - goto aux_unlock; + goto unlock;if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) - goto aux_unlock; + goto unlock;/* already mapped with a different offset */ if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) - goto aux_unlock; + goto unlock;if (aux_size != nr_pages *PAGE_SIZE ) - goto aux_unlock; + goto unlock;/* already mapped with a different size */ if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) - goto aux_unlock; + goto unlock;if (!is_power_of_2 (nr_pages)) - goto aux_unlock; + goto unlock;if (!atomic_inc_not_zero (&rb->mmap_count)) - goto aux_unlock; + goto unlock;if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count);
@@ -7158,7 +7158,6 @@ static int perf_mmap(struct file *file, } unlock: -aux_unlock:if (aux_mutex)mutex_unlock (aux_mutex);mutex_unlock (&event->mmap_mutex);
Re: [PATCH v3 07/15] perf: Remove redundant aux_unlock label
Quoted 3 lines · Peter Zijlstra
unlock and
aux_unlockare now identical, remove theaux_unlockone.Signed-off-by Peter Zijlstra (Intel) <[email protected]>
And in classic fashion, of course you addressed my comment from the last patch in the very next one :P
LGTM, so:
Quoted 60 lines
--- kernel/events/core.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)kernel/events/core.c +8
-9@@ -7095,7 +7095,7 @@ static int perf_mmap(struct file *file, rb = event->rb;if (!rb) - goto aux_unlock; + goto unlock; aux_mutex = &rb->aux_mutex;mutex_lock (aux_mutex);@@ -7104,27 +7104,27 @@ static int perf_mmap(struct file *file, aux_size =READ_ONCE (rb->user_page->aux_size);if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) - goto aux_unlock; + goto unlock;if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) - goto aux_unlock; + goto unlock;/* already mapped with a different offset */ if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) - goto aux_unlock; + goto unlock;if (aux_size != nr_pages *PAGE_SIZE ) - goto aux_unlock; + goto unlock;/* already mapped with a different size */ if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) - goto aux_unlock; + goto unlock;if (!is_power_of_2 (nr_pages)) - goto aux_unlock; + goto unlock;if (!atomic_inc_not_zero (&rb->mmap_count)) - goto aux_unlock; + goto unlock;if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count);@@ -7158,7 +7158,6 @@ static int perf_mmap(struct file *file, } unlock: -aux_unlock:if (aux_mutex)mutex_unlock (aux_mutex);mutex_unlock (&event->mmap_mutex);
[tip: perf/core] perf: Remove redundant aux_unlock label
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 41b80e1d74bdef5e48ea63d186244b9f6f82a4da Gitweb: https://git.kernel.org/tip/41b80e1d74bdef5e48ea63d186244b9f6f82a4da Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:05 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:00 +02:00
perf: Remove redundant aux_unlock label
unlock and aux_unlock are now identical, remove the aux_unlock one.
--- kernel/events/core.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
kernel/events/core.c +8 -9
@@ -7098,7 +7098,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) rb = event->rb;if (!rb) - goto aux_unlock; + goto unlock; aux_mutex = &rb->aux_mutex;mutex_lock (aux_mutex);
@@ -7107,27 +7107,27 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) aux_size =READ_ONCE (rb->user_page->aux_size);if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) - goto aux_unlock; + goto unlock;if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) - goto aux_unlock; + goto unlock;/* already mapped with a different offset */ if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) - goto aux_unlock; + goto unlock;if (aux_size != nr_pages *PAGE_SIZE ) - goto aux_unlock; + goto unlock;/* already mapped with a different size */ if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) - goto aux_unlock; + goto unlock;if (!is_power_of_2 (nr_pages)) - goto aux_unlock; + goto unlock;if (!atomic_inc_not_zero (&rb->mmap_count)) - goto aux_unlock; + goto unlock;if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count);
@@ -7161,7 +7161,6 @@ aux_success: } unlock: -aux_unlock:if (aux_mutex)mutex_unlock (aux_mutex);mutex_unlock (&event->mmap_mutex);
Peter Zijlstra
maintainerPATCH v3 8/15 perf: Use guard() for aux_mutex in perf_mmap()
In reply to Peter ZijlstraAfter duplicating the common code into the rb/aux branches is it possible to use a simple guard() for the aux_mutex. Making the aux branch self-contained.
--- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/events/core.c +1 -5
@@ -6975,7 +6975,6 @@ static int perf_mmap(struct file *file,struct perf_event *event = file->private_data;unsigned long vma_size, nr_pages;long user_extra =0 , extra =0 ; -struct mutex *aux_mutex =NULL ;struct perf_buffer *rb =NULL ;int ret, flags =0 ;mapped_f mapped;
@@ -7098,8 +7097,7 @@ static int perf_mmap(struct file *file,if (!rb) goto unlock; - aux_mutex = &rb->aux_mutex; -mutex_lock (aux_mutex); +guard (mutex)(&rb->aux_mutex); aux_offset =READ_ONCE (rb->user_page->aux_offset); aux_size =READ_ONCE (rb->user_page->aux_size);
@@ -7159,8 +7157,6 @@ static int perf_mmap(struct file *file, } unlock: -if (aux_mutex) -mutex_unlock (aux_mutex);mutex_unlock (&event->mmap_mutex);if (ret)
Re: [PATCH v3 08/15] perf: Use guard() for aux_mutex in perf_mmap()
Quoted 5 lines · Peter Zijlstra
After duplicating the common code into the rb/aux branches is it possible to use a simple
guard()for theaux_mutex. Making the aux branch self-contained.Signed-off-by Peter Zijlstra (Intel) <[email protected]>
LGTM, I really am going to look into how we can use guards in mm to simplify things... :)
So:
Quoted 35 lines
--- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)kernel/events/core.c +1
-5@@ -6975,7 +6975,6 @@ static int perf_mmap(struct file *file,struct perf_event *event = file->private_data;unsigned long vma_size, nr_pages;long user_extra =0 , extra =0 ; -struct mutex *aux_mutex =NULL ;struct perf_buffer *rb =NULL ;int ret, flags =0 ;mapped_f mapped;@@ -7098,8 +7097,7 @@ static int perf_mmap(struct file *file,if (!rb) goto unlock; - aux_mutex = &rb->aux_mutex; -mutex_lock (aux_mutex); +guard (mutex)(&rb->aux_mutex); aux_offset =READ_ONCE (rb->user_page->aux_offset); aux_size =READ_ONCE (rb->user_page->aux_size);@@ -7159,8 +7157,6 @@ static int perf_mmap(struct file *file, } unlock: -if (aux_mutex) -mutex_unlock (aux_mutex);mutex_unlock (&event->mmap_mutex);if (ret)
[tip: perf/core] perf: Use guard() for aux_mutex in perf_mmap()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: b33a51564e3eb6c468979f9f08d9b4ad8451bed7 Gitweb: https://git.kernel.org/tip/b33a51564e3eb6c468979f9f08d9b4ad8451bed7 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:06 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:00 +02:00
perf: Use guard() for aux_mutex in perf_mmap()
After duplicating the common code into the rb/aux branches is it possible to use a simple guard() for the aux_mutex. Making the aux branch self-contained.
--- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/events/core.c +1 -5
@@ -6975,7 +6975,6 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)struct perf_event *event = file->private_data;unsigned long vma_size, nr_pages;long user_extra =0 , extra =0 ; -struct mutex *aux_mutex =NULL ;struct perf_buffer *rb =NULL ;int ret, flags =0 ;mapped_f mapped;
@@ -7100,8 +7099,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (!rb) goto unlock; - aux_mutex = &rb->aux_mutex; -mutex_lock (aux_mutex); +guard (mutex)(&rb->aux_mutex); aux_offset =READ_ONCE (rb->user_page->aux_offset); aux_size =READ_ONCE (rb->user_page->aux_size);
@@ -7161,8 +7159,6 @@ aux_success: } unlock: -if (aux_mutex) -mutex_unlock (aux_mutex);mutex_unlock (&event->mmap_mutex);if (ret)
Peter Zijlstra
maintainerPATCH v3 9/15 perf: Reflow to get rid of aux_success label
In reply to Peter ZijlstraMostly re-indent noise needed to get rid of that label.
--- kernel/events/core.c | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-)
kernel/events/core.c +18 -19
@@ -7127,30 +7127,29 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto aux_success; - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - } + }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; +atomic_dec (&rb->mmap_count); + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags); +if (ret) { +atomic_dec (&rb->mmap_count); + goto unlock; + } -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; -aux_success: +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + }perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }
Re: [PATCH v3 09/15] perf: Reflow to get rid of aux_success label
Quoted 3 lines · Peter Zijlstra
Mostly re-indent noise needed to get rid of that label.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
This is where a side-by-side git diff pager comes in handy :)
LGTM apart from nit/comment below so:
Quoted 8 lines
--- kernel/events/core.c | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-)kernel/events/core.c +0
-0@@ -7127,30 +7127,29 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb)) {
kernel/events/core.c static int perf_mmap(struct file *file, · line 7137
Quoted 10 lines
kernel/events/core.c +0
-7atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto aux_success; - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - }
NIT: These leaves a space above:
if (rb_has_aux(rb)) {
atomic_inc(&rb->aux_mmap_count);
ret = 0;
} else {Quoted 24 lines
kernel/events/core.c +12
-9+ }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; +atomic_dec (&rb->mmap_count); + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags); +if (ret) {
kernel/events/core.c static int perf_mmap(struct file *file, · line 7152
Quoted 10 lines
kernel/events/core.c +6
-3+atomic_dec (&rb->mmap_count); + goto unlock; + } -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; -aux_success: +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + }
This gets rid of the label but leave spretty horrid nesting, but I'm guessing further refactorings will tame it.
Quoted 5 lines
kernel/events/core.c +0
-0perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }
Peter Zijlstra
maintainerRe: [PATCH v3 09/15] perf: Reflow to get rid of aux_success label
Quoted 32 lines · Lorenzo Stoakes
Quoted 3 lines · Peter Zijlstra
Mostly re-indent noise needed to get rid of that label.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>This is where a side-by-side git diff pager comes in handy :)
LGTM apart from nit/comment below so:
Reviewed-by Lorenzo Stoakes <[email protected]>Quoted 8 lines
--- kernel/events/core.c | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-)kernel/events/core.c +0
-0@@ -7127,30 +7127,29 @@ static int perf_mmap(struct file *file,if (rb_has_aux (rb)) {kernel/events/core.c static int perf_mmap(struct file *file, · line 7137
Quoted 10 lines
kernel/events/core.c +0
-7atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto aux_success; - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - }NIT: These leaves a space above:
Quoted 6 lines
if (rb_has_aux(rb)) {
atomic_inc(&rb->aux_mmap_count);
ret = 0;
} else {Yeah, you mention that elsewhere as well. I tend to do that to visually separate the else branch from the previous block. Although I'm not very consistent with it. I can remove these I suppose.
Quoted 28 lines
Quoted 28 lines
kernel/events/core.c +15
-9+ }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; +atomic_dec (&rb->mmap_count); + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags); +if (ret) { +atomic_dec (&rb->mmap_count); + goto unlock; + }
Quoted 9 lines
kernel/events/core.c static int perf_mmap(struct file *file, · line 7152
Quoted 6 lines
kernel/events/core.c +3
-3-atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; -aux_success: +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + }This gets rid of the label but leave spretty horrid nesting, but I'm guessing further refactorings will tame it.
The split out in the next patch removes one indent level.
[tip: perf/core] perf: Reflow to get rid of aux_success label
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 8558dca9fbdf825edf30b5fb74fbbbf3e6ba5dce Gitweb: https://git.kernel.org/tip/8558dca9fbdf825edf30b5fb74fbbbf3e6ba5dce Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:07 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:00 +02:00
perf: Reflow to get rid of aux_success label
Mostly re-indent noise needed to get rid of that label.
--- kernel/events/core.c | 37 ++++++++++++++++++------------------- 1 file changed, 18 insertions(+), 19 deletions(-)
kernel/events/core.c +18 -19
@@ -7130,30 +7130,29 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (rb_has_aux (rb)) {atomic_inc (&rb->aux_mmap_count); ret =0 ; - goto aux_success; - } -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - } + }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { + ret = -EPERM ; +atomic_dec (&rb->mmap_count); + goto unlock; + } -WARN_ON (!rb && event->rb); +WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; +if (vma->vm_flags &VM_WRITE ) + flags |=RING_BUFFER_WRITABLE ; - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, flags); +if (ret) { +atomic_dec (&rb->mmap_count); + goto unlock; + } -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; -aux_success: +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + }perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }
Peter Zijlstra
maintainerPATCH v3 10/15 perf: Split out the AUX buffer allocation
In reply to Peter ZijlstraMove the AUX buffer allocation branch into its own function.
--- kernel/events/core.c | 144 +++++++++++++++++++++++++++------------------------ 1 file changed, 77 insertions(+), 67 deletions(-)
kernel/events/core.c +77 -67
@@ -6970,6 +6970,82 @@ static void perf_mmap_account(struct vm_atomic64_add (extra, &vma->vm_mm->pinned_vm); } +static int perf_mmap_aux (struct vm_area_struct *vma,struct perf_event *event, +unsigned long nr_pages) +{ +long extra =0 , user_extra = nr_pages; +u64 aux_offset, aux_size; +struct perf_buffer *rb; +int ret, rb_flags =0 ; + + rb = event->rb; +if (!rb) +return -EINVAL ; + +guard (mutex)(&rb->aux_mutex); + +/* +* AUX area mapping: if rb->aux_nr_pages != 0, it's already +* mapped, all subsequent mappings should have the same size +* and offset. Must be above the normal perf buffer. +*/ + aux_offset =READ_ONCE (rb->user_page->aux_offset); + aux_size =READ_ONCE (rb->user_page->aux_size); + +if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) +return -EINVAL ; + +if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) +return -EINVAL ; + +/* already mapped with a different offset */ +if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) +return -EINVAL ; + +if (aux_size != nr_pages *PAGE_SIZE ) +return -EINVAL ; + +/* already mapped with a different size */ +if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) +return -EINVAL ; + +if (!is_power_of_2 (nr_pages)) +return -EINVAL ; + +if (!atomic_inc_not_zero (&rb->mmap_count)) +return -EINVAL ; + +if (rb_has_aux (rb)) { +atomic_inc (&rb->aux_mmap_count); + + }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { +atomic_dec (&rb->mmap_count); +return -EPERM ; + } + +WARN_ON (!rb && event->rb); + +if (vma->vm_flags &VM_WRITE ) + rb_flags |=RING_BUFFER_WRITABLE ; + + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, rb_flags); +if (ret) { +atomic_dec (&rb->mmap_count); +return ret; + } + +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + } + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); + +return 0 ; +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data;
@@ -7087,73 +7163,7 @@ static int perf_mmap(struct file *file,perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }else { -/* -* AUX area mapping: if rb->aux_nr_pages != 0, it's already -* mapped, all subsequent mappings should have the same size -* and offset. Must be above the normal perf buffer. -*/ -u64 aux_offset, aux_size; - - rb = event->rb; -if (!rb) - goto unlock; - -guard (mutex)(&rb->aux_mutex); - - aux_offset =READ_ONCE (rb->user_page->aux_offset); - aux_size =READ_ONCE (rb->user_page->aux_size); - -if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) - goto unlock; - -if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) - goto unlock; - -/* already mapped with a different offset */ -if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) - goto unlock; - -if (aux_size != nr_pages *PAGE_SIZE ) - goto unlock; - -/* already mapped with a different size */ -if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) - goto unlock; - -if (!is_power_of_2 (nr_pages)) - goto unlock; - -if (!atomic_inc_not_zero (&rb->mmap_count)) - goto unlock; - -if (rb_has_aux (rb)) { -atomic_inc (&rb->aux_mmap_count); - ret =0 ; - - }else { -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - } - -WARN_ON (!rb && event->rb); - -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; - - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } - -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; - } -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); + ret =perf_mmap_aux (vma, event, nr_pages); } unlock:
Re: [PATCH v3 10/15] perf: Split out the AUX buffer allocation
Quoted 4 lines · Peter Zijlstra
Move the AUX buffer allocation branch into its own function.
Originally-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
LGTM (one nitty note below :), so:
Quoted 48 lines
--- kernel/events/core.c | 144 +++++++++++++++++++++++++++------------------------ 1 file changed, 77 insertions(+), 67 deletions(-)kernel/events/core.c +38
-0@@ -6970,6 +6970,82 @@ static void perf_mmap_account(struct vm_atomic64_add (extra, &vma->vm_mm->pinned_vm); } +static int perf_mmap_aux (struct vm_area_struct *vma,struct perf_event *event, +unsigned long nr_pages) +{ +long extra =0 , user_extra = nr_pages; +u64 aux_offset, aux_size; +struct perf_buffer *rb; +int ret, rb_flags =0 ; + + rb = event->rb; +if (!rb) +return -EINVAL ; + +guard (mutex)(&rb->aux_mutex); + +/* +* AUX area mapping: if rb->aux_nr_pages != 0, it's already +* mapped, all subsequent mappings should have the same size +* and offset. Must be above the normal perf buffer. +*/ + aux_offset =READ_ONCE (rb->user_page->aux_offset); + aux_size =READ_ONCE (rb->user_page->aux_size); + +if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) +return -EINVAL ; + +if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) +return -EINVAL ; + +/* already mapped with a different offset */ +if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) +return -EINVAL ; + +if (aux_size != nr_pages *PAGE_SIZE ) +return -EINVAL ; + +/* already mapped with a different size */ +if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) +return -EINVAL ;
kernel/events/core.c static void perf_mmap_account(struct vm_ · line 7020
Quoted 10 lines
kernel/events/core.c +10
-0+ +if (!is_power_of_2 (nr_pages)) +return -EINVAL ; + +if (!atomic_inc_not_zero (&rb->mmap_count)) +return -EINVAL ; + +if (rb_has_aux (rb)) { +atomic_inc (&rb->aux_mmap_count); +
Still that extra line :>)
Quoted 108 lines
kernel/events/core.c +29
-67+ }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { +atomic_dec (&rb->mmap_count); +return -EPERM ; + } + +WARN_ON (!rb && event->rb); + +if (vma->vm_flags &VM_WRITE ) + rb_flags |=RING_BUFFER_WRITABLE ; + + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, rb_flags); +if (ret) { +atomic_dec (&rb->mmap_count); +return ret; + } + +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + } + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); + +return 0 ; +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data;@@ -7087,73 +7163,7 @@ static int perf_mmap(struct file *file,perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }else { -/* -* AUX area mapping: if rb->aux_nr_pages != 0, it's already -* mapped, all subsequent mappings should have the same size -* and offset. Must be above the normal perf buffer. -*/ -u64 aux_offset, aux_size; - - rb = event->rb; -if (!rb) - goto unlock; - -guard (mutex)(&rb->aux_mutex); - - aux_offset =READ_ONCE (rb->user_page->aux_offset); - aux_size =READ_ONCE (rb->user_page->aux_size); - -if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) - goto unlock; - -if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) - goto unlock; - -/* already mapped with a different offset */ -if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) - goto unlock; - -if (aux_size != nr_pages *PAGE_SIZE ) - goto unlock; - -/* already mapped with a different size */ -if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) - goto unlock; - -if (!is_power_of_2 (nr_pages)) - goto unlock; - -if (!atomic_inc_not_zero (&rb->mmap_count)) - goto unlock; - -if (rb_has_aux (rb)) { -atomic_inc (&rb->aux_mmap_count); - ret =0 ; - - }else { -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - } - -WARN_ON (!rb && event->rb); - -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; - - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } - -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; - } -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); + ret =perf_mmap_aux (vma, event, nr_pages); } unlock:
[tip: perf/core] perf: Split out the AUX buffer allocation
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 2aee37682391332d26c01e703170e0d9358c7252 Gitweb: https://git.kernel.org/tip/2aee37682391332d26c01e703170e0d9358c7252 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:08 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:00 +02:00
perf: Split out the AUX buffer allocation
Move the AUX buffer allocation branch into its own function.
--- kernel/events/core.c | 144 ++++++++++++++++++++++-------------------- 1 file changed, 77 insertions(+), 67 deletions(-)
kernel/events/core.c +77 -67
@@ -6970,6 +6970,82 @@ static void perf_mmap_account(struct vm_area_struct *vma, long user_extra, longatomic64_add (extra, &vma->vm_mm->pinned_vm); } +static int perf_mmap_aux (struct vm_area_struct *vma,struct perf_event *event, +unsigned long nr_pages) +{ +long extra =0 , user_extra = nr_pages; +u64 aux_offset, aux_size; +struct perf_buffer *rb; +int ret, rb_flags =0 ; + + rb = event->rb; +if (!rb) +return -EINVAL ; + +guard (mutex)(&rb->aux_mutex); + +/* +* AUX area mapping: if rb->aux_nr_pages != 0, it's already +* mapped, all subsequent mappings should have the same size +* and offset. Must be above the normal perf buffer. +*/ + aux_offset =READ_ONCE (rb->user_page->aux_offset); + aux_size =READ_ONCE (rb->user_page->aux_size); + +if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) +return -EINVAL ; + +if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) +return -EINVAL ; + +/* already mapped with a different offset */ +if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) +return -EINVAL ; + +if (aux_size != nr_pages *PAGE_SIZE ) +return -EINVAL ; + +/* already mapped with a different size */ +if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) +return -EINVAL ; + +if (!is_power_of_2 (nr_pages)) +return -EINVAL ; + +if (!atomic_inc_not_zero (&rb->mmap_count)) +return -EINVAL ; + +if (rb_has_aux (rb)) { +atomic_inc (&rb->aux_mmap_count); + + }else { +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { +atomic_dec (&rb->mmap_count); +return -EPERM ; + } + +WARN_ON (!rb && event->rb); + +if (vma->vm_flags &VM_WRITE ) + rb_flags |=RING_BUFFER_WRITABLE ; + + ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, + event->attr.aux_watermark, rb_flags); +if (ret) { +atomic_dec (&rb->mmap_count); +return ret; + } + +atomic_set (&rb->aux_mmap_count,1 ); + rb->aux_mmap_locked = extra; + } + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); + +return 0 ; +} +static int perf_mmap (struct file *file,struct vm_area_struct *vma) {struct perf_event *event = file->private_data;
@@ -7088,73 +7164,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); }else { -/* -* AUX area mapping: if rb->aux_nr_pages != 0, it's already -* mapped, all subsequent mappings should have the same size -* and offset. Must be above the normal perf buffer. -*/ -u64 aux_offset, aux_size; - - rb = event->rb; -if (!rb) - goto unlock; - -guard (mutex)(&rb->aux_mutex); - - aux_offset =READ_ONCE (rb->user_page->aux_offset); - aux_size =READ_ONCE (rb->user_page->aux_size); - -if (aux_offset <perf_data_size (rb) +PAGE_SIZE ) - goto unlock; - -if (aux_offset != vma->vm_pgoff <<PAGE_SHIFT ) - goto unlock; - -/* already mapped with a different offset */ -if (rb_has_aux (rb) && rb->aux_pgoff != vma->vm_pgoff) - goto unlock; - -if (aux_size != nr_pages *PAGE_SIZE ) - goto unlock; - -/* already mapped with a different size */ -if (rb_has_aux (rb) && rb->aux_nr_pages != nr_pages) - goto unlock; - -if (!is_power_of_2 (nr_pages)) - goto unlock; - -if (!atomic_inc_not_zero (&rb->mmap_count)) - goto unlock; - -if (rb_has_aux (rb)) { -atomic_inc (&rb->aux_mmap_count); - ret =0 ; - - }else { -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; -atomic_dec (&rb->mmap_count); - goto unlock; - } - -WARN_ON (!rb && event->rb); - -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; - - ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, - event->attr.aux_watermark, flags); -if (ret) { -atomic_dec (&rb->mmap_count); - goto unlock; - } - -atomic_set (&rb->aux_mmap_count,1 ); - rb->aux_mmap_locked = extra; - } -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); + ret =perf_mmap_aux (vma, event, nr_pages); } unlock:
Peter Zijlstra
maintainerPATCH v3 11/15 perf: Make RB allocation branch self sufficient
In reply to Peter ZijlstraEnsure @rb usage doesn't extend out of the branch block.
--- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/events/core.c +1 -5
@@ -7116,8 +7116,6 @@ static int perf_mmap(struct file *file,* multiple times. */ ret =0 ; -/* We need the rb to map pages. */ - rb = event->rb;perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); goto unlock;
@@ -7136,8 +7134,6 @@ static int perf_mmap(struct file *file, goto unlock; } -WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ;
@@ -7190,7 +7186,7 @@ static int perf_mmap(struct file *file,* full cleanup in this case and therefore does not invoke * vmops::close(). */ - ret =map_range (rb, vma); + ret =map_range (event->rb, vma);if (ret)perf_mmap_close (vma);
Re: [PATCH v3 11/15] perf: Make RB allocation branch self sufficient
Quoted 3 lines · Peter Zijlstra
Ensure @rb usage doesn't extend out of the branch block.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Nice to separate this out and make this bit clear.
I'm guessing you're holding off on putting the rb decl into the branch as you intend in a future patch to separate out the rb branch too into a helper like tglx's original did?
Quoted 35 lines
--- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)kernel/events/core.c +1
-5@@ -7116,8 +7116,6 @@ static int perf_mmap(struct file *file,* multiple times. */ ret =0 ; -/* We need the rb to map pages. */ - rb = event->rb;perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); goto unlock;@@ -7136,8 +7134,6 @@ static int perf_mmap(struct file *file, goto unlock; } -WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ;@@ -7190,7 +7186,7 @@ static int perf_mmap(struct file *file,* full cleanup in this case and therefore does not invoke * vmops::close(). */ - ret =map_range (rb, vma); + ret =map_range (event->rb, vma);if (ret)perf_mmap_close (vma);
[tip: perf/core] perf: Make RB allocation branch self sufficient
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 191759e5ea9f6995171ed2ffcc41a2377f946a3a Gitweb: https://git.kernel.org/tip/191759e5ea9f6995171ed2ffcc41a2377f946a3a Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:09 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:01 +02:00
perf: Make RB allocation branch self sufficient
Ensure @rb usage doesn't extend out of the branch block.
--- kernel/events/core.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/events/core.c +1 -5
@@ -7116,8 +7116,6 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)* multiple times. */ ret =0 ; -/* We need the rb to map pages. */ - rb = event->rb;perf_mmap_account (vma, user_extra, extra);atomic_inc (&event->mmap_count); goto unlock;
@@ -7136,8 +7134,6 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) goto unlock; } -WARN_ON (!rb && event->rb); -if (vma->vm_flags &VM_WRITE ) flags |=RING_BUFFER_WRITABLE ;
@@ -7190,7 +7186,7 @@ unlock:* full cleanup in this case and therefore does not invoke * vmops::close(). */ - ret =map_range (rb, vma); + ret =map_range (event->rb, vma);if (ret)perf_mmap_close (vma);
Peter Zijlstra
maintainerPATCH v3 12/15 perf: Split out the RB allocation
In reply to Peter ZijlstraMove the RB buffer allocation branch into its own function.
--- kernel/events/core.c | 145 +++++++++++++++++++++++++-------------------------- 1 file changed, 73 insertions(+), 72 deletions(-)
kernel/events/core.c +73 -72
@@ -6970,6 +6970,75 @@ static void perf_mmap_account(struct vm_atomic64_add (extra, &vma->vm_mm->pinned_vm); } +static int perf_mmap_rb (struct vm_area_struct *vma,struct perf_event *event, +unsigned long nr_pages) +{ +long extra =0 , user_extra = nr_pages; +struct perf_buffer *rb; +int rb_flags =0 ; + + nr_pages -=1 ; + +/* +* If we have rb pages ensure they're a power-of-two number, so we +* can do bitmasks instead of modulo. +*/ +if (nr_pages !=0 && !is_power_of_2 (nr_pages)) +return -EINVAL ; + +WARN_ON_ONCE (event->ctx->parent_ctx); + +if (event->rb) { +if (data_page_nr (event->rb) != nr_pages) +return -EINVAL ; + +if (atomic_inc_not_zero (&event->rb->mmap_count)) { +/* +* Success -- managed to mmap() the same buffer +* multiple times. +*/ +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); +return 0 ; + } + +/* +* Raced against perf_mmap_close()'s +* atomic_dec_and_mutex_lock() remove the +* event and continue as if !event->rb +*/ +ring_buffer_attach (event,NULL ); + } + +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) +return -EPERM ; + +if (vma->vm_flags &VM_WRITE ) + rb_flags |=RING_BUFFER_WRITABLE ; + + rb =rb_alloc (nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark :0 , + event->cpu, rb_flags); + +if (!rb) +return -ENOMEM ; + +atomic_set (&rb->mmap_count,1 ); + rb->mmap_user =get_current_user (); + rb->mmap_locked = extra; + +ring_buffer_attach (event, rb); + +perf_event_update_time (event); +perf_event_init_userpage (event); +perf_event_update_userpage (event); + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); + +return 0 ; +} +static int perf_mmap_aux (struct vm_area_struct *vma,struct perf_event *event,unsigned long nr_pages) {
@@ -7050,10 +7119,8 @@ static int perf_mmap(struct file *file, {struct perf_event *event = file->private_data;unsigned long vma_size, nr_pages; -long user_extra =0 , extra =0 ; -struct perf_buffer *rb =NULL ; -int ret, flags =0 ;mapped_f mapped; +int ret;/* * Don't allow mmap() of inherited per-task counters. This would
@@ -7079,8 +7146,6 @@ static int perf_mmap(struct file *file,if (vma_size !=PAGE_SIZE * nr_pages)return -EINVAL ; - user_extra = nr_pages; -mutex_lock (&event->mmap_mutex); ret = -EINVAL ;
@@ -7094,74 +7159,10 @@ static int perf_mmap(struct file *file, goto unlock; } -if (vma->vm_pgoff ==0 ) { - nr_pages -=1 ; - -/* -* If we have rb pages ensure they're a power-of-two number, so we -* can do bitmasks instead of modulo. -*/ -if (nr_pages !=0 && !is_power_of_2 (nr_pages)) - goto unlock; - -WARN_ON_ONCE (event->ctx->parent_ctx); - -if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) - goto unlock; - -if (atomic_inc_not_zero (&event->rb->mmap_count)) { -/* -* Success -- managed to mmap() the same buffer -* multiple times. -*/ - ret =0 ; -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); - goto unlock; - } - -/* -* Raced against perf_mmap_close()'s -* atomic_dec_and_mutex_lock() remove the -* event and continue as if !event->rb -*/ -ring_buffer_attach (event,NULL ); - } - -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; - goto unlock; - } - -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; - - rb =rb_alloc (nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); - -if (!rb) { - ret = -ENOMEM ; - goto unlock; - } - -atomic_set (&rb->mmap_count,1 ); - rb->mmap_user =get_current_user (); - rb->mmap_locked = extra; - -ring_buffer_attach (event, rb); - -perf_event_update_time (event); -perf_event_init_userpage (event); -perf_event_update_userpage (event); - ret =0 ; - -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); - }else { +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else ret =perf_mmap_aux (vma, event, nr_pages); - } unlock:mutex_unlock (&event->mmap_mutex);
Re: [PATCH v3 12/15] perf: Split out the RB allocation
Quoted 4 lines · Peter Zijlstra
Move the RB buffer allocation branch into its own function.
Originally-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Ah yes of course, the very next patch :)
One nit below about ret assignment, though I wonder if you address later?
Quoted 46 lines
--- kernel/events/core.c | 145 +++++++++++++++++++++++++-------------------------- 1 file changed, 73 insertions(+), 72 deletions(-)kernel/events/core.c +36
-0@@ -6970,6 +6970,75 @@ static void perf_mmap_account(struct vm_atomic64_add (extra, &vma->vm_mm->pinned_vm); } +static int perf_mmap_rb (struct vm_area_struct *vma,struct perf_event *event, +unsigned long nr_pages) +{ +long extra =0 , user_extra = nr_pages; +struct perf_buffer *rb; +int rb_flags =0 ; + + nr_pages -=1 ; + +/* +* If we have rb pages ensure they're a power-of-two number, so we +* can do bitmasks instead of modulo. +*/ +if (nr_pages !=0 && !is_power_of_2 (nr_pages)) +return -EINVAL ; + +WARN_ON_ONCE (event->ctx->parent_ctx); + +if (event->rb) { +if (data_page_nr (event->rb) != nr_pages) +return -EINVAL ; + +if (atomic_inc_not_zero (&event->rb->mmap_count)) { +/* +* Success -- managed to mmap() the same buffer +* multiple times. +*/ +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); +return 0 ; + } + +/* +* Raced against perf_mmap_close()'s +* atomic_dec_and_mutex_lock() remove the +* event and continue as if !event->rb
kernel/events/core.c static void perf_mmap_account(struct vm_ · line 7018
Quoted 10 lines
kernel/events/core.c +10
-0+*/ +ring_buffer_attach (event,NULL ); + } + +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) +return -EPERM ; + +if (vma->vm_flags &VM_WRITE ) + rb_flags |=RING_BUFFER_WRITABLE ; +
I hadn't noticed before actually, but it's really nice that we only assign rb from below for the newly allocated rb, and refer to the prior one via event->rb above.
Which neatly addressed my prior review feedback on this, very nice, cheers!
Quoted 39 lines
kernel/events/core.c +24
-3+ rb =rb_alloc (nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark :0 , + event->cpu, rb_flags); + +if (!rb) +return -ENOMEM ; + +atomic_set (&rb->mmap_count,1 ); + rb->mmap_user =get_current_user (); + rb->mmap_locked = extra; + +ring_buffer_attach (event, rb); + +perf_event_update_time (event); +perf_event_init_userpage (event); +perf_event_update_userpage (event); + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); + +return 0 ; +} +static int perf_mmap_aux (struct vm_area_struct *vma,struct perf_event *event,unsigned long nr_pages) {@@ -7050,10 +7119,8 @@ static int perf_mmap(struct file *file, {struct perf_event *event = file->private_data;unsigned long vma_size, nr_pages; -long user_extra =0 , extra =0 ; -struct perf_buffer *rb =NULL ; -int ret, flags =0 ;mapped_f mapped; +int ret;/* * Don't allow mmap() of inherited per-task counters. This would
kernel/events/core.c static int perf_mmap(struct file *file, · line 7150
Quoted 7 lines
kernel/events/core.c +0
-2@@ -7079,8 +7146,6 @@ static int perf_mmap(struct file *file,if (vma_size !=PAGE_SIZE * nr_pages)return -EINVAL ; - user_extra = nr_pages; -mutex_lock (&event->mmap_mutex); ret = -EINVAL ;
This ret assignment is redundant now, since in all cases you assign ret below.
I wonder if you get rid of this in a later patch though?
Quoted 81 lines
kernel/events/core.c +3
-67@@ -7094,74 +7159,10 @@ static int perf_mmap(struct file *file, goto unlock; } -if (vma->vm_pgoff ==0 ) { - nr_pages -=1 ; - -/* -* If we have rb pages ensure they're a power-of-two number, so we -* can do bitmasks instead of modulo. -*/ -if (nr_pages !=0 && !is_power_of_2 (nr_pages)) - goto unlock; - -WARN_ON_ONCE (event->ctx->parent_ctx); - -if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) - goto unlock; - -if (atomic_inc_not_zero (&event->rb->mmap_count)) { -/* -* Success -- managed to mmap() the same buffer -* multiple times. -*/ - ret =0 ; -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); - goto unlock; - } - -/* -* Raced against perf_mmap_close()'s -* atomic_dec_and_mutex_lock() remove the -* event and continue as if !event->rb -*/ -ring_buffer_attach (event,NULL ); - } - -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; - goto unlock; - } - -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; - - rb =rb_alloc (nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); - -if (!rb) { - ret = -ENOMEM ; - goto unlock; - } - -atomic_set (&rb->mmap_count,1 ); - rb->mmap_user =get_current_user (); - rb->mmap_locked = extra; - -ring_buffer_attach (event, rb); - -perf_event_update_time (event); -perf_event_init_userpage (event); -perf_event_update_userpage (event); - ret =0 ; - -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); - }else { +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else ret =perf_mmap_aux (vma, event, nr_pages); - } unlock:mutex_unlock (&event->mmap_mutex);
[tip: perf/core] perf: Split out the RB allocation
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 5d299897f1e36025400ca84fd36c15925a383b03 Gitweb: https://git.kernel.org/tip/5d299897f1e36025400ca84fd36c15925a383b03 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:10 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:01 +02:00
perf: Split out the RB allocation
Move the RB buffer allocation branch into its own function.
--- kernel/events/core.c | 145 +++++++++++++++++++++--------------------- 1 file changed, 73 insertions(+), 72 deletions(-)
kernel/events/core.c +73 -72
@@ -6970,6 +6970,75 @@ static void perf_mmap_account(struct vm_area_struct *vma, long user_extra, longatomic64_add (extra, &vma->vm_mm->pinned_vm); } +static int perf_mmap_rb (struct vm_area_struct *vma,struct perf_event *event, +unsigned long nr_pages) +{ +long extra =0 , user_extra = nr_pages; +struct perf_buffer *rb; +int rb_flags =0 ; + + nr_pages -=1 ; + +/* +* If we have rb pages ensure they're a power-of-two number, so we +* can do bitmasks instead of modulo. +*/ +if (nr_pages !=0 && !is_power_of_2 (nr_pages)) +return -EINVAL ; + +WARN_ON_ONCE (event->ctx->parent_ctx); + +if (event->rb) { +if (data_page_nr (event->rb) != nr_pages) +return -EINVAL ; + +if (atomic_inc_not_zero (&event->rb->mmap_count)) { +/* +* Success -- managed to mmap() the same buffer +* multiple times. +*/ +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); +return 0 ; + } + +/* +* Raced against perf_mmap_close()'s +* atomic_dec_and_mutex_lock() remove the +* event and continue as if !event->rb +*/ +ring_buffer_attach (event,NULL ); + } + +if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) +return -EPERM ; + +if (vma->vm_flags &VM_WRITE ) + rb_flags |=RING_BUFFER_WRITABLE ; + + rb =rb_alloc (nr_pages, + event->attr.watermark ? event->attr.wakeup_watermark :0 , + event->cpu, rb_flags); + +if (!rb) +return -ENOMEM ; + +atomic_set (&rb->mmap_count,1 ); + rb->mmap_user =get_current_user (); + rb->mmap_locked = extra; + +ring_buffer_attach (event, rb); + +perf_event_update_time (event); +perf_event_init_userpage (event); +perf_event_update_userpage (event); + +perf_mmap_account (vma, user_extra, extra); +atomic_inc (&event->mmap_count); + +return 0 ; +} +static int perf_mmap_aux (struct vm_area_struct *vma,struct perf_event *event,unsigned long nr_pages) {
@@ -7050,10 +7119,8 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) {struct perf_event *event = file->private_data;unsigned long vma_size, nr_pages; -long user_extra =0 , extra =0 ; -struct perf_buffer *rb =NULL ; -int ret, flags =0 ;mapped_f mapped; +int ret;/* * Don't allow mmap() of inherited per-task counters. This would
@@ -7079,8 +7146,6 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (vma_size !=PAGE_SIZE * nr_pages)return -EINVAL ; - user_extra = nr_pages; -mutex_lock (&event->mmap_mutex); ret = -EINVAL ;
@@ -7094,74 +7159,10 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) goto unlock; } -if (vma->vm_pgoff ==0 ) { - nr_pages -=1 ; - -/* -* If we have rb pages ensure they're a power-of-two number, so we -* can do bitmasks instead of modulo. -*/ -if (nr_pages !=0 && !is_power_of_2 (nr_pages)) - goto unlock; - -WARN_ON_ONCE (event->ctx->parent_ctx); - -if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) - goto unlock; - -if (atomic_inc_not_zero (&event->rb->mmap_count)) { -/* -* Success -- managed to mmap() the same buffer -* multiple times. -*/ - ret =0 ; -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); - goto unlock; - } - -/* -* Raced against perf_mmap_close()'s -* atomic_dec_and_mutex_lock() remove the -* event and continue as if !event->rb -*/ -ring_buffer_attach (event,NULL ); - } - -if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { - ret = -EPERM ; - goto unlock; - } - -if (vma->vm_flags &VM_WRITE ) - flags |=RING_BUFFER_WRITABLE ; - - rb =rb_alloc (nr_pages, - event->attr.watermark ? event->attr.wakeup_watermark :0 , - event->cpu, flags); - -if (!rb) { - ret = -ENOMEM ; - goto unlock; - } - -atomic_set (&rb->mmap_count,1 ); - rb->mmap_user =get_current_user (); - rb->mmap_locked = extra; - -ring_buffer_attach (event, rb); - -perf_event_update_time (event); -perf_event_init_userpage (event); -perf_event_update_userpage (event); - ret =0 ; - -perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); - }else { +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else ret =perf_mmap_aux (vma, event, nr_pages); - } unlock:mutex_unlock (&event->mmap_mutex);
Peter Zijlstra
maintainerPATCH v3 13/15 perf: Use scoped_guard() for mmap_mutex in perf_mmap()
In reply to Peter ZijlstraMostly just re-indent noise.
--- kernel/events/core.c | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-)
kernel/events/core.c +14 -18
@@ -7143,27 +7143,23 @@ static int perf_mmap(struct file *file,if (vma_size !=PAGE_SIZE * nr_pages)return -EINVAL ; -mutex_lock (&event->mmap_mutex); - ret = -EINVAL ; + scoped_guard (mutex, &event->mmap_mutex) { +/* +* This relies on __pmu_detach_event() taking mmap_mutex after marking +* the event REVOKED. Either we observe the state, or __pmu_detach_event() +* will detach the rb created here. +*/ +if (event->state <=PERF_EVENT_STATE_REVOKED ) { + ret = -ENODEV ; +break ; + } -/* -* This relies on __pmu_detach_event() taking mmap_mutex after marking -* the event REVOKED. Either we observe the state, or __pmu_detach_event() -* will detach the rb created here. -*/ -if (event->state <=PERF_EVENT_STATE_REVOKED ) { - ret = -ENODEV ; - goto unlock; +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else + ret =perf_mmap_aux (vma, event, nr_pages); } -if (vma->vm_pgoff ==0 ) - ret =perf_mmap_rb (vma, event, nr_pages); -else - ret =perf_mmap_aux (vma, event, nr_pages); - -unlock: -mutex_unlock (&event->mmap_mutex); -if (ret)return ret;
Re: [PATCH v3 13/15] perf: Use scoped_guard() for mmap_mutex in perf_mmap()
Quoted 4 lines · Peter Zijlstra
Mostly just re-indent noise.
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
LGTM, so:
kernel/events/core.c static int perf_mmap(struct file *file, · line 7146
Quoted 11 lines
--- kernel/events/core.c | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-)kernel/events/core.c +0
-1@@ -7143,27 +7143,23 @@ static int perf_mmap(struct file *file,if (vma_size !=PAGE_SIZE * nr_pages)return -EINVAL ; -mutex_lock (&event->mmap_mutex);
Yeah getting rid of this lock goto stuff is great... mm is at risk of some more Lorenzo churn to use this stuff over there... :)
kernel/events/core.c static int perf_mmap(struct file *file, · line 7147
Quoted 1 line
kernel/events/core.c +0
-1- ret = -EINVAL ;
You see, I'm getting in a pattern here with comment on prior patch and you reading my mind and fixing it on next ;)
kernel/events/core.c static int perf_mmap(struct file *file, · line 7154
Quoted 9 lines
kernel/events/core.c +9
-0+ scoped_guard (mutex, &event->mmap_mutex) { +/* +* This relies on __pmu_detach_event() taking mmap_mutex after marking +* the event REVOKED. Either we observe the state, or __pmu_detach_event() +* will detach the rb created here. +*/ +if (event->state <=PERF_EVENT_STATE_REVOKED ) { + ret = -ENODEV ; +break ;
I don't absolutely love this break-for-what-is-not-obviously-a-for-loop formulation (I know scoped_guard in practice _is_ a for loop, but obviously that's hidden by macro), but I guess hey it's C, and we have to do what we have to do :)
Quoted 29 lines
kernel/events/core.c +5
-16+ } -/* -* This relies on __pmu_detach_event() taking mmap_mutex after marking -* the event REVOKED. Either we observe the state, or __pmu_detach_event() -* will detach the rb created here. -*/ -if (event->state <=PERF_EVENT_STATE_REVOKED ) { - ret = -ENODEV ; - goto unlock; +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else + ret =perf_mmap_aux (vma, event, nr_pages); } -if (vma->vm_pgoff ==0 ) - ret =perf_mmap_rb (vma, event, nr_pages); -else - ret =perf_mmap_aux (vma, event, nr_pages); - -unlock: -mutex_unlock (&event->mmap_mutex); -if (ret)return ret;
Peter Zijlstra
maintainerRe: [PATCH v3 13/15] perf: Use scoped_guard() for mmap_mutex in perf_mmap()
Quoted 6 lines · Lorenzo Stoakes
Quoted 6 lines
kernel/events/core.c +6
-0+ scoped_guard (mutex, &event->mmap_mutex) { +/* +* This relies on __pmu_detach_event() taking mmap_mutex after marking +* the event REVOKED. Either we observe the state, or __pmu_detach_event() +* will detach the rb created here. +*/
Quoted 8 lines
kernel/events/core.c static int perf_mmap(struct file *file, · line 7154
Quoted 3 lines
kernel/events/core.c +3
-0+if (event->state <=PERF_EVENT_STATE_REVOKED ) { + ret = -ENODEV ; +break ;I don't absolutely love this break-for-what-is-not-obviously-a-for-loop formulation (I know
scoped_guardin practice _is_ a for loop, but obviously that's hidden by macro), but I guess hey it's C, and we have to do what we have to do :)
Right, don't love it either, but the alternative was a goto and that's arguably worse, so meh.
Quoted 13 lines
Quoted 13 lines
kernel/events/core.c +5
-0+ } +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else + ret =perf_mmap_aux (vma, event, nr_pages); }if (ret)return ret;
Peter Zijlstra
maintainerRe: [PATCH v3 13/15] perf: Use scoped_guard() for mmap_mutex in perf_mmap()
Quoted 28 lines · Peter Zijlstra
Quoted 6 lines · Lorenzo Stoakes
Quoted 6 lines
(quoted diff) +6
-0+ scoped_guard (mutex, &event->mmap_mutex) { + /* + * This relies on __pmu_detach_event() taking mmap_mutex after marking + * the event REVOKED. Either we observe the state, or __pmu_detach_event() + * will detach the rb created here. + */Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ if (event->state <= PERF_EVENT_STATE_REVOKED) { + ret = -ENODEV; + break;I don't absolutely love this break-for-what-is-not-obviously-a-for-loop formulation (I know
scoped_guardin practice _is_ a for loop, but obviously that's hidden by macro), but I guess hey it's C, and we have to do what we have to do :)Right, don't love it either, but the alternative was a goto and that's arguably worse, so meh.
Quoted 8 lines
Quoted 8 lines
(quoted diff) +5
-0+ } + if (vma->vm_pgoff == 0) + ret = perf_mmap_rb(vma, event, nr_pages); + else + ret = perf_mmap_aux(vma, event, nr_pages); }
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
(quoted diff) +0
-0if (ret) return ret;
Nah, I'm an idiot.. How's this?
---
kernel/events/core.c +4 -7
@@ -7151,20 +7151,17 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)* the event REVOKED. Either we observe the state, or __pmu_detach_event() * will detach the rb created here. */ -if (event->state <=PERF_EVENT_STATE_REVOKED ) { - ret = -ENODEV ; -break ; - } +if (event->state <=PERF_EVENT_STATE_REVOKED ) +return -ENODEV ;if (vma->vm_pgoff ==0 ) ret =perf_mmap_rb (vma, event, nr_pages);else ret =perf_mmap_aux (vma, event, nr_pages); +if (ret) +return ret; } -if (ret) -return ret; -/* * Since pinned accounting is per vm we cannot allow fork() to copy our * vma.
Re: [PATCH v3 13/15] perf: Use scoped_guard() for mmap_mutex in perf_mmap()
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Peter Zijlstra
Quoted 6 lines · Lorenzo Stoakes
Quoted 6 lines
(quoted diff) +6
-0+ scoped_guard (mutex, &event->mmap_mutex) { + /* + * This relies on __pmu_detach_event() taking mmap_mutex after marking + * the event REVOKED. Either we observe the state, or __pmu_detach_event() + * will detach the rb created here. + */Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ if (event->state <= PERF_EVENT_STATE_REVOKED) { + ret = -ENODEV; + break;I don't absolutely love this break-for-what-is-not-obviously-a-for-loop formulation (I know
scoped_guardin practice _is_ a for loop, but obviously that's hidden by macro), but I guess hey it's C, and we have to do what we have to do :)Right, don't love it either, but the alternative was a goto and that's arguably worse, so meh.
Quoted 8 lines
Quoted 8 lines
(quoted diff) +5
-0+ } + if (vma->vm_pgoff == 0) + ret = perf_mmap_rb(vma, event, nr_pages); + else + ret = perf_mmap_aux(vma, event, nr_pages); }
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
(quoted diff) +0
-0if (ret) return ret;Nah, I'm an idiot.. How's this?
Right yeah, you could just return instead of break :))
[tip: perf/core] perf: Use scoped_guard() for mmap_mutex in perf_mmap()
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: d23a6dbc0a71741eb7b141fdc04e31360fba46ef Gitweb: https://git.kernel.org/tip/d23a6dbc0a71741eb7b141fdc04e31360fba46ef Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:11 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:01 +02:00
perf: Use scoped_guard() for mmap_mutex in perf_mmap()
Mostly just re-indent noise.
--- kernel/events/core.c | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-)
kernel/events/core.c +14 -21
@@ -7146,30 +7146,23 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)if (vma_size !=PAGE_SIZE * nr_pages)return -EINVAL ; -mutex_lock (&event->mmap_mutex); - ret = -EINVAL ; + scoped_guard (mutex, &event->mmap_mutex) { +/* +* This relies on __pmu_detach_event() taking mmap_mutex after marking +* the event REVOKED. Either we observe the state, or __pmu_detach_event() +* will detach the rb created here. +*/ +if (event->state <=PERF_EVENT_STATE_REVOKED ) +return -ENODEV ; -/* -* This relies on __pmu_detach_event() taking mmap_mutex after marking -* the event REVOKED. Either we observe the state, or __pmu_detach_event() -* will detach the rb created here. -*/ -if (event->state <=PERF_EVENT_STATE_REVOKED ) { - ret = -ENODEV ; - goto unlock; +if (vma->vm_pgoff ==0 ) + ret =perf_mmap_rb (vma, event, nr_pages); +else + ret =perf_mmap_aux (vma, event, nr_pages); +if (ret) +return ret; } -if (vma->vm_pgoff ==0 ) - ret =perf_mmap_rb (vma, event, nr_pages); -else - ret =perf_mmap_aux (vma, event, nr_pages); - -unlock: -mutex_unlock (&event->mmap_mutex); - -if (ret) -return ret; -/* * Since pinned accounting is per vm we cannot allow fork() to copy our * vma.
Peter Zijlstra
maintainerPATCH v3 14/15 perf: Identify the 0->1 transition for event::mmap_count
In reply to Peter ZijlstraNeeded because refcount_inc() doesn't allow the 0->1 transition.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -7033,7 +7033,7 @@ static int perf_mmap_rb(struct vm_area_sperf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +atomic_set (&event->mmap_count,1 );return 0 ; }
Peter Zijlstra
maintainerRe: [PATCH v3 14/15] perf: Identify the 0->1 transition for event::mmap_count
ljs noted that this Changelog was a tad terse, and he had a pending question here.
Quoted 1 line · Peter Zijlstra
Needed because
refcount_inc()doesn't allow the 0->1 transition.
Specifically, this is the case where we've created the RB, this means there was no RB, and as such there could not have been an mmap. Additionally we hold mmap_mutex to serialize everything.
This must be the first.
Quoted 19 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/events/core.c +1
-1@@ -7033,7 +7033,7 @@ static int perf_mmap_rb(struct vm_area_sperf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +atomic_set (&event->mmap_count,1 );return 0 ; }
Lorenzo Stoakes
Reviewed-byRe: [PATCH v3 14/15] perf: Identify the 0->1 transition for event::mmap_count
Quoted 4 lines · Peter Zijlstra
ljs noted that this Changelog was a tad terse, and he had a pending question here.
Quoted 8 lines
Quoted 1 line · Peter Zijlstra
Needed because
refcount_inc()doesn't allow the 0->1 transition.Specifically, this is the case where we've created the RB, this means there was no RB, and as such there could not have been an mmap. Additionally we hold
mmap_mutexto serialize everything.This must be the first.
Thanks, this is great!
And really good to separate this out as a really critical step prepping for refcount_t change.
If we're being nitty it might be worth mentioning that we plan to move to refcount_t and that considers inc from 0 to be evidence of a UAF, hence the absolute requirement for this.
However, since we're clearly stating that this is the first instance of this mmap_count being incremented, it's moot, as it's simply clearer to do this anyway.
Quoted 3 lines
Quoted 3 lines
Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
LGTM so:
Quoted 17 lines
Quoted 17 lines
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/events/core.c +1
-1@@ -7033,7 +7033,7 @@ static int perf_mmap_rb(struct vm_area_sperf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +atomic_set (&event->mmap_count,1 );return 0 ; }
[tip: perf/core] perf: Identify the 0->1 transition for event::mmap_count
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 59741451b49ce9964a9758c19d6f7df2a1255c75 Gitweb: https://git.kernel.org/tip/59741451b49ce9964a9758c19d6f7df2a1255c75 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:12 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:02 +02:00
perf: Identify the 0->1 transition for event::mmap_count
Needed because refcount_inc() doesn't allow the 0->1 transition.
Specifically, this is the case where we've created the RB, this means there was no RB, and as such there could not have been an mmap. Additionally we hold mmap_mutex to serialize everything.
This must be the first.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -7034,7 +7034,7 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event,perf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +atomic_set (&event->mmap_count,1 );return 0 ; }
Peter Zijlstra
maintainerPATCH v3 15/15 perf: Convert mmap() refcounts to refcount_t
In reply to Peter ZijlstraFrom: Thomas Gleixner <[email protected]>
The recently fixed reference count leaks could have been detected by using refcount_t and refcount_t would have mitigated the potential overflow at least.
Now that the code is properly structured, convert the mmap() related mmap_count variants over to refcount_t.
No functional change intended.
--- include/linux/perf_event.h | 2 +- kernel/events/core.c | 40 ++++++++++++++++++++-------------------- kernel/events/internal.h | 4 ++-- kernel/events/ring_buffer.c | 2 +- 4 files changed, 24 insertions(+), 24 deletions(-)
include/linux/perf_event.h +1 -1
@@ -859,7 +859,7 @@ struct perf_event {/* mmap bits */ struct mutex mmap_mutex; -atomic_t mmap_count; +refcount_t mmap_count;struct perf_buffer *rb;struct list_head rb_entry;
kernel/events/core.c +20 -20
@@ -3968,7 +3968,7 @@ static noinline int visit_groups_merge(s*/ static inline bool event_update_userpage (struct perf_event *event) { -if (likely (!atomic_read (&event->mmap_count))) +if (likely (!refcount_read (&event->mmap_count)))return false;perf_event_update_time (event);
@@ -6704,11 +6704,11 @@ static void perf_mmap_open(struct vm_arestruct perf_event *event = vma->vm_file->private_data;mapped_f mapped =get_mapped (event, event_mapped); -atomic_inc (&event->mmap_count); -atomic_inc (&event->rb->mmap_count); +refcount_inc (&event->mmap_count); +refcount_inc (&event->rb->mmap_count);if (vma->vm_pgoff) -atomic_inc (&event->rb->aux_mmap_count); +refcount_inc (&event->rb->aux_mmap_count);if (mapped)mapped (event, vma->vm_mm);
@@ -6743,7 +6743,7 @@ static void perf_mmap_close(struct vm_ar* to avoid complications. */ if (rb_has_aux (rb) && vma->vm_pgoff == rb->aux_pgoff && -atomic_dec_and_mutex_lock (&rb->aux_mmap_count, &rb->aux_mutex)) { +refcount_dec_and_mutex_lock (&rb->aux_mmap_count, &rb->aux_mutex)) {/* * Stop all AUX events that are writing to this buffer, * so that we can free its AUX pages and corresponding PMU
@@ -6763,10 +6763,10 @@ static void perf_mmap_close(struct vm_armutex_unlock (&rb->aux_mutex); } -if (atomic_dec_and_test (&rb->mmap_count)) +if (refcount_dec_and_test (&rb->mmap_count)) detach_rest = true; -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) +if (!refcount_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;ring_buffer_attach (event,NULL );
@@ -6992,19 +6992,19 @@ static int perf_mmap_rb(struct vm_area_sif (data_page_nr (event->rb) != nr_pages)return -EINVAL ; -if (atomic_inc_not_zero (&event->rb->mmap_count)) { +if (refcount_inc_not_zero (&event->rb->mmap_count)) {/* * Success -- managed to mmap() the same buffer * multiple times. */ perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +refcount_inc (&event->mmap_count);return 0 ; }/* * Raced against perf_mmap_close()'s -* atomic_dec_and_mutex_lock() remove the +* refcount_dec_and_mutex_lock() remove the * event and continue as if !event->rb */ ring_buffer_attach (event,NULL );
@@ -7023,7 +7023,7 @@ static int perf_mmap_rb(struct vm_area_sif (!rb)return -ENOMEM ; -atomic_set (&rb->mmap_count,1 ); +refcount_set (&rb->mmap_count,1 ); rb->mmap_user =get_current_user (); rb->mmap_locked = extra;
@@ -7034,7 +7034,7 @@ static int perf_mmap_rb(struct vm_area_sperf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_set (&event->mmap_count,1 ); +refcount_set (&event->mmap_count,1 );return 0 ; }
@@ -7081,15 +7081,15 @@ static int perf_mmap_aux(struct vm_area_if (!is_power_of_2 (nr_pages))return -EINVAL ; -if (!atomic_inc_not_zero (&rb->mmap_count)) +if (!refcount_inc_not_zero (&rb->mmap_count))return -EINVAL ;if (rb_has_aux (rb)) { -atomic_inc (&rb->aux_mmap_count); +refcount_inc (&rb->aux_mmap_count); }else {if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { -atomic_dec (&rb->mmap_count); +refcount_dec (&rb->mmap_count);return -EPERM ; }
@@ -7101,16 +7101,16 @@ static int perf_mmap_aux(struct vm_area_ ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, rb_flags);if (ret) { -atomic_dec (&rb->mmap_count); +refcount_dec (&rb->mmap_count);return ret; } -atomic_set (&rb->aux_mmap_count,1 ); +refcount_set (&rb->aux_mmap_count,1 ); rb->aux_mmap_locked = extra; }perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +refcount_inc (&event->mmap_count);return 0 ; }
@@ -13257,7 +13257,7 @@ perf_event_set_output(struct perf_eventmutex_lock_double (&event->mmap_mutex, &output_event->mmap_mutex); set:/* Can't redirect output if we've got an active mmap() */ -if (atomic_read (&event->mmap_count)) +if (refcount_read (&event->mmap_count)) goto unlock;if (output_event) {
@@ -13270,7 +13270,7 @@ perf_event_set_output(struct perf_event goto unlock;/* did we race against perf_mmap_close() */ -if (!atomic_read (&rb->mmap_count)) { +if (!refcount_read (&rb->mmap_count)) {ring_buffer_put (rb); goto unlock; }
kernel/events/internal.h +2 -2
@@ -35,7 +35,7 @@ struct perf_buffer {spinlock_t event_lock;struct list_head event_list; -atomic_t mmap_count; +refcount_t mmap_count;unsigned long mmap_locked;struct user_struct *mmap_user;
@@ -47,7 +47,7 @@ struct perf_buffer {unsigned long aux_pgoff;int aux_nr_pages;int aux_overwrite; -atomic_t aux_mmap_count; +refcount_t aux_mmap_count;unsigned long aux_mmap_locked;void (*free_aux)(void *);refcount_t aux_refcount;
kernel/events/ring_buffer.c +1 -1
@@ -400,7 +400,7 @@ void *perf_aux_output_begin(struct perf_* the same order, see perf_mmap_close. Otherwise we end up freeing * aux pages in this path, which is a bug, because in_atomic(). */ -if (!atomic_read (&rb->aux_mmap_count)) +if (!refcount_read (&rb->aux_mmap_count)) goto err;if (!refcount_inc_not_zero (&rb->aux_refcount))
Re: [PATCH v3 15/15] perf: Convert mmap() refcounts to refcount_t
Quoted 6 lines · Peter Zijlstra
From: Thomas Gleixner <[email protected]>
The recently fixed reference count leaks could have been detected by using
refcount_tandrefcount_twould have mitigated the potential overflow at least.
Quoted 7 lines
Now that the code is properly structured, convert the
mmap()relatedmmap_countvariants over torefcount_t.No functional change intended.
Signed-off-by Thomas Gleixner <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
All LGTM, checked the mmap test passes and no splat, the atomic_inc() -> atomic_set() on &rb->mmap_count (which here becomes refcount_set()) sorted that out.
This is a nice fix up as is the rest of the series, kudos to tglx and additionally you Peter for adding further nice changes + fixing up the bit we missed.
This is a _massive_ improvement :)
Quoted 198 lines
--- include/linux/perf_event.h | 2 +- kernel/events/core.c | 40 ++++++++++++++++++++-------------------- kernel/events/internal.h | 4 ++-- kernel/events/ring_buffer.c | 2 +- 4 files changed, 24 insertions(+), 24 deletions(-)include/linux/perf_event.h +1
-1@@ -859,7 +859,7 @@ struct perf_event {/* mmap bits */ struct mutex mmap_mutex; -atomic_t mmap_count; +refcount_t mmap_count;struct perf_buffer *rb;struct list_head rb_entry;kernel/events/core.c +20
-20@@ -3968,7 +3968,7 @@ static noinline int visit_groups_merge(s*/ static inline bool event_update_userpage (struct perf_event *event) { -if (likely (!atomic_read (&event->mmap_count))) +if (likely (!refcount_read (&event->mmap_count)))return false;perf_event_update_time (event);@@ -6704,11 +6704,11 @@ static void perf_mmap_open(struct vm_arestruct perf_event *event = vma->vm_file->private_data;mapped_f mapped =get_mapped (event, event_mapped); -atomic_inc (&event->mmap_count); -atomic_inc (&event->rb->mmap_count); +refcount_inc (&event->mmap_count); +refcount_inc (&event->rb->mmap_count);if (vma->vm_pgoff) -atomic_inc (&event->rb->aux_mmap_count); +refcount_inc (&event->rb->aux_mmap_count);if (mapped)mapped (event, vma->vm_mm);@@ -6743,7 +6743,7 @@ static void perf_mmap_close(struct vm_ar* to avoid complications. */ if (rb_has_aux (rb) && vma->vm_pgoff == rb->aux_pgoff && -atomic_dec_and_mutex_lock (&rb->aux_mmap_count, &rb->aux_mutex)) { +refcount_dec_and_mutex_lock (&rb->aux_mmap_count, &rb->aux_mutex)) {/* * Stop all AUX events that are writing to this buffer, * so that we can free its AUX pages and corresponding PMU @@ -6763,10 +6763,10 @@ static void perf_mmap_close(struct vm_armutex_unlock (&rb->aux_mutex); } -if (atomic_dec_and_test (&rb->mmap_count)) +if (refcount_dec_and_test (&rb->mmap_count)) detach_rest = true; -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) +if (!refcount_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;ring_buffer_attach (event,NULL );@@ -6992,19 +6992,19 @@ static int perf_mmap_rb(struct vm_area_sif (data_page_nr (event->rb) != nr_pages)return -EINVAL ; -if (atomic_inc_not_zero (&event->rb->mmap_count)) { +if (refcount_inc_not_zero (&event->rb->mmap_count)) {/* * Success -- managed to mmap() the same buffer * multiple times. */ perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +refcount_inc (&event->mmap_count);return 0 ; }/* * Raced against perf_mmap_close()'s -* atomic_dec_and_mutex_lock() remove the +* refcount_dec_and_mutex_lock() remove the * event and continue as if !event->rb */ ring_buffer_attach (event,NULL );@@ -7023,7 +7023,7 @@ static int perf_mmap_rb(struct vm_area_sif (!rb)return -ENOMEM ; -atomic_set (&rb->mmap_count,1 ); +refcount_set (&rb->mmap_count,1 ); rb->mmap_user =get_current_user (); rb->mmap_locked = extra;@@ -7034,7 +7034,7 @@ static int perf_mmap_rb(struct vm_area_sperf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_set (&event->mmap_count,1 ); +refcount_set (&event->mmap_count,1 );return 0 ; }@@ -7081,15 +7081,15 @@ static int perf_mmap_aux(struct vm_area_if (!is_power_of_2 (nr_pages))return -EINVAL ; -if (!atomic_inc_not_zero (&rb->mmap_count)) +if (!refcount_inc_not_zero (&rb->mmap_count))return -EINVAL ;if (rb_has_aux (rb)) { -atomic_inc (&rb->aux_mmap_count); +refcount_inc (&rb->aux_mmap_count); }else {if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { -atomic_dec (&rb->mmap_count); +refcount_dec (&rb->mmap_count);return -EPERM ; }@@ -7101,16 +7101,16 @@ static int perf_mmap_aux(struct vm_area_ ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, rb_flags);if (ret) { -atomic_dec (&rb->mmap_count); +refcount_dec (&rb->mmap_count);return ret; } -atomic_set (&rb->aux_mmap_count,1 ); +refcount_set (&rb->aux_mmap_count,1 ); rb->aux_mmap_locked = extra; }perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +refcount_inc (&event->mmap_count);return 0 ; }@@ -13257,7 +13257,7 @@ perf_event_set_output(struct perf_eventmutex_lock_double (&event->mmap_mutex, &output_event->mmap_mutex); set:/* Can't redirect output if we've got an active mmap() */ -if (atomic_read (&event->mmap_count)) +if (refcount_read (&event->mmap_count)) goto unlock;if (output_event) {@@ -13270,7 +13270,7 @@ perf_event_set_output(struct perf_event goto unlock;/* did we race against perf_mmap_close() */ -if (!atomic_read (&rb->mmap_count)) { +if (!refcount_read (&rb->mmap_count)) {ring_buffer_put (rb); goto unlock; }kernel/events/internal.h +2
-2@@ -35,7 +35,7 @@ struct perf_buffer {spinlock_t event_lock;struct list_head event_list; -atomic_t mmap_count; +refcount_t mmap_count;unsigned long mmap_locked;struct user_struct *mmap_user;@@ -47,7 +47,7 @@ struct perf_buffer {unsigned long aux_pgoff;int aux_nr_pages;int aux_overwrite; -atomic_t aux_mmap_count; +refcount_t aux_mmap_count;unsigned long aux_mmap_locked;void (*free_aux)(void *);refcount_t aux_refcount;kernel/events/ring_buffer.c +1
-1@@ -400,7 +400,7 @@ void *perf_aux_output_begin(struct perf_* the same order, see perf_mmap_close. Otherwise we end up freeing * aux pages in this path, which is a bug, because in_atomic(). */ -if (!atomic_read (&rb->aux_mmap_count)) +if (!refcount_read (&rb->aux_mmap_count)) goto err;if (!refcount_inc_not_zero (&rb->aux_refcount))
[tip: perf/core] perf: Convert mmap() refcounts to refcount_t
In reply to Peter ZijlstraThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 448f97fba9013ffa13f5dd82febd18836b189499 Gitweb: https://git.kernel.org/tip/448f97fba9013ffa13f5dd82febd18836b189499 Author: Thomas Gleixner <[email protected]> AuthorDate: Tue, 12 Aug 2025 12:39:13 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 15 Aug 2025 13:13:02 +02:00
perf: Convert mmap() refcounts to refcount_t
The recently fixed reference count leaks could have been detected by using refcount_t and refcount_t would have mitigated the potential overflow at least.
Now that the code is properly structured, convert the mmap() related mmap_count variants over to refcount_t.
No functional change intended.
--- include/linux/perf_event.h | 2 +- kernel/events/core.c | 40 ++++++++++++++++++------------------ kernel/events/internal.h | 4 ++-- kernel/events/ring_buffer.c | 2 +- 4 files changed, 24 insertions(+), 24 deletions(-)
include/linux/perf_event.h +1 -1
@@ -859,7 +859,7 @@ struct perf_event {/* mmap bits */ struct mutex mmap_mutex; -atomic_t mmap_count; +refcount_t mmap_count;struct perf_buffer *rb;struct list_head rb_entry;
kernel/events/core.c +20 -20
@@ -3968,7 +3968,7 @@ static noinline int visit_groups_merge(struct perf_event_context *ctx,*/ static inline bool event_update_userpage (struct perf_event *event) { -if (likely (!atomic_read (&event->mmap_count))) +if (likely (!refcount_read (&event->mmap_count)))return false;perf_event_update_time (event);
@@ -6704,11 +6704,11 @@ static void perf_mmap_open(struct vm_area_struct *vma)struct perf_event *event = vma->vm_file->private_data;mapped_f mapped =get_mapped (event, event_mapped); -atomic_inc (&event->mmap_count); -atomic_inc (&event->rb->mmap_count); +refcount_inc (&event->mmap_count); +refcount_inc (&event->rb->mmap_count);if (vma->vm_pgoff) -atomic_inc (&event->rb->aux_mmap_count); +refcount_inc (&event->rb->aux_mmap_count);if (mapped)mapped (event, vma->vm_mm);
@@ -6743,7 +6743,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)* to avoid complications. */ if (rb_has_aux (rb) && vma->vm_pgoff == rb->aux_pgoff && -atomic_dec_and_mutex_lock (&rb->aux_mmap_count, &rb->aux_mutex)) { +refcount_dec_and_mutex_lock (&rb->aux_mmap_count, &rb->aux_mutex)) {/* * Stop all AUX events that are writing to this buffer, * so that we can free its AUX pages and corresponding PMU
@@ -6763,10 +6763,10 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&rb->aux_mutex); } -if (atomic_dec_and_test (&rb->mmap_count)) +if (refcount_dec_and_test (&rb->mmap_count)) detach_rest = true; -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) +if (!refcount_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;ring_buffer_attach (event,NULL );
@@ -6992,19 +6992,19 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event,if (data_page_nr (event->rb) != nr_pages)return -EINVAL ; -if (atomic_inc_not_zero (&event->rb->mmap_count)) { +if (refcount_inc_not_zero (&event->rb->mmap_count)) {/* * Success -- managed to mmap() the same buffer * multiple times. */ perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +refcount_inc (&event->mmap_count);return 0 ; }/* * Raced against perf_mmap_close()'s -* atomic_dec_and_mutex_lock() remove the +* refcount_dec_and_mutex_lock() remove the * event and continue as if !event->rb */ ring_buffer_attach (event,NULL );
@@ -7023,7 +7023,7 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event,if (!rb)return -ENOMEM ; -atomic_set (&rb->mmap_count,1 ); +refcount_set (&rb->mmap_count,1 ); rb->mmap_user =get_current_user (); rb->mmap_locked = extra;
@@ -7034,7 +7034,7 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event,perf_event_update_userpage (event);perf_mmap_account (vma, user_extra, extra); -atomic_set (&event->mmap_count,1 ); +refcount_set (&event->mmap_count,1 );return 0 ; }
@@ -7081,15 +7081,15 @@ static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event,if (!is_power_of_2 (nr_pages))return -EINVAL ; -if (!atomic_inc_not_zero (&rb->mmap_count)) +if (!refcount_inc_not_zero (&rb->mmap_count))return -EINVAL ;if (rb_has_aux (rb)) { -atomic_inc (&rb->aux_mmap_count); +refcount_inc (&rb->aux_mmap_count); }else {if (!perf_mmap_calc_limits (vma, &user_extra, &extra)) { -atomic_dec (&rb->mmap_count); +refcount_dec (&rb->mmap_count);return -EPERM ; }
@@ -7101,16 +7101,16 @@ static int perf_mmap_aux(struct vm_area_struct *vma, struct perf_event *event, ret =rb_alloc_aux (rb, event, vma->vm_pgoff, nr_pages, event->attr.aux_watermark, rb_flags);if (ret) { -atomic_dec (&rb->mmap_count); +refcount_dec (&rb->mmap_count);return ret; } -atomic_set (&rb->aux_mmap_count,1 ); +refcount_set (&rb->aux_mmap_count,1 ); rb->aux_mmap_locked = extra; }perf_mmap_account (vma, user_extra, extra); -atomic_inc (&event->mmap_count); +refcount_inc (&event->mmap_count);return 0 ; }
@@ -13254,7 +13254,7 @@ perf_event_set_output(struct perf_event *event, struct perf_event *output_event)mutex_lock_double (&event->mmap_mutex, &output_event->mmap_mutex); set:/* Can't redirect output if we've got an active mmap() */ -if (atomic_read (&event->mmap_count)) +if (refcount_read (&event->mmap_count)) goto unlock;if (output_event) {
@@ -13267,7 +13267,7 @@ set: goto unlock;/* did we race against perf_mmap_close() */ -if (!atomic_read (&rb->mmap_count)) { +if (!refcount_read (&rb->mmap_count)) {ring_buffer_put (rb); goto unlock; }
kernel/events/internal.h +2 -2
@@ -35,7 +35,7 @@ struct perf_buffer {spinlock_t event_lock;struct list_head event_list; -atomic_t mmap_count; +refcount_t mmap_count;unsigned long mmap_locked;struct user_struct *mmap_user;
@@ -47,7 +47,7 @@ struct perf_buffer {unsigned long aux_pgoff;int aux_nr_pages;int aux_overwrite; -atomic_t aux_mmap_count; +refcount_t aux_mmap_count;unsigned long aux_mmap_locked;void (*free_aux)(void *);refcount_t aux_refcount;
kernel/events/ring_buffer.c +1 -1
@@ -400,7 +400,7 @@ void *perf_aux_output_begin(struct perf_output_handle *handle,* the same order, see perf_mmap_close. Otherwise we end up freeing * aux pages in this path, which is a bug, because in_atomic(). */ -if (!atomic_read (&rb->aux_mmap_count)) +if (!refcount_read (&rb->aux_mmap_count)) goto err;if (!refcount_inc_not_zero (&rb->aux_refcount))
Quoted 26 lines · Peter Zijlstra
Took over the series from Thomas; much thanks to him for cleaning this up.
The recently fixed reference count leaks could have been detected by using
refcount_tandrefcount_twould have mitigated the potential overflow at least.It turned out that converting the code as is does not work as the allocation code ends up doing a
refcount_inc()for the first allocation, which causesrefcount_tsanity checks to emit a UAF warning.The reason is that the code is sharing functionality at the wrong level and ends up being overly complicated for no reason. That's what inevitable led to the refcount leak problems.
Address this by splitting the ringbuffer and the AUX buffer mapping and allocation parts out into seperate functions, which handle the reference counts in a sane way.
That not only simplifies the code and makes it halfways comprehensible, but also allows to convert the
mmap()related reference counts torefcount_t.It survives lightweight testing with perf and passes the perf/mmap selftest.
Also available at:
Quoted 8 lines
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.gitperf/core---
v2: https://lkml.kernel.org/r/[email protected]Changes vs v2:
- replaced patches 4,5 with fine grained steps
Whole series LGTM, various nits/comments but nothing of importance.
I also build-checked every commit and checked the new mmap self test all looks good.
Thanks for this guys, huge improvement! :)