perf: Fix race in perf_mmap_close function
Jiri Olsa
reviewerPATCH 1/1 perf: Fix race in perf_mmap_close function
There's a possible race in perf_mmap_close when checking ring buffer's mmap_count refcount value. The problem is that the mmap_count check is not atomic because we call atomic_dec and atomic_read separately.
perf_mmap_close:
...
atomic_dec(&rb->mmap_count);
...
if (atomic_read(&rb->mmap_count))
goto out_put; <ring buffer detach>
free_uid
out_put:
ring_buffer_put(rb); /* could be last */
The race can happen when we have two (or more) events sharing same ring buffer and they go through atomic_dec and then they both see 0 as refcount value later in atomic_read. Then both will go on and execute code which is meant to be run just once.
The code that detaches ring buffer is probably fine to be executed more than once, but the problem is in calling free_uid, which will later on demonstrate in related crashes and refcount warnings, like:
refcount_t: addition on 0; use-after-free.
...
RIP: 0010:refcount_warn_saturate+0x6d/0xf ... Call Trace: prepare_creds+0x190/0x1e0 copy_creds+0x35/0x172 copy_process+0x471/0x1a80 _do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
--- kernel/events/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/events/core.c +1 -3
@@ -5903,8 +5903,6 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;
@@ -5912,7 +5910,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!atomic_dec_and_test (&rb->mmap_count)) goto out_put;/*
Signature · 1 line
2.26.2
Namhyung Kim
maintainerQuoted 54 lines · Jiri Olsa
There's a possible race in
perf_mmap_closewhen checking ring buffer'smmap_countrefcount value. The problem is that themmap_countcheck is not atomic because we callatomic_decandatomic_readseparately.perf_mmap_close: ... atomic_dec(&rb->mmap_count); ... if (atomic_read(&rb->mmap_count)) goto out_put;<ring buffer detach>
free_uid
out_put:
ring_buffer_put(rb); /* could be last */The race can happen when we have two (or more) events sharing same ring buffer and they go through
atomic_decand then they both see 0 as refcount value later inatomic_read. Then both will go on and execute code which is meant to be run just once.The code that detaches ring buffer is probably fine to be executed more than once, but the problem is in calling
free_uid, which will later on demonstrate in related crashes and refcount warnings, like:
refcount_t: addition on 0; use-after-free.
...RIP: 0010:refcount_warn_saturate+0x6d/0xf ... Call Trace: prepare_creds+0x190/0x1e0 copy_creds+0x35/0x172 copy_process+0x471/0x1a80 _do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
Signed-off-by Jiri Olsa <[email protected]>--- kernel/events/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)kernel/events/core.c +0
-0
kernel/events/core.c static void perf_mmap_close(struct vm_area_struct *vma) · line 5907
Quoted 7 lines
kernel/events/core.c +0
-2@@ -5903,8 +5903,6 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;
But when it takes the goto, rb->mmap_count won't decrement anymore..
Thanks
Namhyung
Quoted 13 lines
kernel/events/core.c +1
-2@@ -5912,7 +5910,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!atomic_dec_and_test (&rb->mmap_count)) goto out_put;/* -- 2.26.2
On Thu, Sep 10, 2020 at 10:48:02PM +0900, Namhyung Kim wrote:
SNIP
Quoted 22 lines
Quoted 22 lines
_do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
Signed-off-by Jiri Olsa <[email protected]>--- kernel/events/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)kernel/events/core.c +0
-0@@ -5903,8 +5903,6 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); }
Quoted 6 lines
kernel/events/core.c static void perf_mmap_close(struct vm_area_struct *vma) · line 5907
Quoted 4 lines
kernel/events/core.c +0
-2-atomic_dec (&rb->mmap_count); -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;But when it takes the goto,
rb->mmap_countwon't decrement anymore..
event->mmap_count is per event, so if we have have race in here, 2 threads can go through with each event->mmap_count reaching zero
jirka
Namhyung Kim
maintainerHi Jiri,
Quoted 34 lines · Jiri Olsa
On Thu, Sep 10, 2020 at 10:48:02PM +0900, Namhyung Kim wrote:
SNIP
Quoted 22 lines
Quoted 22 lines
_do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
Signed-off-by Jiri Olsa <[email protected]>--- kernel/events/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)kernel/events/core.c +0
-0@@ -5903,8 +5903,6 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); }Quoted 6 lines
kernel/events/core.c static void perf_mmap_close(struct vm_area_struct *vma) · line 5907
Quoted 4 lines
kernel/events/core.c +0
-2-atomic_dec (&rb->mmap_count); -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;But when it takes the goto,
rb->mmap_countwon't decrement anymore..
Quoted 2 lines
event->mmap_countis per event, so if we have have race in here, 2 threads can go through with eachevent->mmap_countreaching zero
Maybe I'm missing something.
But as far as I can see, perf_mmap_close() always decremented both rb->mmap_count and event->mmap_count. But with this change, it seems not decrement rb->mmap_count when event->mmap_count doesn't go to zero, right?
Thanks
Namhyung
Quoted 40 lines · Namhyung Kim
Hi Jiri,
Quoted 34 lines · Jiri Olsa
On Thu, Sep 10, 2020 at 10:48:02PM +0900, Namhyung Kim wrote:
SNIP
Quoted 22 lines
Quoted 22 lines
_do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
Signed-off-by Jiri Olsa <[email protected]>--- kernel/events/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)kernel/events/core.c +0
-0@@ -5903,8 +5903,6 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); }Quoted 6 lines
kernel/events/core.c static void perf_mmap_close(struct vm_area_struct *vma) · line 5907
Quoted 4 lines
kernel/events/core.c +0
-2-atomic_dec (&rb->mmap_count); -if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;But when it takes the goto,
rb->mmap_countwon't decrement anymore..Quoted 2 lines
event->mmap_countis per event, so if we have have race in here, 2 threads can go through with eachevent->mmap_countreaching zero
Quoted 6 lines
Maybe I'm missing something.
But as far as I can see,
perf_mmap_close()always decremented bothrb->mmap_countandevent->mmap_count. But with this change, it seems not decrementrb->mmap_countwhenevent->mmap_countdoesn't go to zero, right?
ugh, that's right.. how about change below
jirka
---
kernel/events/core.c +4 -3
@@ -5868,11 +5868,11 @@ static void perf_pmu_output_stop(struct perf_event *event);static void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; -struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;unsigned long size =perf_data_size (rb); +bool detach_rest = false;if (event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);
@@ -5903,7 +5903,8 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); +if (atomic_dec_and_test (&rb->mmap_count)) + detach_rest = true;if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;
@@ -5912,7 +5913,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!detach_rest) goto out_put;/*
Quoted 1 line · Jiri Olsa
ugh, that's right.. how about change below
Thanks
Namhyung
Quoted 42 lines
jirka
---
kernel/events/core.c +4
-3@@ -5868,11 +5868,11 @@ static void perf_pmu_output_stop(struct perf_event *event);static void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; -struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;unsigned long size =perf_data_size (rb); +bool detach_rest = false;if (event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);@@ -5903,7 +5903,8 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); +if (atomic_dec_and_test (&rb->mmap_count)) + detach_rest = true;if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;@@ -5912,7 +5913,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!detach_rest) goto out_put;/*
Quoted 4 lines · Namhyung Kim
Quoted 1 line · Jiri Olsa
ugh, that's right.. how about change below
Acked-by Namhyung Kim <[email protected]>
thanks, I'll send full patch after we're done testing this
jirka
Quoted 48 lines
Thanks
NamhyungQuoted 42 lines
jirka
---
kernel/events/core.c +4
-3@@ -5868,11 +5868,11 @@ static void perf_pmu_output_stop(struct perf_event *event);static void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; -struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;unsigned long size =perf_data_size (rb); +bool detach_rest = false;if (event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);@@ -5903,7 +5903,8 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); +if (atomic_dec_and_test (&rb->mmap_count)) + detach_rest = true;if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;@@ -5912,7 +5913,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!detach_rest) goto out_put;/*
Michael Petlan
Tested-byQuoted 9 lines · Jiri Olsa
Quoted 4 lines · Namhyung Kim
Quoted 1 line · Jiri Olsa
ugh, that's right.. how about change below
Acked-by Namhyung Kim <[email protected]>thanks, I'll send full patch after we're done testing this
jirka
I've tested the change and seems OK to me.
Quoted 50 lines
Quoted 49 lines
Thanks
NamhyungQuoted 42 lines
jirka
---
kernel/events/core.c +4
-3@@ -5868,11 +5868,11 @@ static void perf_pmu_output_stop(struct perf_event *event);static void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; -struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;unsigned long size =perf_data_size (rb); +bool detach_rest = false;if (event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);@@ -5903,7 +5903,8 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); +if (atomic_dec_and_test (&rb->mmap_count)) + detach_rest = true;if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;@@ -5912,7 +5913,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!detach_rest) goto out_put;/*
PATCH v2 1/1 perf: Fix race in perf_mmap_close function
There's a possible race in perf_mmap_close when checking ring buffer's mmap_count refcount value. The problem is that the mmap_count check is not atomic because we call atomic_dec and atomic_read separately.
perf_mmap_close:
...
atomic_dec(&rb->mmap_count);
...
if (atomic_read(&rb->mmap_count))
goto out_put; <ring buffer detach>
free_uid
out_put:
ring_buffer_put(rb); /* could be last */
The race can happen when we have two (or more) events sharing same ring buffer and they go through atomic_dec and then they both see 0 as refcount value later in atomic_read. Then both will go on and execute code which is meant to be run just once.
The code that detaches ring buffer is probably fine to be executed more than once, but the problem is in calling free_uid, which will later on demonstrate in related crashes and refcount warnings, like:
refcount_t: addition on 0; use-after-free.
...
RIP: 0010:refcount_warn_saturate+0x6d/0xf ... Call Trace: prepare_creds+0x190/0x1e0 copy_creds+0x35/0x172 copy_process+0x471/0x1a80 _do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
--- kernel/events/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/events/core.c +4 -3
@@ -5868,11 +5868,11 @@ static void perf_pmu_output_stop(struct perf_event *event);static void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; -struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;unsigned long size =perf_data_size (rb); +bool detach_rest = false;if (event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);
@@ -5903,7 +5903,8 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); +if (atomic_dec_and_test (&rb->mmap_count)) + detach_rest = true;if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;
@@ -5912,7 +5913,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!detach_rest) goto out_put;/*
Signature · 1 line
2.26.2
[email protected]
maintainerQuoted 40 lines · Jiri Olsa
There's a possible race in
perf_mmap_closewhen checking ring buffer'smmap_countrefcount value. The problem is that themmap_countcheck is not atomic because we callatomic_decandatomic_readseparately.perf_mmap_close: ... atomic_dec(&rb->mmap_count); ... if (atomic_read(&rb->mmap_count)) goto out_put;<ring buffer detach>
free_uid
out_put:
ring_buffer_put(rb); /* could be last */The race can happen when we have two (or more) events sharing same ring buffer and they go through
atomic_decand then they both see 0 as refcount value later inatomic_read. Then both will go on and execute code which is meant to be run just once.The code that detaches ring buffer is probably fine to be executed more than once, but the problem is in calling
free_uid, which will later on demonstrate in related crashes and refcount warnings, like:
refcount_t: addition on 0; use-after-free.
...RIP: 0010:refcount_warn_saturate+0x6d/0xf ... Call Trace: prepare_creds+0x190/0x1e0 copy_creds+0x35/0x172 copy_process+0x471/0x1a80 _do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 2 lines
Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
I'm tempted to remove that line; I can never seem to find anything useful in a CVE.
Quoted 3 lines
Acked-by Namhyung Kim <[email protected]>Tested-by Michael Petlan <[email protected]>Signed-off-by Jiri Olsa <[email protected]>
Quoted 41 lines · On Wed, Sep 16, 2020 at 03:54:02PM +0200, [email protected] wrote:
Quoted 40 lines · Jiri Olsa
There's a possible race in
perf_mmap_closewhen checking ring buffer'smmap_countrefcount value. The problem is that themmap_countcheck is not atomic because we callatomic_decandatomic_readseparately.perf_mmap_close: ... atomic_dec(&rb->mmap_count); ... if (atomic_read(&rb->mmap_count)) goto out_put;<ring buffer detach>
free_uid
out_put:
ring_buffer_put(rb); /* could be last */The race can happen when we have two (or more) events sharing same ring buffer and they go through
atomic_decand then they both see 0 as refcount value later inatomic_read. Then both will go on and execute code which is meant to be run just once.The code that detaches ring buffer is probably fine to be executed more than once, but the problem is in calling
free_uid, which will later on demonstrate in related crashes and refcount warnings, like:
refcount_t: addition on 0; use-after-free.
...RIP: 0010:refcount_warn_saturate+0x6d/0xf ... Call Trace: prepare_creds+0x190/0x1e0 copy_creds+0x35/0x172 copy_process+0x471/0x1a80 _do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 5 lines
Quoted 2 lines
Using atomic decrease and check instead of separated calls. This fixes CVE-2020-14351.
I'm tempted to remove that line; I can never seem to find anything useful in a CVE.
I was asked by security guys to add this, Wade?
Quoted 2 lines
Fixes ?
right, sry..
thanks,
jirka
Quoted 5 lines
Quoted 4 lines
Acked-by Namhyung Kim <[email protected]>Tested-by Michael Petlan <[email protected]>Signed-off-by Jiri Olsa <[email protected]>
[email protected]
maintainerQuoted 14 lines · Jiri Olsa
There's a possible race in
perf_mmap_closewhen checking ring buffer'smmap_countrefcount value. The problem is that themmap_countcheck is not atomic because we callatomic_decandatomic_readseparately.perf_mmap_close: ... atomic_dec(&rb->mmap_count); ... if (atomic_read(&rb->mmap_count)) goto out_put;<ring buffer detach>
free_uid
Quoted 7 lines
out_put:
ring_buffer_put(rb); /* could be last */The race can happen when we have two (or more) events sharing same ring buffer and they go through
atomic_decand then they both see 0 as refcount value later inatomic_read. Then both will go on and execute code which is meant to be run just once.
The trival case should be protected by mmap_sem, we call vm_ops->close() with mmap_sem held for writing IIRC. But yes, I think it's possible to construct a failure here.
[tip: perf/core] perf/core: Fix race in the perf_mmap_close() function
In reply to Jiri OlsaThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: f91072ed1b7283b13ca57fcfbece5a3b92726143 Gitweb: https://git.kernel.org/tip/f91072ed1b7283b13ca57fcfbece5a3b92726143 Author: Jiri Olsa <[email protected]> AuthorDate: Wed, 16 Sep 2020 13:53:11 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 12 Oct 2020 13:24:26 +02:00
perf/core: Fix race in the perf_mmap_close() function
There's a possible race in perf_mmap_close() when checking ring buffer's mmap_count refcount value. The problem is that the mmap_count check is not atomic because we call atomic_dec() and atomic_read() separately.
perf_mmap_close:
...
atomic_dec(&rb->mmap_count);
...
if (atomic_read(&rb->mmap_count))
goto out_put; <ring buffer detach>
free_uid
out_put:
ring_buffer_put(rb); /* could be last */
The race can happen when we have two (or more) events sharing same ring buffer and they go through atomic_dec() and then they both see 0 as refcount value later in atomic_read(). Then both will go on and execute code which is meant to be run just once.
The code that detaches ring buffer is probably fine to be executed more than once, but the problem is in calling free_uid(), which will later on demonstrate in related crashes and refcount warnings, like:
refcount_t: addition on 0; use-after-free.
...
RIP: 0010:refcount_warn_saturate+0x6d/0xf ... Call Trace: prepare_creds+0x190/0x1e0 copy_creds+0x35/0x172 copy_process+0x471/0x1a80 _do_fork+0x83/0x3a0 __do_sys_wait4+0x83/0x90 __do_sys_clone+0x85/0xa0 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Using atomic decrease and check instead of separated calls.
--- kernel/events/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/events/core.c +4 -3
@@ -5895,11 +5895,11 @@ static void perf_pmu_output_stop(struct perf_event *event);static void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; -struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;unsigned long size =perf_data_size (rb); +bool detach_rest = false;if (event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);
@@ -5930,7 +5930,8 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex); } -atomic_dec (&rb->mmap_count); +if (atomic_dec_and_test (&rb->mmap_count)) + detach_rest = true;if (!atomic_dec_and_mutex_lock (&event->mmap_count, &event->mmap_mutex)) goto out_put;
@@ -5939,7 +5940,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)mutex_unlock (&event->mmap_mutex);/* If there's still other mmap()s of this buffer, we're done. */ -if (atomic_read (&rb->mmap_count)) +if (!detach_rest) goto out_put;/*