Skip to contentSkip to conversation

perf: Intel PT and 64-bit AUX area size fixes

PATCH Applied Adrian Hunter 2 participants 15 messages Jun 24, 2024 – Jul 5, 2024

Subsystems (3)
lore
Share & tools
lore

Adrian Hunter

author reviewer
#0 lore raw

cover perf: Intel PT and 64-bit AUX area size fixes

Hi

Here are some fixes.

I have taken the liberty of re-posting Marco Cavenati's patch from:

	https://lore.kernel.org/lkml/[email protected]/

with updated commit suggested by Dave Hansen.

Adrian Hunter (6):
perf/x86/intel/pt: Fix a topa_entry base address calculation perf/x86/intel/pt: Fix pt_topa_entry_for_page() address calculation
perf: Fix perf_aux_size() for greater-than 32-bit size
perf: Prevent passing zero nr_pages to rb_alloc_aux()
perf: Fix default aux_watermark calculation
perf: Make rb_alloc_aux() return an error immediately if nr_pages <= 0

Marco Cavenati (1):
perf/x86/intel/pt: Fix topa_entry base length

 arch/x86/events/intel/pt.c  | 4 ++--
 arch/x86/events/intel/pt.h  | 4 ++--
 kernel/events/core.c        | 2 ++
 kernel/events/internal.h    | 2 +-
 kernel/events/ring_buffer.c | 7 ++++++-
 5 files changed, 13 insertions(+), 6 deletions(-)

Regards
Adrian

author reviewer
#1 lore raw

PATCH 1/7 perf/x86/intel/pt: Fix topa_entry base length

From: Marco Cavenati <[email protected]>

topa_entry->base needs to store a pfn. It obviously needs to be large enough to store the largest possible x86 pfn which is MAXPHYADDR-PAGE_SIZE (52-12). So it is 4 bits too small.

Increase the size of topa_entry->base from 36 bits to 40 bits.

Note, systems where physical addresses can be 256TiB or more are affected.

[ Adrian: Amend commit message as suggested by Dave Hansen ]

Fixes 52ca9ced3f70 ("perf/x86/intel/pt: Add Intel PT PMU driver")
---
 arch/x86/events/intel/pt.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/events/intel/pt.h +2 -2
index 96906a62aacd..f5e46c04c145 100644
@@ -33,8 +33,8 @@ struct topa_entry {
 	u64	rsvd2	: 1;
 	u64	size	: 4;
 	u64	rsvd3	: 2;
-	u64	base	: 36;
-	u64	rsvd4	: 16;
+	u64	base	: 40;
+	u64	rsvd4	: 12;
 };
 
 /* TSC to Core Crystal Clock Ratio */
Signature · 1 line
2.34.1
merge bot Applied
#2 lore raw

[tip: perf/core] perf/x86/intel/pt: Fix topa_entry base length

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     5638bd722a44bbe97c1a7b3fae5b9efddb3e70ff
Gitweb:        https://git.kernel.org/tip/5638bd722a44bbe97c1a7b3fae5b9efddb3e70ff
Author:        Marco Cavenati <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:10:55 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:20 +02:00

perf/x86/intel/pt: Fix topa_entry base length

topa_entry->base needs to store a pfn. It obviously needs to be large enough to store the largest possible x86 pfn which is MAXPHYADDR-PAGE_SIZE (52-12). So it is 4 bits too small.

Increase the size of topa_entry->base from 36 bits to 40 bits.

Note, systems where physical addresses can be 256TiB or more are affected.

[ Adrian: Amend commit message as suggested by Dave Hansen ]

---
 arch/x86/events/intel/pt.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/events/intel/pt.h +2 -2
index 96906a6..f5e46c0 100644
@@ -33,8 +33,8 @@ struct topa_entry {
 	u64	rsvd2	: 1;
 	u64	size	: 4;
 	u64	rsvd3	: 2;
-	u64	base	: 36;
-	u64	rsvd4	: 16;
+	u64	base	: 40;
+	u64	rsvd4	: 12;
 };
 
 /* TSC to Core Crystal Clock Ratio */
author reviewer
#3 lore raw

PATCH 2/7 perf/x86/intel/pt: Fix a topa_entry base address calculation

In reply to Adrian Hunter

topa_entry->base is a bit-field. Bit-fields are not promoted to a 64-bit type, even if the underlying type is 64-bit, and so, if necessary, must be cast to a larger type when calculations are done.

Fix a topa_entry->base address calculation by adding a cast.

Without the cast, the address was limited to 36-bits i.e. 64GiB.

The address calculation is used on systems that do not support Multiple Entry ToPA (only Broadwell), and affects physical addresses on or above 64GiB. Instead of writing to the correct address, the address comprising the first 36 bits would be written to.

Intel PT snapshot and sampling modes are not affected.

Fixes 52ca9ced3f70 ("perf/x86/intel/pt: Add Intel PT PMU driver")
---
 arch/x86/events/intel/pt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/pt.c +1 -1
index 14db6d9d318b..047a2cd5b3fe 100644
@@ -878,7 +878,7 @@ static void pt_update_head(struct pt *pt)
  */
 static void *pt_buffer_region(struct pt_buffer *buf)
 {
-	return phys_to_virt(TOPA_ENTRY(buf->cur, buf->cur_idx)->base << TOPA_SHIFT);
+	return phys_to_virt((phys_addr_t)TOPA_ENTRY(buf->cur, buf->cur_idx)->base << TOPA_SHIFT);
 }
 
 /**
Signature · 1 line
2.34.1
merge bot Applied
#4 lore raw

[tip: perf/core] perf/x86/intel/pt: Fix a topa_entry base address calculation

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     ad97196379d0b8cb24ef3d5006978a6554e6467f
Gitweb:        https://git.kernel.org/tip/ad97196379d0b8cb24ef3d5006978a6554e6467f
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:10:56 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:21 +02:00

perf/x86/intel/pt: Fix a topa_entry base address calculation

topa_entry->base is a bit-field. Bit-fields are not promoted to a 64-bit type, even if the underlying type is 64-bit, and so, if necessary, must be cast to a larger type when calculations are done.

Fix a topa_entry->base address calculation by adding a cast.

Without the cast, the address was limited to 36-bits i.e. 64GiB.

The address calculation is used on systems that do not support Multiple Entry ToPA (only Broadwell), and affects physical addresses on or above 64GiB. Instead of writing to the correct address, the address comprising the first 36 bits would be written to.

Intel PT snapshot and sampling modes are not affected.

---
 arch/x86/events/intel/pt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/pt.c +1 -1
index 14db6d9..047a2cd 100644
@@ -878,7 +878,7 @@ static void pt_update_head(struct pt *pt)
  */
 static void *pt_buffer_region(struct pt_buffer *buf)
 {
-	return phys_to_virt(TOPA_ENTRY(buf->cur, buf->cur_idx)->base << TOPA_SHIFT);
+	return phys_to_virt((phys_addr_t)TOPA_ENTRY(buf->cur, buf->cur_idx)->base << TOPA_SHIFT);
 }
 
 /**
author reviewer
#5 lore raw

PATCH 3/7 perf/x86/intel/pt: Fix pt_topa_entry_for_page() address calculation

In reply to Adrian Hunter

Currently, perf allocates an array of page pointers which is limited in size by MAX_PAGE_ORDER. That in turn limits the maximum Intel PT buffer size to 2GiB. Should that limitation be lifted, the Intel PT driver can support larger sizes, except for one calculation in pt_topa_entry_for_page(), which is limited to 32-bits.

Fix pt_topa_entry_for_page() address calculation by adding a cast.

Fixes 39152ee51b77 ("perf/x86/intel/pt: Get rid of reverse lookup table for ToPA")
---
 arch/x86/events/intel/pt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/pt.c +1 -1
index 047a2cd5b3fe..b4aa8daa4773 100644
@@ -990,7 +990,7 @@ pt_topa_entry_for_page(struct pt_buffer *buf, unsigned int pg)
 	 * order allocations, there shouldn't be many of these.
 	 */
 	list_for_each_entry(topa, &buf->tables, list) {
-		if (topa->offset + topa->size > pg << PAGE_SHIFT)
+		if (topa->offset + topa->size > (unsigned long)pg << PAGE_SHIFT)
 			goto found;
 	}
 
Signature · 1 line
2.34.1
merge bot Applied
#6 lore raw

[tip: perf/core] perf/x86/intel/pt: Fix pt_topa_entry_for_page() address calculation

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     3520b251dcae2b4a27b95cd6f745c54fd658bda5
Gitweb:        https://git.kernel.org/tip/3520b251dcae2b4a27b95cd6f745c54fd658bda5
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:10:57 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:21 +02:00

perf/x86/intel/pt: Fix pt_topa_entry_for_page() address calculation

Currently, perf allocates an array of page pointers which is limited in size by MAX_PAGE_ORDER. That in turn limits the maximum Intel PT buffer size to 2GiB. Should that limitation be lifted, the Intel PT driver can support larger sizes, except for one calculation in pt_topa_entry_for_page(), which is limited to 32-bits.

Fix pt_topa_entry_for_page() address calculation by adding a cast.

Fixes 39152ee51b77 ("perf/x86/intel/pt: Get rid of reverse lookup table for ToPA")
---
 arch/x86/events/intel/pt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/pt.c +1 -1
index 047a2cd..b4aa8da 100644
@@ -990,7 +990,7 @@ pt_topa_entry_for_page(struct pt_buffer *buf, unsigned int pg)
 	 * order allocations, there shouldn't be many of these.
 	 */
 	list_for_each_entry(topa, &buf->tables, list) {
-		if (topa->offset + topa->size > pg << PAGE_SHIFT)
+		if (topa->offset + topa->size > (unsigned long)pg << PAGE_SHIFT)
 			goto found;
 	}
 
author reviewer
#7 lore raw

PATCH 4/7 perf: Fix perf_aux_size() for greater-than 32-bit size

In reply to Adrian Hunter

perf_buffer->aux_nr_pages uses a 32-bit type, so a cast is needed to calculate a 64-bit size.

Fixes 45bfb2e50471 ("perf: Add AUX area to ring buffer for raw data streams")
---
 kernel/events/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/internal.h +1 -1
index 5150d5f84c03..386d21c7edfa 100644
@@ -128,7 +128,7 @@ static inline unsigned long perf_data_size(struct perf_buffer *rb)
 
 static inline unsigned long perf_aux_size(struct perf_buffer *rb)
 {
-	return rb->aux_nr_pages << PAGE_SHIFT;
+	return (unsigned long)rb->aux_nr_pages << PAGE_SHIFT;
 }
 
 #define __DEFINE_OUTPUT_COPY_BODY(advance_buf, memcpy_func, ...)	\
Signature · 1 line
2.34.1
merge bot Applied
#8 lore raw

[tip: perf/core] perf: Fix perf_aux_size() for greater-than 32-bit size

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     3df94a5b1078dfe2b0c03f027d018800faf44c82
Gitweb:        https://git.kernel.org/tip/3df94a5b1078dfe2b0c03f027d018800faf44c82
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:10:58 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:22 +02:00

perf: Fix perf_aux_size() for greater-than 32-bit size

perf_buffer->aux_nr_pages uses a 32-bit type, so a cast is needed to calculate a 64-bit size.

Fixes 45bfb2e50471 ("perf: Add AUX area to ring buffer for raw data streams")
---
 kernel/events/internal.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/internal.h +1 -1
index 5150d5f..386d21c 100644
@@ -128,7 +128,7 @@ static inline unsigned long perf_data_size(struct perf_buffer *rb)
 
 static inline unsigned long perf_aux_size(struct perf_buffer *rb)
 {
-	return rb->aux_nr_pages << PAGE_SHIFT;
+	return (unsigned long)rb->aux_nr_pages << PAGE_SHIFT;
 }
 
 #define __DEFINE_OUTPUT_COPY_BODY(advance_buf, memcpy_func, ...)	\
author reviewer
#9 lore raw

PATCH 5/7 perf: Prevent passing zero nr_pages to rb_alloc_aux()

In reply to Adrian Hunter

nr_pages is unsigned long but gets passed to rb_alloc_aux() as an int, and is stored as an int.

Only power-of-2 values are accepted, so if nr_pages is a 64_bit value, it will be passed to rb_alloc_aux() as zero.

That is not ideal because:
1. the value is incorrect
2. rb_alloc_aux() is at risk of misbehaving, although it manages to return -ENOMEM in that case, it is a result of passing zero to get_order() even though the get_order() result is documented to be undefined in that case.

Fix by simply validating the maximum supported value in the first place. Use -ENOMEM error code for consistency with the current error code that is returned in that case.

Fixes 45bfb2e50471 ("perf: Add AUX area to ring buffer for raw data streams")
---
 kernel/events/core.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/events/core.c +2 -0
index 8f908f077935..053e546d5bf0 100644
@@ -6509,6 +6509,8 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
 			return -EINVAL;
 
 		nr_pages = vma_size / PAGE_SIZE;
+		if (nr_pages > INT_MAX)
+			return -ENOMEM;
 
 		mutex_lock(&event->mmap_mutex);
 		ret = -EINVAL;
Signature · 1 line
2.34.1
merge bot Applied
#10 lore raw

[tip: perf/core] perf: Prevent passing zero nr_pages to rb_alloc_aux()

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     dbc48c8f41c208082cfa95e973560134489e3309
Gitweb:        https://git.kernel.org/tip/dbc48c8f41c208082cfa95e973560134489e3309
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:10:59 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:22 +02:00

perf: Prevent passing zero nr_pages to rb_alloc_aux()

nr_pages is unsigned long but gets passed to rb_alloc_aux() as an int, and is stored as an int.

Only power-of-2 values are accepted, so if nr_pages is a 64_bit value, it will be passed to rb_alloc_aux() as zero.

That is not ideal because:
1. the value is incorrect
2. rb_alloc_aux() is at risk of misbehaving, although it manages to return -ENOMEM in that case, it is a result of passing zero to get_order() even though the get_order() result is documented to be undefined in that case.

Fix by simply validating the maximum supported value in the first place. Use -ENOMEM error code for consistency with the current error code that is returned in that case.

Fixes 45bfb2e50471 ("perf: Add AUX area to ring buffer for raw data streams")
---
 kernel/events/core.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/events/core.c +2 -0
index 6b0a66e..e70d3dd 100644
@@ -6496,6 +6496,8 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
 			return -EINVAL;
 
 		nr_pages = vma_size / PAGE_SIZE;
+		if (nr_pages > INT_MAX)
+			return -ENOMEM;
 
 		mutex_lock(&event->mmap_mutex);
 		ret = -EINVAL;
author reviewer
#11 lore raw

PATCH 6/7 perf: Fix default aux_watermark calculation

In reply to Adrian Hunter

The default aux_watermark is half the AUX area buffer size. In general, on a 64-bit architecture, the AUX area buffer size could be a bigger than fits in a 32-bit type, but the calculation does not allow for that possibility.

However the aux_watermark value is recorded in a u32, so should not be more than U32_MAX either.

Fix by doing the calculation in a correctly sized type, and limiting the result to U32_MAX.

Fixes d68e6799a5c8 ("perf: Cap allocation order at aux_watermark")
---
 kernel/events/ring_buffer.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/events/ring_buffer.c +3 -1
index 4013408ce012..485cf0a66631 100644
@@ -688,7 +688,9 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
 		 * max_order, to aid PMU drivers in double buffering.
 		 */
 		if (!watermark)
-			watermark = nr_pages << (PAGE_SHIFT - 1);
+			watermark = min_t(unsigned long,
+					  U32_MAX,
+					  (unsigned long)nr_pages << (PAGE_SHIFT - 1));
 
 		/*
 		 * Use aux_watermark as the basis for chunking to
Signature · 1 line
2.34.1
merge bot Applied
#12 lore raw

[tip: perf/core] perf: Fix default aux_watermark calculation

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     43deb76b19663a96ec2189d8f4eb9a9dc2d7623f
Gitweb:        https://git.kernel.org/tip/43deb76b19663a96ec2189d8f4eb9a9dc2d7623f
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:11:00 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:23 +02:00

perf: Fix default aux_watermark calculation

The default aux_watermark is half the AUX area buffer size. In general, on a 64-bit architecture, the AUX area buffer size could be a bigger than fits in a 32-bit type, but the calculation does not allow for that possibility.

However the aux_watermark value is recorded in a u32, so should not be more than U32_MAX either.

Fix by doing the calculation in a correctly sized type, and limiting the result to U32_MAX.

Fixes d68e6799a5c8 ("perf: Cap allocation order at aux_watermark")
---
 kernel/events/ring_buffer.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/events/ring_buffer.c +3 -1
index 4013408..485cf0a 100644
@@ -688,7 +688,9 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
 		 * max_order, to aid PMU drivers in double buffering.
 		 */
 		if (!watermark)
-			watermark = nr_pages << (PAGE_SHIFT - 1);
+			watermark = min_t(unsigned long,
+					  U32_MAX,
+					  (unsigned long)nr_pages << (PAGE_SHIFT - 1));
 
 		/*
 		 * Use aux_watermark as the basis for chunking to
author reviewer
#13 lore raw

PATCH 7/7 perf: Make rb_alloc_aux() return an error immediately if nr_pages <= 0

In reply to Adrian Hunter

rb_alloc_aux() should not be called with nr_pages <= 0. Make it more robust and readable by returning an error immediately in that case.

---
 kernel/events/ring_buffer.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/events/ring_buffer.c +3 -0
index 485cf0a66631..8cadf97bc290 100644
@@ -682,6 +682,9 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -EOPNOTSUPP;
 
+	if (nr_pages <= 0)
+		return -EINVAL;
+
 	if (!overwrite) {
 		/*
 		 * Watermark defaults to half the buffer, and so does the
Signature · 1 line
2.34.1
merge bot Applied
#14 lore raw

[tip: perf/core] perf: Make rb_alloc_aux() return an error immediately if nr_pages <= 0

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     0ca4da2412da05fb9dd0b5d90dcc8026219f0f29
Gitweb:        https://git.kernel.org/tip/0ca4da2412da05fb9dd0b5d90dcc8026219f0f29
Author:        Adrian Hunter <[email protected]>
AuthorDate:    Mon, 24 Jun 2024 23:11:01 +03:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 04 Jul 2024 16:00:23 +02:00

perf: Make rb_alloc_aux() return an error immediately if nr_pages <= 0

rb_alloc_aux() should not be called with nr_pages <= 0. Make it more robust and readable by returning an error immediately in that case.

---
 kernel/events/ring_buffer.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/events/ring_buffer.c +3 -0
index 485cf0a..8cadf97 100644
@@ -682,6 +682,9 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
 	if (!has_aux(event))
 		return -EOPNOTSUPP;
 
+	if (nr_pages <= 0)
+		return -EINVAL;
+
 	if (!overwrite) {
 		/*
 		 * Watermark defaults to half the buffer, and so does the