Skip to contentSkip to conversation

Add the page size in the perf record (kernel)

PATCH Superseded [email protected] 2 participants 10 messages Sep 17, 2020 1 message not in the archive

Subsystems (3)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#1 lore raw

PATCH v7 1/4 perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]

From: Kan Liang <[email protected]>

Current perf can report both virtual addresses and physical addresses, but not the MMU page size. Without the MMU page size information of the utilized page, users cannot decide whether to promote/demote large pages to optimize memory usage.

Add a new sample type for the data MMU page size.

Current perf already has a facility to collect data virtual addresses. A page walker is required to walk the pages tables and calculate the MMU page size from a given virtual address.

On some platforms, e.g., X86, the page walker is invoked in an NMI handler. So the page walker must be NMI-safe and low overhead. Besides, the page walker should work for both user and kernel virtual address. The existing generic page walker, e.g., walk_page_range_novma(), is a little bit complex and doesn't guarantee the NMI-safe. The follow_page() is only for user-virtual address.

Add a new function perf_get_page_size() to walk the page tables and calculate the MMU page size. In the function:
- Interrupts have to be disabled to prevent any teardown of the page tables.
- The MMU page size is calculated from the page table level. The method should work for all architectures, but it has only been verified on X86. Should there be some architectures, which support perf, where the method doesn't work, it can be fixed later separately. Reporting the wrong page size would not be fatal for the architecture.

---
 include/linux/perf_event.h      |   1 +
 include/uapi/linux/perf_event.h |   4 +-
 kernel/events/core.c            | 103 ++++++++++++++++++++++++++++++++
 3 files changed, 107 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index 0c19d279b97f..7e3785dd27d9 100644
@@ -1034,6 +1034,7 @@ struct perf_sample_data {
 
 	u64				phys_addr;
 	u64				cgroup;
+	u64				data_page_size;
 } ____cacheline_aligned;
 
 /* default value for data source */
include/uapi/linux/perf_event.h +3 -1
index 077e7ee69e3d..cc6ea346e9f9 100644
@@ -143,8 +143,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_PHYS_ADDR			= 1U << 19,
 	PERF_SAMPLE_AUX				= 1U << 20,
 	PERF_SAMPLE_CGROUP			= 1U << 21,
+	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
 
-	PERF_SAMPLE_MAX = 1U << 22,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 23,		/* non-ABI */
 
 	__PERF_SAMPLE_CALLCHAIN_EARLY		= 1ULL << 63, /* non-ABI; internal use */
 };
@@ -896,6 +897,7 @@ enum perf_event_type {
 	 *	{ u64			phys_addr;} && PERF_SAMPLE_PHYS_ADDR
 	 *	{ u64			size;
 	 *	  char			data[size]; } && PERF_SAMPLE_AUX
+	 *	{ u64			data_page_size;} && PERF_SAMPLE_DATA_PAGE_SIZE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +103 -0

133 lines not on this page · raw message

Signature · 1 line
2.17.1
#0 lore raw

v7 cover Add the page size in the perf record (kernel)

From: Kan Liang <[email protected]>

Changes since V6
- Return the MMU page size of a given virtual address, not the kernel software page size
- Add PERF_SAMPLE_DATA_PAGE_SIZE support for Power
- Allow large PEBS for PERF_SAMPLE_CODE_PAGE_SIZE
- Only include the kernel patches. The perf tool patches will be posted later separately once the kernel patches are accepted.

Changes since V5
- Introduce a new universal page walker for the page size in the perf subsystem.
- Rebased on Peter's tree.

Current perf can report both virtual addresses and physical addresses, but not the page size. Without the page size information of the utilized page, users cannot decide whether to promote/demote large pages to optimize memory usage.

The patch set was submitted a year ago.
https://lkml.kernel.org/r/[email protected]
It introduced a __weak function, perf_get_page_size(), aim to retrieve the page size via a given virtual address in the generic code, and implemented a x86 specific version of perf_get_page_size(). However, the proposal was rejected, because it's a pure x86 implementation.
https://lkml.kernel.org/r/[email protected]

At that time, it's not easy to support perf_get_page_size() universally, because some key functions, e.g., p?d_large, are not supported by some architectures.

Now, the generic p?d_leaf() functions are added in the latest kernel.
https://lkml.kernel.org/r/[email protected]
Starts from V6, a new universal perf_get_page_size() function is implemented based on the generic p?d_leaf() functions.

On some platforms, e.g., X86, the page walker is invoked in an NMI handler. So the page walker must be NMI-safe and low overhead. Besides, the page walker should work for both user and kernel virtual address. The existing generic page walker, e.g., walk_page_range_novma(), is a little bit complex and doesn't guarantee the NMI-safe. The follow_page() is only for the user-virtual address. So a simpler page walk function is implemented here.

Kan Liang (3):
perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE perf/x86/intel: Support PERF_SAMPLE_DATA_PAGE_SIZE powerpc/perf: Support PERF_SAMPLE_DATA_PAGE_SIZE

Stephane Eranian (1):
perf/core: Add support for PERF_SAMPLE_CODE_PAGE_SIZE

 arch/powerpc/perf/core-book3s.c |   6 +-
 arch/x86/events/intel/ds.c      |  11 ++-
 arch/x86/events/perf_event.h    |   2 +-
 include/linux/perf_event.h      |   2 +
 include/uapi/linux/perf_event.h |   6 +-
 kernel/events/core.c            | 114 +++++++++++++++++++++++++++++++-
 6 files changed, 133 insertions(+), 8 deletions(-)
Signature · 1 line
2.17.1
#8 lore raw

PATCH v7 2/4 perf/x86/intel: Support PERF_SAMPLE_DATA_PAGE_SIZE

From: Kan Liang <[email protected]>

The new sample type, PERF_SAMPLE_DATA_PAGE_SIZE, requires the virtual address. Update the data->addr if the sample type is set.

The large PEBS is disabled with the sample type, because perf doesn't support munmap tracking yet. The PEBS buffer for large PEBS cannot be flushed for each munmap. Wrong page size may be calculated. The large PEBS can be enabled later separately when munmap tracking is supported.

---
 arch/x86/events/intel/ds.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/ds.c +8 -3
index 404315df1e16..444e5f061d04 100644
@@ -959,7 +959,8 @@ static void adaptive_pebs_record_size_update(void)
 
 #define PERF_PEBS_MEMINFO_TYPE	(PERF_SAMPLE_ADDR | PERF_SAMPLE_DATA_SRC |   \
 				PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_WEIGHT | \
-				PERF_SAMPLE_TRANSACTION)
+				PERF_SAMPLE_TRANSACTION |		     \
+				PERF_SAMPLE_DATA_PAGE_SIZE)
 
 static u64 pebs_update_adaptive_cfg(struct perf_event *event)
 {
@@ -1335,6 +1336,10 @@ static u64 get_data_src(struct perf_event *event, u64 aux)
 	return val;
 }
 
+#define PERF_SAMPLE_ADDR_TYPE	(PERF_SAMPLE_ADDR |		\
+				 PERF_SAMPLE_PHYS_ADDR |	\
+				 PERF_SAMPLE_DATA_PAGE_SIZE)
+
 static void setup_pebs_fixed_sample_data(struct perf_event *event,
 				   struct pt_regs *iregs, void *__pebs,
 				   struct perf_sample_data *data,
@@ -1449,7 +1454,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
 	}
 
 
-	if ((sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR)) &&
+	if ((sample_type & PERF_SAMPLE_ADDR_TYPE) &&
 	    x86_pmu.intel_cap.pebs_format >= 1)
 		data->addr = pebs->dla;
 
@@ -1577,7 +1582,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
 		if (sample_type & PERF_SAMPLE_DATA_SRC)
 			data->data_src.val = get_data_src(event, meminfo->aux);
 
-		if (sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
+		if (sample_type & PERF_SAMPLE_ADDR_TYPE)
 			data->addr = meminfo->address;
 
 		if (sample_type & PERF_SAMPLE_TRANSACTION)
Signature · 1 line
2.17.1
#10 lore raw

PATCH v7 4/4 perf/core: Add support for PERF_SAMPLE_CODE_PAGE_SIZE

In reply to [email protected]

From: Stephane Eranian <[email protected]>

When studying code layout, it is useful to capture the page size of the sampled code address.

Add a new sample type for code page size.
The new sample type requires collecting the ip. The code page size can be calculated from the NMI-safe perf_get_page_size().

For large PEBS, it's very unlikely that the mapping is gone for the earlier PEBS records. Enable the feature for the large PEBS. The worst case is that page-size '0' is returned.

---
 arch/x86/events/perf_event.h    |  2 +-
 include/linux/perf_event.h      |  1 +
 include/uapi/linux/perf_event.h |  4 +++-
 kernel/events/core.c            | 11 ++++++++++-
 4 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/events/perf_event.h +1 -1
index 345442410a4d..10629ef1b626 100644
@@ -132,7 +132,7 @@ struct amd_nb {
 	PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | \
 	PERF_SAMPLE_TRANSACTION | PERF_SAMPLE_PHYS_ADDR | \
 	PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER | \
-	PERF_SAMPLE_PERIOD)
+	PERF_SAMPLE_PERIOD | PERF_SAMPLE_CODE_PAGE_SIZE)
 
 #define PEBS_GP_REGS			\
 	((1ULL << PERF_REG_X86_AX)    | \
include/linux/perf_event.h +1 -0
index 7e3785dd27d9..e533b03af053 100644
@@ -1035,6 +1035,7 @@ struct perf_sample_data {
 	u64				phys_addr;
 	u64				cgroup;
 	u64				data_page_size;
+	u64				code_page_size;
 } ____cacheline_aligned;
 
 /* default value for data source */
include/uapi/linux/perf_event.h +3 -1
index cc6ea346e9f9..c2f20ee3124d 100644
@@ -144,8 +144,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_AUX				= 1U << 20,
 	PERF_SAMPLE_CGROUP			= 1U << 21,
 	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
+	PERF_SAMPLE_CODE_PAGE_SIZE		= 1U << 23,
 
-	PERF_SAMPLE_MAX = 1U << 23,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 24,		/* non-ABI */
 
 	__PERF_SAMPLE_CALLCHAIN_EARLY		= 1ULL << 63, /* non-ABI; internal use */
 };
@@ -898,6 +899,7 @@ enum perf_event_type {
 	 *	{ u64			size;
 	 *	  char			data[size]; } && PERF_SAMPLE_AUX
 	 *	{ u64			data_page_size;} && PERF_SAMPLE_DATA_PAGE_SIZE
+	 *	{ u64			code_page_size;} && PERF_SAMPLE_CODE_PAGE_SIZE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +10 -1
index dc0ae692e32b..51452d5edfac 100644
@@ -1898,6 +1898,9 @@ static void __perf_event_header_size(struct perf_event *event, u64 sample_type)
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		size += sizeof(data->data_page_size);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		size += sizeof(data->code_page_size);
+
 	event->header_size = size;
 }
 
@@ -6944,6 +6947,9 @@ void perf_output_sample(struct perf_output_handle *handle,
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		perf_output_put(handle, data->data_page_size);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		perf_output_put(handle, data->code_page_size);
+
 	if (sample_type & PERF_SAMPLE_AUX) {
 		perf_output_put(handle, data->aux_size);
 
@@ -7124,7 +7130,7 @@ void perf_prepare_sample(struct perf_event_header *header,
 
 	__perf_event_header__init_id(header, data, event);
 
-	if (sample_type & PERF_SAMPLE_IP)
+	if (sample_type & (PERF_SAMPLE_IP | PERF_SAMPLE_CODE_PAGE_SIZE))
 		data->ip = perf_instruction_pointer(regs);
 
 	if (sample_type & PERF_SAMPLE_CALLCHAIN) {
@@ -7252,6 +7258,9 @@ void perf_prepare_sample(struct perf_event_header *header,
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		data->data_page_size = perf_get_page_size(data->addr);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		data->code_page_size = perf_get_page_size(data->ip);
+
 	if (sample_type & PERF_SAMPLE_AUX) {
 		u64 size;
 
Signature · 1 line
2.17.1
#9 lore raw

PATCH v7 3/4 powerpc/perf: Support PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]

From: Kan Liang <[email protected]>

The new sample type, PERF_SAMPLE_DATA_PAGE_SIZE, requires the virtual address. Update the data->addr if the sample type is set.

---
 arch/powerpc/perf/core-book3s.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/powerpc/perf/core-book3s.c +4 -2
index 78fe34986594..ce22bd23082d 100644
@@ -2065,6 +2065,9 @@ static struct pmu power_pmu = {
 	.sched_task	= power_pmu_sched_task,
 };
 
+#define PERF_SAMPLE_ADDR_TYPE  (PERF_SAMPLE_ADDR |		\
+				PERF_SAMPLE_PHYS_ADDR |		\
+				PERF_SAMPLE_DATA_PAGE_SIZE)
 /*
  * A counter has overflowed; update its count and record
  * things if requested.  Note that interrupts are hard-disabled
@@ -2120,8 +2123,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
 
 		perf_sample_data_init(&data, ~0ULL, event->hw.last_period);
 
-		if (event->attr.sample_type &
-		    (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
+		if (event->attr.sample_type & PERF_SAMPLE_ADDR_TYPE)
 			perf_get_data_addr(event, regs, &data.addr);
 
 		if (event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK) {
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH V7 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]

kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7083

Quoted 8 lines · On 9/17/20 6:52 AM, [email protected] wrote:
kernel/events/core.c +8 -0
+	mm = current->mm;
+	if (!mm) {
+		/*
+		 * For kernel threads and the like, use init_mm so that
+		 * we can find kernel memory.
+		 */
+		mm = &init_mm;
+	}

I think it might be better to use current->active_mm instead of current->mm. Kernel threads can "take over" the mm of the threads that switched to them, so they're not actually using all of the page tables from the init_mm all the time.

It's not _common_, thought. The only time that I think they can diverge is when vmalloc PGD sync'ing needs to be done, and there's even an effort to remove that.

But, it's probably more more precise to just use ->active_mm since that's what will actually be more consistent with the values loaded into CR3.

I _think_ ->active_mm is always non-NULL, too.

One last concern as I look at this: I wish it was a bit more future-proof. There are lots of weird things folks are trying to do with the page tables, like Address Space Isolation. For instance, if you get a perf NMI when running userspace, current->mm->pgd is *different* than the PGD that was in use when userspace was running. It's close enough today, but it might not stay that way. But I can't think of any great ways to future proof this code, other than spitting out an error message if too many of the page table walks fail when they shouldn't.

author
#3 lore raw

Re: [PATCH V7 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 19 lines · Dave Hansen

kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7083

Quoted 8 lines · On 9/17/20 6:52 AM, [email protected] wrote:
kernel/events/core.c +8 -0
+	mm = current->mm;
+	if (!mm) {
+		/*
+		 * For kernel threads and the like, use init_mm so that
+		 * we can find kernel memory.
+		 */
+		mm = &init_mm;
+	}

I think it might be better to use current->active_mm instead of current->mm. Kernel threads can "take over" the mm of the threads that switched to them, so they're not actually using all of the page tables from the init_mm all the time.

It's not _common_, thought. The only time that I think they can diverge is when vmalloc PGD sync'ing needs to be done, and there's even an effort to remove that.

Quoted 6 lines

But, it's probably more more precise to just use ->active_mm since that's what will actually be more consistent with the values loaded into CR3.

I _think_ ->active_mm is always non-NULL, too.

Thanks. yes, active_mm looks better here. I will use active_mm to replace the mm and &init_mm.

Quoted 10 lines

One last concern as I look at this: I wish it was a bit more future-proof. There are lots of weird things folks are trying to do with the page tables, like Address Space Isolation. For instance, if you get a perf NMI when running userspace, current->mm->pgd is *different* than the PGD that was in use when userspace was running. It's close enough today, but it might not stay that way. But I can't think of any great ways to future proof this code, other than spitting out an error message if too many of the page table walks fail when they shouldn't.

If the page table walks fail, page size 0 will return. So the worst case is that the page size is not available for users, which is not a fatal error.

If my understanding is correct, when the above case happens, there is nothing we can do for now (because we have no idea what it will become), except disabling the page size support and throw an error/warning.

From the user's perspective, throwing an error message or marking the page size unavailable should be the same. I think we may leave the code as-is. We can fix the future case later separately.

Thanks,
Kan

author
#5 lore raw

Re: [PATCH V7 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to a message not in the archive
Quoted 27 lines · Dave Hansen
Quoted 11 lines · Kan
Quoted 10 lines

One last concern as I look at this: I wish it was a bit more future-proof.  There are lots of weird things folks are trying to do with the page tables, like Address Space Isolation.  For instance, if you get a perf NMI when running userspace, current->mm->pgd is *different* than the PGD that was in use when userspace was running. It's close enough today, but it might not stay that way.  But I can't think of any great ways to future proof this code, other than spitting out an error message if too many of the page table walks fail when they shouldn't.

Quoted 3 lines

If the page table walks fail, page size 0 will return. So the worst case is that the page size is not available for users, which is not a fatal error.

Right, it's not a fatal error. It will just more or less silently break this feature.

Quoted 7 lines

If my understanding is correct, when the above case happens, there is nothing we can do for now (because we have no idea what it will become), except disabling the page size support and throw an error/warning.

From the user's perspective, throwing an error message or marking the
page size unavailable should be the same. I think we may leave the code as-is. We can fix the future case later separately.

Quoted 8 lines

The only thing I can think of is to record the number of consecutive page walk errors without a success. Occasional failures are OK and expected, such as if reclaim zeroes a PTE and a later perf event goes and looks at it. But a *LOT* of consecutive errors indicates a real problem somewhere.

Maybe if you have 10,000 or 1,000,000 successive walk failures, you do a WARN_ON_ONCE().

The user space perf tool looks like a better place for this kind of warning. The perf tool knows the total number of the samples. It also knows the number of the page size 0 samples. We can set a threshold, e.g., 90%. If 90% of the samples have the page size 0, perf tool will throw out a warning message.

The problem is that the warning from the perf tool usually includes some hints regarding the cause of the warning or possible solution to workaround/fix the warning. What message should we deliver to the users? "Warning: Too many error page size. Address space isolation feature may be enabled, please check"?

Thanks,
Kan

#6 lore raw

Re: [PATCH V7 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 5 lines · Kan

The user space perf tool looks like a better place for this kind of warning. The perf tool knows the total number of the samples. It also knows the number of the page size 0 samples. We can set a threshold, e.g., 90%. If 90% of the samples have the page size 0, perf tool will throw out a warning message.

Sounds good to me.

Quoted 5 lines

The problem is that the warning from the perf tool usually includes some hints regarding the cause of the warning or possible solution to workaround/fix the warning. What message should we deliver to the users? "Warning: Too many error page size. Address space isolation feature may be enabled, please check"?

That's not particularly actionable for an end user. Do any of the perf errors just tell them to submit a bug report? "Excessive number of page size lookup errors, please report to..."

author
#7 lore raw

Re: [PATCH V7 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 8 lines · Dave Hansen
Quoted 5 lines

The problem is that the warning from the perf tool usually includes some hints regarding the cause of the warning or possible solution to workaround/fix the warning. What message should we deliver to the users? "Warning: Too many error page size. Address space isolation feature may be enabled, please check"?

That's not particularly actionable for an end user. Do any of the perf errors just tell them to submit a bug report? "Excessive number of page size lookup errors, please report to..."

Yes, found some perf tool warnings end with "Consider reporting to [email protected]."

I will add a warning "Excessive number of page size lookup errors, please report to [email protected].", if 90% of the samples have the page size 0.

Thanks,
Kan