Skip to contentSkip to conversation

Add the page size in the perf record (kernel)

PATCH Applied [email protected] 7 participants 39 messages Oct 1, 2020 – Nov 12, 2020

Subsystems (6)
x86 PowerPC perf events MM core +2 more subsystems
lore
Share & tools
lore
#1 lore raw

PATCH v9 1/4 perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]

From: Kan Liang <[email protected]>

Current perf can report both virtual addresses and physical addresses, but not the MMU page size. Without the MMU page size information of the utilized page, users cannot decide whether to promote/demote large pages to optimize memory usage.

Add a new sample type for the data MMU page size.

Current perf already has a facility to collect data virtual addresses. A page walker is required to walk the pages tables and calculate the MMU page size from a given virtual address.

On some platforms, e.g., X86, the page walker is invoked in an NMI handler. So the page walker must be NMI-safe and low overhead. Besides, the page walker should work for both user and kernel virtual address. The existing generic page walker, e.g., walk_page_range_novma(), is a little bit complex and doesn't guarantee the NMI-safe. The follow_page() is only for user-virtual address.

Add a new function perf_get_page_size() to walk the page tables and calculate the MMU page size. In the function:
- Interrupts have to be disabled to prevent any teardown of the page tables.
- For user space threads, the current->mm is used for the page walker. For kernel threads and the like, the current->mm is NULL. The init_mm is used for the page walker. The active_mm is not used here, because it can be NULL.
Quote from Peter Zijlstra,
"context_switch() can set prev->active_mm to NULL when it transfers it
to @next. It does this before @current is updated. So an NMI that comes in between this active_mm swizzling and updating @current will see !active_mm."
- The MMU page size is calculated from the page table level.

The method should work for all architectures, but it has only been verified on X86. Should there be some architectures, which support perf, where the method doesn't work, it can be fixed later separately. Reporting the wrong page size would not be fatal for the architecture.

Some under discussion features may impact the method in the future. Quote from Dave Hansen,
"There are lots of weird things folks are trying to do with the page
tables, like Address Space Isolation. For instance, if you get a perf NMI when running userspace, current->mm->pgd is *different* than the PGD that was in use when userspace was running. It's close enough today, but it might not stay that way."
If the case happens later, lots of consecutive page walk errors will happen. The worst case is that lots of page-size '0' are returned, which would not be fatal.
In the perf tool, a check is implemented to detect this case. Once it happens, a kernel patch could be implemented accordingly then.

---
 include/linux/perf_event.h      |   1 +
 include/uapi/linux/perf_event.h |   4 +-
 kernel/events/core.c            | 103 ++++++++++++++++++++++++++++++++
 3 files changed, 107 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index 0c19d279b97f..7e3785dd27d9 100644
@@ -1034,6 +1034,7 @@ struct perf_sample_data {
 
 	u64				phys_addr;
 	u64				cgroup;
+	u64				data_page_size;
 } ____cacheline_aligned;
 
 /* default value for data source */
include/uapi/linux/perf_event.h +3 -1
index 077e7ee69e3d..cc6ea346e9f9 100644
@@ -143,8 +143,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_PHYS_ADDR			= 1U << 19,
 	PERF_SAMPLE_AUX				= 1U << 20,
 	PERF_SAMPLE_CGROUP			= 1U << 21,
+	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
 
-	PERF_SAMPLE_MAX = 1U << 22,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 23,		/* non-ABI */
 
 	__PERF_SAMPLE_CALLCHAIN_EARLY		= 1ULL << 63, /* non-ABI; internal use */
 };
@@ -896,6 +897,7 @@ enum perf_event_type {
 	 *	{ u64			phys_addr;} && PERF_SAMPLE_PHYS_ADDR
 	 *	{ u64			size;
 	 *	  char			data[size]; } && PERF_SAMPLE_AUX
+	 *	{ u64			data_page_size;} && PERF_SAMPLE_DATA_PAGE_SIZE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +103 -0

133 lines not on this page · raw message

Signature · 1 line
2.17.1
#33 lore raw

PATCH v9 2/4 perf/x86/intel: Support PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]

From: Kan Liang <[email protected]>

The new sample type, PERF_SAMPLE_DATA_PAGE_SIZE, requires the virtual address. Update the data->addr if the sample type is set.

The large PEBS is disabled with the sample type, because perf doesn't support munmap tracking yet. The PEBS buffer for large PEBS cannot be flushed for each munmap. Wrong page size may be calculated. The large PEBS can be enabled later separately when munmap tracking is supported.

---
 arch/x86/events/intel/ds.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/ds.c +8 -3
index 404315df1e16..444e5f061d04 100644
@@ -959,7 +959,8 @@ static void adaptive_pebs_record_size_update(void)
 
 #define PERF_PEBS_MEMINFO_TYPE	(PERF_SAMPLE_ADDR | PERF_SAMPLE_DATA_SRC |   \
 				PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_WEIGHT | \
-				PERF_SAMPLE_TRANSACTION)
+				PERF_SAMPLE_TRANSACTION |		     \
+				PERF_SAMPLE_DATA_PAGE_SIZE)
 
 static u64 pebs_update_adaptive_cfg(struct perf_event *event)
 {
@@ -1335,6 +1336,10 @@ static u64 get_data_src(struct perf_event *event, u64 aux)
 	return val;
 }
 
+#define PERF_SAMPLE_ADDR_TYPE	(PERF_SAMPLE_ADDR |		\
+				 PERF_SAMPLE_PHYS_ADDR |	\
+				 PERF_SAMPLE_DATA_PAGE_SIZE)
+
 static void setup_pebs_fixed_sample_data(struct perf_event *event,
 				   struct pt_regs *iregs, void *__pebs,
 				   struct perf_sample_data *data,
@@ -1449,7 +1454,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
 	}
 
 
-	if ((sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR)) &&
+	if ((sample_type & PERF_SAMPLE_ADDR_TYPE) &&
 	    x86_pmu.intel_cap.pebs_format >= 1)
 		data->addr = pebs->dla;
 
@@ -1577,7 +1582,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
 		if (sample_type & PERF_SAMPLE_DATA_SRC)
 			data->data_src.val = get_data_src(event, meminfo->aux);
 
-		if (sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
+		if (sample_type & PERF_SAMPLE_ADDR_TYPE)
 			data->addr = meminfo->address;
 
 		if (sample_type & PERF_SAMPLE_TRANSACTION)
Signature · 1 line
2.17.1
#35 lore raw

PATCH v9 3/4 powerpc/perf: Support PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]

From: Kan Liang <[email protected]>

The new sample type, PERF_SAMPLE_DATA_PAGE_SIZE, requires the virtual address. Update the data->addr if the sample type is set.

---
 arch/powerpc/perf/core-book3s.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/powerpc/perf/core-book3s.c +4 -2
index 78fe34986594..ce22bd23082d 100644
@@ -2065,6 +2065,9 @@ static struct pmu power_pmu = {
 	.sched_task	= power_pmu_sched_task,
 };
 
+#define PERF_SAMPLE_ADDR_TYPE  (PERF_SAMPLE_ADDR |		\
+				PERF_SAMPLE_PHYS_ADDR |		\
+				PERF_SAMPLE_DATA_PAGE_SIZE)
 /*
  * A counter has overflowed; update its count and record
  * things if requested.  Note that interrupts are hard-disabled
@@ -2120,8 +2123,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
 
 		perf_sample_data_init(&data, ~0ULL, event->hw.last_period);
 
-		if (event->attr.sample_type &
-		    (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
+		if (event->attr.sample_type & PERF_SAMPLE_ADDR_TYPE)
 			perf_get_data_addr(event, regs, &data.addr);
 
 		if (event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK) {
Signature · 1 line
2.17.1
#0 lore raw

v9 cover Add the page size in the perf record (kernel)

From: Kan Liang <[email protected]>

Changes since V8
- Drop active_mm which can cause kernel panic

Changes since V7
- Use active_mm to replace mm and init_mm
- Update the commit message of the patch 1

Changes since V6
- Return the MMU page size of a given virtual address, not the kernel software page size
- Add PERF_SAMPLE_DATA_PAGE_SIZE support for Power
- Allow large PEBS for PERF_SAMPLE_CODE_PAGE_SIZE
- Only include the kernel patches. The perf tool patches will be posted later separately once the kernel patches are accepted.

Changes since V5
- Introduce a new universal page walker for the page size in the perf subsystem.
- Rebased on Peter's tree.

Current perf can report both virtual addresses and physical addresses, but not the page size. Without the page size information of the utilized page, users cannot decide whether to promote/demote large pages to optimize memory usage.

The patch set was submitted a year ago.
https://lkml.kernel.org/r/[email protected]
It introduced a __weak function, perf_get_page_size(), aim to retrieve the page size via a given virtual address in the generic code, and implemented a x86 specific version of perf_get_page_size(). However, the proposal was rejected, because it's a pure x86 implementation.
https://lkml.kernel.org/r/[email protected]

At that time, it's not easy to support perf_get_page_size() universally, because some key functions, e.g., p?d_large, are not supported by some architectures.

Now, the generic p?d_leaf() functions are added in the latest kernel.
https://lkml.kernel.org/r/[email protected]
Starts from V6, a new universal perf_get_page_size() function is implemented based on the generic p?d_leaf() functions.

On some platforms, e.g., X86, the page walker is invoked in an NMI handler. So the page walker must be NMI-safe and low overhead. Besides, the page walker should work for both user and kernel virtual address. The existing generic page walker, e.g., walk_page_range_novma(), is a little bit complex and doesn't guarantee the NMI-safe. The follow_page() is only for the user-virtual address. So a simpler page walk function is implemented here.

Kan Liang (3):
perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE perf/x86/intel: Support PERF_SAMPLE_DATA_PAGE_SIZE powerpc/perf: Support PERF_SAMPLE_DATA_PAGE_SIZE

Stephane Eranian (1):
perf/core: Add support for PERF_SAMPLE_CODE_PAGE_SIZE

 arch/powerpc/perf/core-book3s.c |   6 +-
 arch/x86/events/intel/ds.c      |  11 ++-
 arch/x86/events/perf_event.h    |   2 +-
 include/linux/perf_event.h      |   2 +
 include/uapi/linux/perf_event.h |   6 +-
 kernel/events/core.c            | 114 +++++++++++++++++++++++++++++++-
 6 files changed, 133 insertions(+), 8 deletions(-)
Signature · 1 line
2.17.1
#37 lore raw

PATCH v9 4/4 perf/core: Add support for PERF_SAMPLE_CODE_PAGE_SIZE

From: Stephane Eranian <[email protected]>

When studying code layout, it is useful to capture the page size of the sampled code address.

Add a new sample type for code page size.
The new sample type requires collecting the ip. The code page size can be calculated from the NMI-safe perf_get_page_size().

For large PEBS, it's very unlikely that the mapping is gone for the earlier PEBS records. Enable the feature for the large PEBS. The worst case is that page-size '0' is returned.

---
 arch/x86/events/perf_event.h    |  2 +-
 include/linux/perf_event.h      |  1 +
 include/uapi/linux/perf_event.h |  4 +++-
 kernel/events/core.c            | 11 ++++++++++-
 4 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/events/perf_event.h +1 -1
index 345442410a4d..10629ef1b626 100644
@@ -132,7 +132,7 @@ struct amd_nb {
 	PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | \
 	PERF_SAMPLE_TRANSACTION | PERF_SAMPLE_PHYS_ADDR | \
 	PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER | \
-	PERF_SAMPLE_PERIOD)
+	PERF_SAMPLE_PERIOD | PERF_SAMPLE_CODE_PAGE_SIZE)
 
 #define PEBS_GP_REGS			\
 	((1ULL << PERF_REG_X86_AX)    | \
include/linux/perf_event.h +1 -0
index 7e3785dd27d9..e533b03af053 100644
@@ -1035,6 +1035,7 @@ struct perf_sample_data {
 	u64				phys_addr;
 	u64				cgroup;
 	u64				data_page_size;
+	u64				code_page_size;
 } ____cacheline_aligned;
 
 /* default value for data source */
include/uapi/linux/perf_event.h +3 -1
index cc6ea346e9f9..c2f20ee3124d 100644
@@ -144,8 +144,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_AUX				= 1U << 20,
 	PERF_SAMPLE_CGROUP			= 1U << 21,
 	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
+	PERF_SAMPLE_CODE_PAGE_SIZE		= 1U << 23,
 
-	PERF_SAMPLE_MAX = 1U << 23,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 24,		/* non-ABI */
 
 	__PERF_SAMPLE_CALLCHAIN_EARLY		= 1ULL << 63, /* non-ABI; internal use */
 };
@@ -898,6 +899,7 @@ enum perf_event_type {
 	 *	{ u64			size;
 	 *	  char			data[size]; } && PERF_SAMPLE_AUX
 	 *	{ u64			data_page_size;} && PERF_SAMPLE_DATA_PAGE_SIZE
+	 *	{ u64			code_page_size;} && PERF_SAMPLE_CODE_PAGE_SIZE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +10 -1
index dc0ae692e32b..51452d5edfac 100644
@@ -1898,6 +1898,9 @@ static void __perf_event_header_size(struct perf_event *event, u64 sample_type)
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		size += sizeof(data->data_page_size);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		size += sizeof(data->code_page_size);
+
 	event->header_size = size;
 }
 
@@ -6944,6 +6947,9 @@ void perf_output_sample(struct perf_output_handle *handle,
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		perf_output_put(handle, data->data_page_size);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		perf_output_put(handle, data->code_page_size);
+
 	if (sample_type & PERF_SAMPLE_AUX) {
 		perf_output_put(handle, data->aux_size);
 
@@ -7124,7 +7130,7 @@ void perf_prepare_sample(struct perf_event_header *header,
 
 	__perf_event_header__init_id(header, data, event);
 
-	if (sample_type & PERF_SAMPLE_IP)
+	if (sample_type & (PERF_SAMPLE_IP | PERF_SAMPLE_CODE_PAGE_SIZE))
 		data->ip = perf_instruction_pointer(regs);
 
 	if (sample_type & PERF_SAMPLE_CALLCHAIN) {
@@ -7252,6 +7258,9 @@ void perf_prepare_sample(struct perf_event_header *header,
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		data->data_page_size = perf_get_page_size(data->addr);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		data->code_page_size = perf_get_page_size(data->ip);
+
 	if (sample_type & PERF_SAMPLE_AUX) {
 		u64 size;
 
Signature · 1 line
2.17.1
maintainer
#2 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]
Quoted 35 lines · On Thu, Oct 01, 2020 at 06:57:46AM -0700, [email protected] wrote:
(quoted diff) +35 -0
+/*
+ * Return the MMU page size of a given virtual address
+ */
+static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+{
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+	pmd_t *pmd;
+	pte_t *pte;
+
+	pgd = pgd_offset(mm, addr);
+	if (pgd_none(*pgd))
+		return 0;
+
+	p4d = p4d_offset(pgd, addr);
+	if (!p4d_present(*p4d))
+		return 0;
+
+	if (p4d_leaf(*p4d))
+		return 1ULL << P4D_SHIFT;
+
+	pud = pud_offset(p4d, addr);
+	if (!pud_present(*pud))
+		return 0;
+
+	if (pud_leaf(*pud))
+		return 1ULL << PUD_SHIFT;
+
+	pmd = pmd_offset(pud, addr);
+	if (!pmd_present(*pmd))
+		return 0;
+
+	if (pmd_leaf(*pmd))
+		return 1ULL << PMD_SHIFT;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	pte = pte_offset_map(pmd, addr);
+	if (!pte_present(*pte)) {
+		pte_unmap(pte);
+		return 0;
+	}
+
+	pte_unmap(pte);
+	return PAGE_SIZE;
+}

So this mostly works, but gets a number of hugetlb and arch specific things wrong.

With the first 3 patches, this is only exposed to x86 and Power. Michael, does the above work for you?

Looking at:

arch/powerpc/include/asm/book3s/64/hugetlb.h:check_and_get_huge_psize()

You seem to limit yourself to page-table sizes, however if I then look at the same function in:

arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h
arch/powerpc/include/asm/nohash/hugetlb-book3e.h

it doesn't seem to constrain itself so.

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

Will, can you live with the below, if not, what would you like to do, make the entire function __weak so that you can override it, or hook into it somewhere?

DaveM, I saw Sparc64 also has 'funny' hugetlb sizes. Are those only for hugetlb, or are you also using them for the kernel map?

(we might not need the #ifdef gunk, but I've not yet dug out my cross
compilers this morning)

---

kernel/events/core.c +19 -2
@@ -7009,6 +7009,7 @@ static u64 perf_virt_to_phys(u64 virt)
  */
 static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
 {
+	struct page *page;
 	pgd_t *pgd;
 	p4d_t *p4d;
 	pud_t *pud;
@@ -7030,15 +7031,27 @@ static u64 __perf_get_page_size(struct m
 	if (!pud_present(*pud))
 		return 0;
 
-	if (pud_leaf(*pud))
+	if (pud_leaf(*pud)) {
+#ifdef pud_page
+		page = pud_page(*pud);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
 		return 1ULL << PUD_SHIFT;
+	}
 
 	pmd = pmd_offset(pud, addr);
 	if (!pmd_present(*pmd))
 		return 0;
 
-	if (pmd_leaf(*pmd))
+	if (pmd_leaf(*pmd)) {
+#ifdef pmd_page
+		page = pmd_page(*pmd);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
 		return 1ULL << PMD_SHIFT;
+	}
 
 	pte = pte_offset_map(pmd, addr);
 	if (!pte_present(*pte)) {
@@ -7046,6 +7059,10 @@ static u64 __perf_get_page_size(struct m
 		return 0;
 	}
 
+	page = pte_page(*pte);
+	if (PageHuge(page))
+		return page_size(compound_head(page));
+
 	pte_unmap(pte);
 	return PAGE_SIZE;
 }
maintainer
#3 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7064

Quoted 7 lines · Peter Zijlstra
kernel/events/core.c +3 -0
@@ -7046,6 +7059,10 @@ static u64 __perf_get_page_size(struct m
 		return 0;
 	}
 
+	page = pte_page(*pte);
+	if (PageHuge(page))
+		return page_size(compound_head(page));

Argh, this misses pte_unmap()..

Quoted 4 lines
kernel/events/core.c +1 -0
+
 	pte_unmap(pte);
 	return PAGE_SIZE;
 }
maintainer
#4 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Peter Zijlstra
Quoted 67 lines · Peter Zijlstra
Quoted 35 lines · On Thu, Oct 01, 2020 at 06:57:46AM -0700, [email protected] wrote:
(quoted diff) +35 -0
+/*
+ * Return the MMU page size of a given virtual address
+ */
+static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+{
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+	pmd_t *pmd;
+	pte_t *pte;
+
+	pgd = pgd_offset(mm, addr);
+	if (pgd_none(*pgd))
+		return 0;
+
+	p4d = p4d_offset(pgd, addr);
+	if (!p4d_present(*p4d))
+		return 0;
+
+	if (p4d_leaf(*p4d))
+		return 1ULL << P4D_SHIFT;
+
+	pud = pud_offset(p4d, addr);
+	if (!pud_present(*pud))
+		return 0;
+
+	if (pud_leaf(*pud))
+		return 1ULL << PUD_SHIFT;
+
+	pmd = pmd_offset(pud, addr);
+	if (!pmd_present(*pmd))
+		return 0;
+
+	if (pmd_leaf(*pmd))
+		return 1ULL << PMD_SHIFT;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	pte = pte_offset_map(pmd, addr);
+	if (!pte_present(*pte)) {
+		pte_unmap(pte);
+		return 0;
+	}
+
+	pte_unmap(pte);
+	return PAGE_SIZE;
+}

So this mostly works, but gets a number of hugetlb and arch specific things wrong.

With the first 3 patches, this is only exposed to x86 and Power. Michael, does the above work for you?

Looking at:

arch/powerpc/include/asm/book3s/64/hugetlb.h:check_and_get_huge_psize()

You seem to limit yourself to page-table sizes, however if I then look at the same function in:

arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h
arch/powerpc/include/asm/nohash/hugetlb-book3e.h

it doesn't seem to constrain itself so.

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Quoted 8 lines

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

Will, can you live with the below, if not, what would you like to do, make the entire function __weak so that you can override it, or hook into it somewhere?

Hmm, so I don't think we currently have any PMUs that set 'data->addr' on arm64, in which case maybe none of this currently matters for us.

However, I must admit that I couldn't figure out exactly what gets exposed to userspace when the backend drivers don't look at the sample_type or do anything with the addr field.

Will

maintainer
#5 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 13 lines · Will Deacon
Quoted 10 lines · Peter Zijlstra

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

Will, can you live with the below, if not, what would you like to do, make the entire function __weak so that you can override it, or hook into it somewhere?

Quoted 6 lines

Hmm, so I don't think we currently have any PMUs that set 'data->addr' on arm64, in which case maybe none of this currently matters for us.

However, I must admit that I couldn't figure out exactly what gets exposed to userspace when the backend drivers don't look at the sample_type or do anything with the addr field.

Patch 4:

https://lkml.kernel.org/r/[email protected]

is the one that exposes this to everybody with perf support. It will then report the page-size for the code address (SAMPLE_IP).

author
#9 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra

(we might not need the #ifdef gunk, but I've not yet dug out my cross
compilers this morning)

---

kernel/events/core.c +1 -0
@@ -7009,6 +7009,7 @@ static u64 perf_virt_to_phys(u64 virt)
   */
  static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
  {
+	struct page *page;
  	pgd_t *pgd;
  	p4d_t *p4d;
  	pud_t *pud;

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7038

Quoted 9 lines
kernel/events/core.c +5 -1
@@ -7030,15 +7031,27 @@ static u64 __perf_get_page_size(struct m
  	if (!pud_present(*pud))
  		return 0;
 
-	if (pud_leaf(*pud))
+	if (pud_leaf(*pud)) {
+#ifdef pud_page
+		page = pud_page(*pud);
+		if (PageHuge(page))
+			return page_size(compound_head(page));

I think the page_size() returns the Kernel Page Size of a compound page. What we want is the MMU page size.

If it's for the generic code, I think it should be a problem for X86.

Thanks,
Kan

Quoted 31 lines
kernel/events/core.c +13 -1
+#endif
  		return 1ULL << PUD_SHIFT;
+	}
 
  	pmd = pmd_offset(pud, addr);
  	if (!pmd_present(*pmd))
  		return 0;
 
-	if (pmd_leaf(*pmd))
+	if (pmd_leaf(*pmd)) {
+#ifdef pmd_page
+		page = pmd_page(*pmd);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
  		return 1ULL << PMD_SHIFT;
+	}
 
  	pte = pte_offset_map(pmd, addr);
  	if (!pte_present(*pte)) {
@@ -7046,6 +7059,10 @@ static u64 __perf_get_page_size(struct m
  		return 0;
  	}
 
+	page = pte_page(*pte);
+	if (PageHuge(page))
+		return page_size(compound_head(page));
+
  	pte_unmap(pte);
  	return PAGE_SIZE;
  }
maintainer
#10 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 30 lines · Kan
Quoted 15 lines · Peter Zijlstra

(we might not need the #ifdef gunk, but I've not yet dug out my cross
compilers this morning)

---

kernel/events/core.c +1 -0
@@ -7009,6 +7009,7 @@ static u64 perf_virt_to_phys(u64 virt)
   */
  static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
  {
+	struct page *page;
  	pgd_t *pgd;
  	p4d_t *p4d;
  	pud_t *pud;

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7037

Quoted 8 lines
kernel/events/core.c +5 -1
@@ -7030,15 +7031,27 @@ static u64 __perf_get_page_size(struct m
  	if (!pud_present(*pud))
  		return 0;
-	if (pud_leaf(*pud))
+	if (pud_leaf(*pud)) {
+#ifdef pud_page
+		page = pud_page(*pud);
+		if (PageHuge(page))
+			return page_size(compound_head(page));

I think the page_size() returns the Kernel Page Size of a compound page. What we want is the MMU page size.

Quoted 1 line

If it's for the generic code, I think it should be a problem for X86.

See the PageHuge() condition before it. It only makes sense to provide a hugetlb page-size if the actual hardware supports it.

For x86 hugetlb only supports PMD and PUD sized pages, so the added code is pointless and should result in identical behaviour.

For architectures that have hugetlb page sizes that do not align with the page-table levels (arm64, sparc64 and possibly power) this will (hopefully) give the right number.

#11 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Peter Zijlstra
Quoted 50 lines · Peter Zijlstra
Quoted 35 lines · On Thu, Oct 01, 2020 at 06:57:46AM -0700, [email protected] wrote:
(quoted diff) +35 -0
+/*
+ * Return the MMU page size of a given virtual address
+ */
+static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+{
+	pgd_t *pgd;
+	p4d_t *p4d;
+	pud_t *pud;
+	pmd_t *pmd;
+	pte_t *pte;
+
+	pgd = pgd_offset(mm, addr);
+	if (pgd_none(*pgd))
+		return 0;
+
+	p4d = p4d_offset(pgd, addr);
+	if (!p4d_present(*p4d))
+		return 0;
+
+	if (p4d_leaf(*p4d))
+		return 1ULL << P4D_SHIFT;
+
+	pud = pud_offset(p4d, addr);
+	if (!pud_present(*pud))
+		return 0;
+
+	if (pud_leaf(*pud))
+		return 1ULL << PUD_SHIFT;
+
+	pmd = pmd_offset(pud, addr);
+	if (!pmd_present(*pmd))
+		return 0;
+
+	if (pmd_leaf(*pmd))
+		return 1ULL << PMD_SHIFT;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	pte = pte_offset_map(pmd, addr);
+	if (!pte_present(*pte)) {
+		pte_unmap(pte);
+		return 0;
+	}
+
+	pte_unmap(pte);
+	return PAGE_SIZE;
+}

So this mostly works, but gets a number of hugetlb and arch specific things wrong.

Quoted 2 lines

With the first 3 patches, this is only exposed to x86 and Power. Michael, does the above work for you?

It might work on our server CPUs (book3s) but in general no I don't think it will work for all powerpc.

Quoted 11 lines

Looking at:

arch/powerpc/include/asm/book3s/64/hugetlb.h:check_and_get_huge_psize()

You seem to limit yourself to page-table sizes, however if I then look at the same function in:

arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h
arch/powerpc/include/asm/nohash/hugetlb-book3e.h

it doesn't seem to constrain itself so.

Yeah the embedded CPUs have more weird sizes.

I think we have all the logic in our __find_linux_pte().

Quoted 6 lines

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

I think we probably need it to be weak so we can provide our own version.

cheers

maintainer
#12 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 10 lines · Michael Ellerman
Quoted 6 lines · Peter Zijlstra

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

I think we probably need it to be weak so we can provide our own version.

I guess the same thing applies to us, but I can't really tell how accurate this stuff needs to be for userspace. If it's trying to use the page-table configuration to infer properties of the TLB, that's never going to be reliable because the TLB can choose both to split and coalesce entries as long as software can't tell.

Will

author
#13 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 9 lines · Will Deacon
Quoted 8 lines · Michael Ellerman
Quoted 6 lines · Peter Zijlstra

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

Quoted 9 lines
Quoted 2 lines

I think we probably need it to be weak so we can provide our own version.

I guess the same thing applies to us, but I can't really tell how accurate this stuff needs to be for userspace. If it's trying to use the page-table configuration to infer properties of the TLB, that's never going to be reliable because the TLB can choose both to split and coalesce entries as long as software can't tell.

Hi Peter,

It looks like everybody wants a __weak function. If so, I guess we should drop the generic code in this patch. For X86, we have existing functions to retrieve the page level and the page size. I think we don't need the generic code either.
https://lkml.kernel.org/r/[email protected]/

Should I send a V10 patch to drop the generic code and implement an X86 specific perf_get_page_size()? Will, Michael, and others can implement their version later separately.

Thanks,
Kan

maintainer
#14 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 30 lines · Kan
Quoted 9 lines · Will Deacon
Quoted 8 lines · Michael Ellerman
Quoted 6 lines · Peter Zijlstra

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

Quoted 9 lines
Quoted 2 lines

I think we probably need it to be weak so we can provide our own version.

I guess the same thing applies to us, but I can't really tell how accurate this stuff needs to be for userspace. If it's trying to use the page-table configuration to infer properties of the TLB, that's never going to be reliable because the TLB can choose both to split and coalesce entries as long as software can't tell.

Hi Peter,

It looks like everybody wants a __weak function. If so, I guess we should drop the generic code in this patch. For X86, we have existing functions to retrieve the page level and the page size. I think we don't need the generic code either.
https://lkml.kernel.org/r/[email protected]/

Quoted 3 lines

Should I send a V10 patch to drop the generic code and implement an X86 specific perf_get_page_size()? Will, Michael, and others can implement their version later separately.

Nah, that generic function, should work for 90% of all archs, it's just a few oddballs that need something else.

Also, if we add that hugetlb exception, we'll even get the usermap for those oddballs right.

I'll take this version after the merge window, I'll add __weak for the oddballs and also add the hugetlb userspace thing on top.

maintainer
#15 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 8 lines · Peter Zijlstra

Nah, that generic function, should work for 90% of all archs, it's just a few oddballs that need something else.

Also, if we add that hugetlb exception, we'll even get the usermap for those oddballs right.

I'll take this version after the merge window, I'll add __weak for the oddballs and also add the hugetlb userspace thing on top.

Like so..

---
Subject: perf,mm: Handle non-page-table-aligned hugetlbfs
From: Peter Zijlstra <[email protected]>
Date: Fri, 9 Oct 2020 11:09:27 +0200

A limited nunmber of architectures support hugetlbfs sizes that do not align with the page-tables (ARM64, Power, Sparc64). Add support for this to the generic perf_get_page_size() implementation, and also allow an architecture to override this implementation.

This latter is only needed when it uses non-page-table aligned huge pages in its kernel map.

---
 kernel/events/core.c |   39 +++++++++++++++++++++++++++++++++------
 1 file changed, 33 insertions(+), 6 deletions(-)
kernel/events/core.c +33 -6
@@ -6996,10 +6996,18 @@ static u64 perf_virt_to_phys(u64 virt)
 #ifdef CONFIG_MMU
 
 /*
- * Return the MMU page size of a given virtual address
+ * Return the MMU page size of a given virtual address.
+ *
+ * This generic implementation handles page-table aligned huge pages, as well
+ * as non-page-table aligned hugetlbfs compound pages.
+ *
+ * If an architecture supports and uses non-page-table aligned pages in their
+ * kernel mapping it will need to provide it's own implementation of this
+ * function.
  */
-static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+__weak u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)
 {
+	struct page *page;
 	pgd_t *pgd;
 	p4d_t *p4d;
 	pud_t *pud;
@@ -7021,15 +7029,27 @@ static u64 __perf_get_page_size(struct m
 	if (!pud_present(*pud))
 		return 0;
 
-	if (pud_leaf(*pud))
+	if (pud_leaf(*pud)) {
+#ifdef pud_page
+		page = pud_page(*pud);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
 		return 1ULL << PUD_SHIFT;
+	}
 
 	pmd = pmd_offset(pud, addr);
 	if (!pmd_present(*pmd))
 		return 0;
 
-	if (pmd_leaf(*pmd))
+	if (pmd_leaf(*pmd)) {
+#ifdef pmd_page
+		page = pmd_page(*pmd);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
 		return 1ULL << PMD_SHIFT;
+	}
 
 	pte = pte_offset_map(pmd, addr);
 	if (!pte_present(*pte)) {
@@ -7037,13 +7057,20 @@ static u64 __perf_get_page_size(struct m
 		return 0;
 	}
 
+	page = pte_page(*pte);
+	if (PageHuge(page)) {
+		u64 size = page_size(compound_head(page));
+		pte_unmap(pte);
+		return size;
+	}
+
 	pte_unmap(pte);
 	return PAGE_SIZE;
 }
 
 #else
 
-static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+static u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)
 {
 	return 0;
 }
@@ -7074,7 +7101,7 @@ static u64 perf_get_page_size(unsigned l
 		mm = &init_mm;
 	}
 
-	size = __perf_get_page_size(mm, addr);
+	size = arch_perf_get_page_size(mm, addr);
 
 	local_irq_restore(flags);
 
#6 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 13 lines · Will Deacon
Quoted 10 lines · Peter Zijlstra

Patch 4 makes it all far worse by exposing it to pretty much everybody.

Now, I think we can fix at least the user mappings with the below delta, but if archs are using non-page-table MMU sizes we'll need arch helpers.

ARM64 is in that last boat.

Will, can you live with the below, if not, what would you like to do, make the entire function __weak so that you can override it, or hook into it somewhere?

Quoted 6 lines

Hmm, so I don't think we currently have any PMUs that set 'data->addr' on arm64, in which case maybe none of this currently matters for us.

However, I must admit that I couldn't figure out exactly what gets exposed to userspace when the backend drivers don't look at the sample_type or do anything with the addr field.

Quoted 6 lines

Patch 4:

https://lkml.kernel.org/r/[email protected]

is the one that exposes this to everybody with perf support. It will then report the page-size for the code address (SAMPLE_IP).

I can see there have another potentail customer to use page-size is Arm SPE, but Arm SPE is hardware trace based sample but not interrupt based sample. For this case, I think this patch set cannot be directly applied to the AUX trace data.

Thanks,
Leo

maintainer
#7 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 4 lines · Leo Yan

I can see there have another potentail customer to use page-size is Arm SPE, but Arm SPE is hardware trace based sample but not interrupt based sample. For this case, I think this patch set cannot be directly applied to the AUX trace data.

IIRC SPE is decoded in userspace, at which point we simply cannot access this data.

#8 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Leo Yan

I can see there have another potentail customer to use page-size is Arm SPE, but Arm SPE is hardware trace based sample but not interrupt based sample. For this case, I think this patch set cannot be directly applied to the AUX trace data.

IIRC SPE is decoded in userspace, at which point we simply cannot access this data.

Yes, it's decoded in userspace.

Thanks for clarification!
Leo

#38 lore raw

[tip: perf/core] perf/core: Add support for PERF_SAMPLE_CODE_PAGE_SIZE

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     995f088efebe1eba0282a6ffa12411b37f8990c2
Gitweb:        https://git.kernel.org/tip/995f088efebe1eba0282a6ffa12411b37f8990c2
Author:        Stephane Eranian <[email protected]>
AuthorDate:    Thu, 01 Oct 2020 06:57:49 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 29 Oct 2020 11:00:39 +01:00

perf/core: Add support for PERF_SAMPLE_CODE_PAGE_SIZE

When studying code layout, it is useful to capture the page size of the sampled code address.

Add a new sample type for code page size.
The new sample type requires collecting the ip. The code page size can be calculated from the NMI-safe perf_get_page_size().

For large PEBS, it's very unlikely that the mapping is gone for the earlier PEBS records. Enable the feature for the large PEBS. The worst case is that page-size '0' is returned.

---
 arch/x86/events/perf_event.h    |  2 +-
 include/linux/perf_event.h      |  1 +
 include/uapi/linux/perf_event.h |  4 +++-
 kernel/events/core.c            | 11 ++++++++++-
 4 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/events/perf_event.h +1 -1
index ee2b9b9..10032f0 100644
@@ -132,7 +132,7 @@ struct amd_nb {
 	PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | \
 	PERF_SAMPLE_TRANSACTION | PERF_SAMPLE_PHYS_ADDR | \
 	PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER | \
-	PERF_SAMPLE_PERIOD)
+	PERF_SAMPLE_PERIOD | PERF_SAMPLE_CODE_PAGE_SIZE)
 
 #define PEBS_GP_REGS			\
 	((1ULL << PERF_REG_X86_AX)    | \
include/linux/perf_event.h +1 -0
index 7e3785d..e533b03 100644
@@ -1035,6 +1035,7 @@ struct perf_sample_data {
 	u64				phys_addr;
 	u64				cgroup;
 	u64				data_page_size;
+	u64				code_page_size;
 } ____cacheline_aligned;
 
 /* default value for data source */
include/uapi/linux/perf_event.h +3 -1
index cc6ea34..c2f20ee 100644
@@ -144,8 +144,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_AUX				= 1U << 20,
 	PERF_SAMPLE_CGROUP			= 1U << 21,
 	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
+	PERF_SAMPLE_CODE_PAGE_SIZE		= 1U << 23,
 
-	PERF_SAMPLE_MAX = 1U << 23,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 24,		/* non-ABI */
 
 	__PERF_SAMPLE_CALLCHAIN_EARLY		= 1ULL << 63, /* non-ABI; internal use */
 };
@@ -898,6 +899,7 @@ enum perf_event_type {
 	 *	{ u64			size;
 	 *	  char			data[size]; } && PERF_SAMPLE_AUX
 	 *	{ u64			data_page_size;} && PERF_SAMPLE_DATA_PAGE_SIZE
+	 *	{ u64			code_page_size;} && PERF_SAMPLE_CODE_PAGE_SIZE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +10 -1
index a796db2..7f655d1 100644
@@ -1898,6 +1898,9 @@ static void __perf_event_header_size(struct perf_event *event, u64 sample_type)
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		size += sizeof(data->data_page_size);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		size += sizeof(data->code_page_size);
+
 	event->header_size = size;
 }
 
@@ -6945,6 +6948,9 @@ void perf_output_sample(struct perf_output_handle *handle,
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		perf_output_put(handle, data->data_page_size);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		perf_output_put(handle, data->code_page_size);
+
 	if (sample_type & PERF_SAMPLE_AUX) {
 		perf_output_put(handle, data->aux_size);
 
@@ -7125,7 +7131,7 @@ void perf_prepare_sample(struct perf_event_header *header,
 
 	__perf_event_header__init_id(header, data, event);
 
-	if (sample_type & PERF_SAMPLE_IP)
+	if (sample_type & (PERF_SAMPLE_IP | PERF_SAMPLE_CODE_PAGE_SIZE))
 		data->ip = perf_instruction_pointer(regs);
 
 	if (sample_type & PERF_SAMPLE_CALLCHAIN) {
@@ -7253,6 +7259,9 @@ void perf_prepare_sample(struct perf_event_header *header,
 	if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE)
 		data->data_page_size = perf_get_page_size(data->addr);
 
+	if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE)
+		data->code_page_size = perf_get_page_size(data->ip);
+
 	if (sample_type & PERF_SAMPLE_AUX) {
 		u64 size;
 
merge bot Applied
#36 lore raw

[tip: perf/core] powerpc/perf: Support PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     4cb6a42e4c4bc1902644eced67563e7405d4588e
Gitweb:        https://git.kernel.org/tip/4cb6a42e4c4bc1902644eced67563e7405d4588e
Author:        Kan Liang <[email protected]>
AuthorDate:    Thu, 01 Oct 2020 06:57:48 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 29 Oct 2020 11:00:39 +01:00

powerpc/perf: Support PERF_SAMPLE_DATA_PAGE_SIZE

The new sample type, PERF_SAMPLE_DATA_PAGE_SIZE, requires the virtual address. Update the data->addr if the sample type is set.

---
 arch/powerpc/perf/core-book3s.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/powerpc/perf/core-book3s.c +4 -2
index 78fe349..ce22bd2 100644
@@ -2065,6 +2065,9 @@ static struct pmu power_pmu = {
 	.sched_task	= power_pmu_sched_task,
 };
 
+#define PERF_SAMPLE_ADDR_TYPE  (PERF_SAMPLE_ADDR |		\
+				PERF_SAMPLE_PHYS_ADDR |		\
+				PERF_SAMPLE_DATA_PAGE_SIZE)
 /*
  * A counter has overflowed; update its count and record
  * things if requested.  Note that interrupts are hard-disabled
@@ -2120,8 +2123,7 @@ static void record_and_restart(struct perf_event *event, unsigned long val,
 
 		perf_sample_data_init(&data, ~0ULL, event->hw.last_period);
 
-		if (event->attr.sample_type &
-		    (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
+		if (event->attr.sample_type & PERF_SAMPLE_ADDR_TYPE)
 			perf_get_data_addr(event, regs, &data.addr);
 
 		if (event->attr.sample_type & PERF_SAMPLE_BRANCH_STACK) {
merge bot Applied
#32 lore raw

[tip: perf/core] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     8d97e71811aaafe4abf611dc24822fd6e73df1a1
Gitweb:        https://git.kernel.org/tip/8d97e71811aaafe4abf611dc24822fd6e73df1a1
Author:        Kan Liang <[email protected]>
AuthorDate:    Thu, 01 Oct 2020 06:57:46 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 29 Oct 2020 11:00:38 +01:00

perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Current perf can report both virtual addresses and physical addresses, but not the MMU page size. Without the MMU page size information of the utilized page, users cannot decide whether to promote/demote large pages to optimize memory usage.

Add a new sample type for the data MMU page size.

Current perf already has a facility to collect data virtual addresses. A page walker is required to walk the pages tables and calculate the MMU page size from a given virtual address.

On some platforms, e.g., X86, the page walker is invoked in an NMI handler. So the page walker must be NMI-safe and low overhead. Besides, the page walker should work for both user and kernel virtual address. The existing generic page walker, e.g., walk_page_range_novma(), is a little bit complex and doesn't guarantee the NMI-safe. The follow_page() is only for user-virtual address.

Add a new function perf_get_page_size() to walk the page tables and calculate the MMU page size. In the function:
- Interrupts have to be disabled to prevent any teardown of the page tables.
- For user space threads, the current->mm is used for the page walker. For kernel threads and the like, the current->mm is NULL. The init_mm is used for the page walker. The active_mm is not used here, because it can be NULL.
Quote from Peter Zijlstra,
"context_switch() can set prev->active_mm to NULL when it transfers it
to @next. It does this before @current is updated. So an NMI that comes in between this active_mm swizzling and updating @current will see !active_mm."
- The MMU page size is calculated from the page table level.

The method should work for all architectures, but it has only been verified on X86. Should there be some architectures, which support perf, where the method doesn't work, it can be fixed later separately. Reporting the wrong page size would not be fatal for the architecture.

Some under discussion features may impact the method in the future. Quote from Dave Hansen,
"There are lots of weird things folks are trying to do with the page
tables, like Address Space Isolation. For instance, if you get a perf NMI when running userspace, current->mm->pgd is *different* than the PGD that was in use when userspace was running. It's close enough today, but it might not stay that way."
If the case happens later, lots of consecutive page walk errors will happen. The worst case is that lots of page-size '0' are returned, which would not be fatal.
In the perf tool, a check is implemented to detect this case. Once it happens, a kernel patch could be implemented accordingly then.

---
 include/linux/perf_event.h      |   1 +-
 include/uapi/linux/perf_event.h |   4 +-
 kernel/events/core.c            | 103 +++++++++++++++++++++++++++++++-
 3 files changed, 107 insertions(+), 1 deletion(-)
include/linux/perf_event.h +1 -0
index 0c19d27..7e3785d 100644
@@ -1034,6 +1034,7 @@ struct perf_sample_data {
 
 	u64				phys_addr;
 	u64				cgroup;
+	u64				data_page_size;
 } ____cacheline_aligned;
 
 /* default value for data source */
include/uapi/linux/perf_event.h +3 -1
index 077e7ee..cc6ea34 100644
@@ -143,8 +143,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_PHYS_ADDR			= 1U << 19,
 	PERF_SAMPLE_AUX				= 1U << 20,
 	PERF_SAMPLE_CGROUP			= 1U << 21,
+	PERF_SAMPLE_DATA_PAGE_SIZE		= 1U << 22,
 
-	PERF_SAMPLE_MAX = 1U << 22,		/* non-ABI */
+	PERF_SAMPLE_MAX = 1U << 23,		/* non-ABI */
 
 	__PERF_SAMPLE_CALLCHAIN_EARLY		= 1ULL << 63, /* non-ABI; internal use */
 };
@@ -896,6 +897,7 @@ enum perf_event_type {
 	 *	{ u64			phys_addr;} && PERF_SAMPLE_PHYS_ADDR
 	 *	{ u64			size;
 	 *	  char			data[size]; } && PERF_SAMPLE_AUX
+	 *	{ u64			data_page_size;} && PERF_SAMPLE_DATA_PAGE_SIZE
 	 * };
 	 */
 	PERF_RECORD_SAMPLE			= 9,
kernel/events/core.c +103 -0

133 lines not on this page · raw message

merge bot Applied
#34 lore raw

[tip: perf/core] perf/x86/intel: Support PERF_SAMPLE_DATA_PAGE_SIZE

In reply to [email protected]
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     76a5433f95f32d8a17c9f836be2084ed947c466b
Gitweb:        https://git.kernel.org/tip/76a5433f95f32d8a17c9f836be2084ed947c466b
Author:        Kan Liang <[email protected]>
AuthorDate:    Thu, 01 Oct 2020 06:57:47 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 29 Oct 2020 11:00:38 +01:00

perf/x86/intel: Support PERF_SAMPLE_DATA_PAGE_SIZE

The new sample type, PERF_SAMPLE_DATA_PAGE_SIZE, requires the virtual address. Update the data->addr if the sample type is set.

The large PEBS is disabled with the sample type, because perf doesn't support munmap tracking yet. The PEBS buffer for large PEBS cannot be flushed for each munmap. Wrong page size may be calculated. The large PEBS can be enabled later separately when munmap tracking is supported.

---
 arch/x86/events/intel/ds.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/events/intel/ds.c +8 -3
index 404315d..444e5f0 100644
@@ -959,7 +959,8 @@ static void adaptive_pebs_record_size_update(void)
 
 #define PERF_PEBS_MEMINFO_TYPE	(PERF_SAMPLE_ADDR | PERF_SAMPLE_DATA_SRC |   \
 				PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_WEIGHT | \
-				PERF_SAMPLE_TRANSACTION)
+				PERF_SAMPLE_TRANSACTION |		     \
+				PERF_SAMPLE_DATA_PAGE_SIZE)
 
 static u64 pebs_update_adaptive_cfg(struct perf_event *event)
 {
@@ -1335,6 +1336,10 @@ static u64 get_data_src(struct perf_event *event, u64 aux)
 	return val;
 }
 
+#define PERF_SAMPLE_ADDR_TYPE	(PERF_SAMPLE_ADDR |		\
+				 PERF_SAMPLE_PHYS_ADDR |	\
+				 PERF_SAMPLE_DATA_PAGE_SIZE)
+
 static void setup_pebs_fixed_sample_data(struct perf_event *event,
 				   struct pt_regs *iregs, void *__pebs,
 				   struct perf_sample_data *data,
@@ -1449,7 +1454,7 @@ static void setup_pebs_fixed_sample_data(struct perf_event *event,
 	}
 
 
-	if ((sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR)) &&
+	if ((sample_type & PERF_SAMPLE_ADDR_TYPE) &&
 	    x86_pmu.intel_cap.pebs_format >= 1)
 		data->addr = pebs->dla;
 
@@ -1577,7 +1582,7 @@ static void setup_pebs_adaptive_sample_data(struct perf_event *event,
 		if (sample_type & PERF_SAMPLE_DATA_SRC)
 			data->data_src.val = get_data_src(event, meminfo->aux);
 
-		if (sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR))
+		if (sample_type & PERF_SAMPLE_ADDR_TYPE)
 			data->addr = meminfo->address;
 
 		if (sample_type & PERF_SAMPLE_TRANSACTION)
author
#16 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Peter Zijlstra
Quoted 71 lines · Peter Zijlstra

Subject: perf,mm: Handle non-page-table-aligned hugetlbfs
From: Peter Zijlstra<[email protected]>
Date: Fri, 9 Oct 2020 11:09:27 +0200

A limited nunmber of architectures support hugetlbfs sizes that do not align with the page-tables (ARM64, Power, Sparc64). Add support for this to the generic perf_get_page_size() implementation, and also allow an architecture to override this implementation.

This latter is only needed when it uses non-page-table aligned huge pages in its kernel map.

---
  kernel/events/core.c |   39 +++++++++++++++++++++++++++++++++------
  1 file changed, 33 insertions(+), 6 deletions(-)
kernel/events/core.c +24 -4
@@ -6996,10 +6996,18 @@ static u64 perf_virt_to_phys(u64 virt)
  #ifdef CONFIG_MMU
 
  /*
- * Return the MMU page size of a given virtual address
+ * Return the MMU page size of a given virtual address.
+ *
+ * This generic implementation handles page-table aligned huge pages, as well
+ * as non-page-table aligned hugetlbfs compound pages.
+ *
+ * If an architecture supports and uses non-page-table aligned pages in their
+ * kernel mapping it will need to provide it's own implementation of this
+ * function.
   */
-static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+__weak u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)
  {
+	struct page *page;
  	pgd_t *pgd;
  	p4d_t *p4d;
  	pud_t *pud;
@@ -7021,15 +7029,27 @@ static u64 __perf_get_page_size(struct m
  	if (!pud_present(*pud))
  		return 0;
 
-	if (pud_leaf(*pud))
+	if (pud_leaf(*pud)) {
+#ifdef pud_page
+		page = pud_page(*pud);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
  		return 1ULL << PUD_SHIFT;
+	}
 
  	pmd = pmd_offset(pud, addr);
  	if (!pmd_present(*pmd))
  		return 0;
 
-	if (pmd_leaf(*pmd))
+	if (pmd_leaf(*pmd)) {
+#ifdef pmd_page
+		page = pmd_page(*pmd);
+		if (PageHuge(page))
+			return page_size(compound_head(page));
+#endif
  		return 1ULL << PMD_SHIFT;
+	}
 
  	pte = pte_offset_map(pmd, addr);
  	if (!pte_present(*pte)) {

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7066

Quoted 10 lines
kernel/events/core.c +7 -0
@@ -7037,13 +7057,20 @@ static u64 __perf_get_page_size(struct m
  		return 0;
  	}
 
+	page = pte_page(*pte);
+	if (PageHuge(page)) {
+		u64 size = page_size(compound_head(page));
+		pte_unmap(pte);
+		return size;
+	}
+

The PageHuge() check for PTE crashes my machine when I did page size test. (Sorry, I didn't find the issue earlier. I just found some time to re-run the page size test.)

It seems we don't need the check for PTE here. The size should be always PAGE_SIZE, no? After I remove the check, everything looks good.

Thanks,
Kan

[  167.383120] BUG: unable to handle page fault for address: fffffca803fb8000
[  167.383121] #PF: supervisor read access in kernel mode
[  167.383121] #PF: error_code(0x0000) - not-present page
[  167.383121] PGD 4adfc8067 P4D 4adfc8067 PUD 4adfc7067 PMD 0
[  167.383122] Oops: 0000 [#1] SMP NOPTI
[  167.383122] CPU: 0 PID: 2461 Comm: perf Not tainted 5.10.0-rc1_page_size+ #54
[  167.383123] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake U DDR4 SODIMM PD RVP TLC, BIOS ICLSFWR1.R00.3162.A00.1904162000 04/16/2019
[  167.383123] traps: PANIC: double fault, error_code: 0x0
[  167.383123] double fault: 0000 [#2] SMP NOPTI
[  167.383123] CPU: 0 PID: 2461 Comm: perf Not tainted 5.10.0-rc1_page_size+ #54
[  167.383124] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake U DDR4 SODIMM PD RVP TLC, BIOS ICLSFWR1.R00.3162.A00.1904162000 04/16/2019
[  167.383124] RIP: 0010:__sprint_symbol+0xb/0x100
[  167.383124] Code: 85 e4 75 b9 48 85 c0 75 bc 49 89 d8 4c 89 e1 4c 89 fa 4c 89 f6 4c 89 ef e8 42 70 04 00 eb a6 0f 1f 44 00 00 55 48 89 e5 41 57 <41> 56 49 89 f6 41 55 41 89 cd 48 8d 4d b8 41 54 49 89 fc 53 48 63
[  167.383125] RSP: 0018:fffffe000000b000 EFLAGS: 00010046
[  167.383125] RAX: 0000000000000053 RBX: 00000000ffff0a00 RCX: 0000000000000001
[  167.383125] RDX: 0000000000000000 RSI: ffffffff9f8a6176 RDI: fffffe000000b029
[  167.383126] RBP: fffffe000000b008 R08: ffffffffa0bf8661 R09: 0000000000000001
[  167.383126] R10: 000000000000000f R11: ffff9e641dc189c8 R12: ffff9e641dc189c9
[  167.383126] R13: ffff9e641dc1a7e0 R14: ffff0a00ffffff05 R15: fffffe000000b029
[  167.383126] FS:  0000000000000000(0000) GS:ffff9e641dc00000(0000) knlGS:0000000000000000
[  167.383127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  167.383127] CR2: fffffe000000aff8 CR3: 000000014a0ba004 CR4: 0000000000770ef0
[  167.383127] PKRU: 55555554
[  167.383127] Call Trace:
[  167.383127]  <NMI>
[  167.383128]  sprint_symbol+0x15/0x20
[  167.383128]  symbol_string+0x49/0x90
[  167.383128]  pointer+0x118/0x3e0
[  167.383128]  vsnprintf+0x1ec/0x4e0
[  167.383128]  vscnprintf+0xd/0x30
[  167.383129]  printk_safe_log_store+0x65/0xe0
[  167.383129]  vprintk_func+0x8d/0x100
[  167.383129]  printk+0x58/0x6f
[  167.383129]  ? PageHuge+0x6/0x40
[  167.383129]  show_ip+0x2c/0x3d
[  167.383130]  show_iret_regs+0x17/0x40
[  167.383130]  __show_regs+0x27/0x40
[  167.383130]  ? dump_stack_print_info+0x9e/0xb0
[  167.383130]  show_regs+0x3b/0x50
[  167.383130]  __die_body+0x20/0x70
[  167.383131]  __die+0x2b/0x33
[  167.383131]  no_context+0x152/0x350
[  167.383131]  __bad_area_nosemaphore+0x50/0x160
[  167.383131]  bad_area_nosemaphore+0x16/0x20
[  167.383131]  do_kern_addr_fault+0x62/0x70
[  167.383132]  exc_page_fault+0xcd/0x150
[  167.383132]  asm_exc_page_fault+0x1e/0x30
[  167.383132] RIP: 0010:PageHuge+0x6/0x40
[  167.383132] Code: c8 48 89 0e 48 39 47 08 48 0f 46 47 08 48 89 02 c3 0f 1f 00 55 be 00 04 00 00 48 89 e5 e8 72 90 2a 00 5d c3 0f 1f 44 00 00 55 <48> 8b 07 48 89 e5 a9 00 00 01 00 75 09 48 8b 47 08 83 e0 01 74 17
[  167.383133] RSP: 0018:fffffe000000b5c0 EFLAGS: 00010086
[  167.383133] RAX: 00000000fee0017b RBX: fffffca803fb8000 RCX: 0000000000000000
[  167.383133] RDX: 7fffffff011ffe84 RSI: ffff9e635b614fe8 RDI: fffffca803fb8000
[  167.383134] RBP: fffffe000000b5d8 R08: 000fffffc0000000 R09: 000000000000000d
[  167.383134] R10: 0000000000000001 R11: 00000000000011da R12: 000000000048c14f
[  167.383134] R13: fffffe000000b8c0 R14: ffff9e60955aa800 R15: ffffffffff5fd340
[  167.383134]  ? arch_perf_get_page_size+0x2e4/0x330
[  167.383135]  perf_get_page_size.part.0+0x3c/0x50
[  167.383135]  perf_prepare_sample+0x1cc/0x740
[  167.383135]  perf_event_output_forward+0x30/0x90
[  167.383135]  ? sched_clock+0x9/0x10
[  167.383135]  ? __perf_event_account_interrupt+0xcc/0xe0
[  167.383136]  __perf_event_overflow+0x57/0xf0
[  167.383136]  perf_event_overflow+0x14/0x20
[  167.383136]  __intel_pmu_pebs_event+0x2a8/0x3a0
[  167.383136]  ? get_data_src.isra.0+0xc0/0xc0
[  167.383136]  ? perf_event_stop+0xa0/0xa0
[  167.383137]  ? native_apic_mem_write+0x2/0x10
[  167.383137]  ? native_apic_mem_write+0x2/0x10
[  167.383137]  intel_pmu_drain_pebs_icl+0x1bf/0x1f0
[  167.383137]  ? get_data_src.isra.0+0xc0/0xc0
[  167.383137]  handle_pmi_common+0xb6/0x2a0
[  167.383138]  intel_pmu_handle_irq+0xca/0x170
[  167.383138]  perf_event_nmi_handler+0x2d/0x50
[  167.383138]  nmi_handle+0x5d/0x100
[  167.383138]  default_do_nmi+0x45/0x110
[  167.383138]  exc_nmi+0x15a/0x1a0
[  167.383139]  end_repeat_nmi+0x16/0x55
[  167.383139] RIP: 0010:perf_iterate_ctx+0x0/0x170
[  167.383139] Code: b0 95 01 41 89 c5 72 d9 48 c7 c7 40 a0 fb a0 e8 16 47 b8 00 b8 f4 ff ff ff e9 cb fe ff ff 66 90 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 41 57 41 56 41 55 41 54 4c 8d 67 60 53 48 83 ec 08 48
[  167.383140] RSP: 0018:ffffb51d01907ca8 EFLAGS: 00000246
[  167.383140] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[  167.383140] RDX: 0000000000000000 RSI: ffffffff9f815460 RDI: ffff9e60c5423500
[  167.383140] RBP: ffffb51d01907d08 R08: 00000026f8cdcae0 R09: 0000000000000021
[  167.383141] R10: 0000000000000000 R11: ffff9e641dc11ec8 R12: 0000000000000000
[  167.383141] R13: ffff9e641dc33300 R14: ffff9e60aaf69e00 R15: ffff9e60c5423500
[  167.383141]  ? perf_event_stop+0xa0/0xa0
[  167.383141]  ? perf_swevent_init+0x190/0x190
[  167.383142]  ? perf_swevent_init+0x190/0x190
[  167.383142]  </NMI>
[  167.383142]  ? perf_event_exec+0x1ca/0x210
[  167.383142]  begin_new_exec+0x5ba/0x980
[  167.383142]  load_elf_binary+0x6ae/0x17a0
[  167.383143]  ? __kernel_read+0x1a0/0x2d0
[  167.383143]  ? __kernel_read+0x1a0/0x2d0
[  167.383143]  bprm_execve+0x2f6/0x660
[  167.383143]  do_execveat_common.isra.0+0x189/0x1c0
[  167.383143]  __x64_sys_execve+0x37/0x50
[  167.383144]  do_syscall_64+0x38/0x90
[  167.383144]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  167.383144] RIP: 0033:0x7f2c347dae97
[  167.383144] Code: Unable to access opcode bytes at RIP 0x7f2c347dae6d.
[  167.383144] RSP: 002b:00007ffe73f584a8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  167.383145] RAX: ffffffffffffffda RBX: 00005625a990cd70 RCX: 00007f2c347dae97
[  167.383145] RDX: 00005625a990caf0 RSI: 00005625a990cd70 RDI: 00007ffe73f6069f
[  167.383145] RBP: 00007ffe73f58540 R08: 00007ffe73f584a0 R09: 00007f2c368502b0
[  167.383146] R10: 0000000000000016 R11: 0000000000000202 R12: 00005625a990caf0
[  167.383146] R13: 00005625a8c06870 R14: 0000000000000000 R15: 00007ffe73f6069f
[  167.383146] Modules linked in: snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay bnep hid_sensor_magn_3d hid_sensor_gyro_3d hid_sensor_als hid_sensor_accel_3d hid_sensor_rotation hid_sensor_incl_3d hid_sensor_trigger industrialio_triggered_buffer kfifo_buf hid_sensor_iio_common industrialio hid_sensor_custom hid_sensor_hub intel_ishtp_hid binfmt_misc spi_pxa2xx_platform dw_dmac dw_dmac_core 8250_dw mei_hdcp i2c_designware_platform i2c_designware_core intel_rapl_msr wmi_bmof intl_wmi_thunderbolt x86_pkg_temp_thermal snd_hda_intel intel_powerclamp coretemp snd_intel_dspcfg nls_iso8859_1 snd_hda_codec kvm_intel snd_hda_core snd_hwdep kvm
[  167.383158]  snd_pcm irqbypass snd_seq_midi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq_midi_event snd_rawmidi aesni_intel crypto_simd cryptd snd_seq glue_helper rapl snd_seq_device intel_cstate snd_timer ofpart cmdlinepart intel_spi_pci snd e1000e intel_spi spi_nor soundcore mtd iwlmvm mei_me mei mac80211 libarc4 iwlwifi intel_lpss_pci intel_lpss cfg80211 asix usbnet mii joydev btusb btrtl btbcm btintel bluetooth ecdh_generic ecc intel_ish_ipc processor_thermal_device intel_ishtp intel_rapl_common int340x_thermal_zone thunderbolt intel_soc_dts_iosf ucsi_acpi typec_ucsi typec wmi intel_hid int3400_thermal sparse_keymap acpi_pad acpi_thermal_rel acpi_tad sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables hid_generic usbhid hid input_leds serio_raw mac_hid pinctrl_icelake
[  168.121417] ---[ end trace 7f4c6d2d09e5e90f ]---
[  168.121417] RIP: 0010:PageHuge+0x6/0x40
[  168.121417] ode: c8 48 89 0e 48 39 47 08 48 0f 46 47 08 48 89 02 c3 0f 1f 00 55 be 00 04 00 00 48 89 e5 e8 72 90 2a 00 5d c3 0f 1f 44 00 00 55 <48> 8b 07 48 89 e5 a9 00 00 01 00 75 09 48 8b 47 08 83 e0 01 74 17
[  168.121418] RSP: 0018:fffffe000000b5c0 EFLAGS: 00010086
[  168.121418] RAX: 00000000fee0017b RBX: fffffca803fb8000 RCX: 0000000000000000
[  168.121419] RDX: 7fffffff011ffe84 RSI: ffff9e635b614fe8 RDI: fffffca803fb8000
[  168.121419] RBP: fffffe000000b5d8 R08: 000fffffc0000000 R09: 000000000000000d
[  168.121419] R10: 0000000000000001 R11: 00000000000011da R12: 000000000048c14f
[  168.121419] R13: fffffe000000b8c0 R14: ffff9e60955aa800 R15: ffffffffff5fd340
[  168.121420] FS:  0000000000000000(0000) GS:ffff9e641dc00000(0000) knlGS:0000000000000000
[  168.121420] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  168.121420] CR2: 00007f2c347dae6d CR3: 000000014a0ba004 CR4: 0000000000770ef0
[  168.121420] PKRU: 55555554
[  168.121421] Kernel panic - not syncing: Fatal exception in interrupt
[  168.121520] Kernel Offset: 0x1e600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
author
#17 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 86 lines · Kan
Quoted 71 lines · Peter Zijlstra

Subject: perf,mm: Handle non-page-table-aligned hugetlbfs
From: Peter Zijlstra<[email protected]>
Date: Fri, 9 Oct 2020 11:09:27 +0200

A limited nunmber of architectures support hugetlbfs sizes that do not align with the page-tables (ARM64, Power, Sparc64). Add support for this to the generic perf_get_page_size() implementation, and also allow an architecture to override this implementation.

This latter is only needed when it uses non-page-table aligned huge pages in its kernel map.

---
  kernel/events/core.c |   39 +++++++++++++++++++++++++++++++++------

  1 file changed, 33 insertions(+), 6 deletions(-)
kernel/events/core.c +3 -1
@@ -6996,10 +6996,18 @@ static u64 perf_virt_to_phys(u64 virt)
   #ifdef CONFIG_MMU
   /*
- * Return the MMU page size of a given virtual address
+ * Return the MMU page size of a given virtual address.
+ *
+ * This generic implementation handles page-table aligned huge pages, 

as well
+ * as non-page-table aligned hugetlbfs compound pages.
+ *
+ * If an architecture supports and uses non-page-table aligned pages in their
+ * kernel mapping it will need to provide it's own implementation of this
+ * function.
   */
-static u64 __perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+__weak u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)
  {
+    struct page *page;
      pgd_t *pgd;
      p4d_t *p4d;
      pud_t *pud;

kernel/events/core.c +14 -2
@@ -7021,15 +7029,27 @@ static u64 __perf_get_page_size(struct m
       if (!pud_present(*pud))
           return 0;
-    if (pud_leaf(*pud))
+    if (pud_leaf(*pud)) {
+#ifdef pud_page
+        page = pud_page(*pud);
+        if (PageHuge(page))
+            return page_size(compound_head(page));
+#endif
           return 1ULL << PUD_SHIFT;
+    }
       pmd = pmd_offset(pud, addr);
       if (!pmd_present(*pmd))
           return 0;
-    if (pmd_leaf(*pmd))
+    if (pmd_leaf(*pmd)) {
+#ifdef pmd_page
+        page = pmd_page(*pmd);
+        if (PageHuge(page))
+            return page_size(compound_head(page));
+#endif
           return 1ULL << PMD_SHIFT;
+    }
       pte = pte_offset_map(pmd, addr);
       if (!pte_present(*pte)) {

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7065

Quoted 9 lines
kernel/events/core.c +7 -0
@@ -7037,13 +7057,20 @@ static u64 __perf_get_page_size(struct m
           return 0;
       }
+    page = pte_page(*pte);
+    if (PageHuge(page)) {
+        u64 size = page_size(compound_head(page));
+        pte_unmap(pte);
+        return size;
+    }
+

The PageHuge() check for PTE crashes my machine when I did page size test. (Sorry, I didn't find the issue earlier. I just found some time to re-run the page size test.)

Quoted 2 lines

It seems we don't need the check for PTE here. The size should be always PAGE_SIZE, no? After I remove the check, everything looks good.

Hi Peter,

Any comments for this issue?

As my understanding, we don't need the PageHuge() check for PTE page here. However, I don't expect that the page fault crashes here. Because perf already check the pte_present(*pte) before invoking PageHuge().

I'm not sure if it's triggered by other potential issues, so I'm asking here rather than submiting a patch to simply remove the PageHuge() check.

Thanks,
Kan

Quoted 125 lines

[  167.383120] BUG: unable to handle page fault for address: fffffca803fb8000
[  167.383121] #PF: supervisor read access in kernel mode
[  167.383121] #PF: error_code(0x0000) - not-present page
[  167.383121] PGD 4adfc8067 P4D 4adfc8067 PUD 4adfc7067 PMD 0
[  167.383122] Oops: 0000 [#1] SMP NOPTI
[  167.383122] CPU: 0 PID: 2461 Comm: perf Not tainted 5.10.0-rc1_page_size+ #54
[  167.383123] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake U DDR4 SODIMM PD RVP TLC, BIOS ICLSFWR1.R00.3162.A00.1904162000 04/16/2019
[  167.383123] traps: PANIC: double fault, error_code: 0x0
[  167.383123] double fault: 0000 [#2] SMP NOPTI
[  167.383123] CPU: 0 PID: 2461 Comm: perf Not tainted 5.10.0-rc1_page_size+ #54
[  167.383124] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake U DDR4 SODIMM PD RVP TLC, BIOS ICLSFWR1.R00.3162.A00.1904162000 04/16/2019
[  167.383124] RIP: 0010:__sprint_symbol+0xb/0x100
[  167.383124] Code: 85 e4 75 b9 48 85 c0 75 bc 49 89 d8 4c 89 e1 4c 89 fa 4c 89 f6 4c 89 ef e8 42 70 04 00 eb a6 0f 1f 44 00 00 55 48 89 e5 41 57 <41> 56 49 89 f6 41 55 41 89 cd 48 8d 4d b8 41 54 49 89 fc 53 48 63
[  167.383125] RSP: 0018:fffffe000000b000 EFLAGS: 00010046
[  167.383125] RAX: 0000000000000053 RBX: 00000000ffff0a00 RCX: 0000000000000001
[  167.383125] RDX: 0000000000000000 RSI: ffffffff9f8a6176 RDI: fffffe000000b029
[  167.383126] RBP: fffffe000000b008 R08: ffffffffa0bf8661 R09: 0000000000000001
[  167.383126] R10: 000000000000000f R11: ffff9e641dc189c8 R12: ffff9e641dc189c9
[  167.383126] R13: ffff9e641dc1a7e0 R14: ffff0a00ffffff05 R15: fffffe000000b029
[  167.383126] FS:  0000000000000000(0000) GS:ffff9e641dc00000(0000) knlGS:0000000000000000
[  167.383127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  167.383127] CR2: fffffe000000aff8 CR3: 000000014a0ba004 CR4: 0000000000770ef0
[  167.383127] PKRU: 55555554
[  167.383127] Call Trace:
[  167.383127]  <NMI>
[  167.383128]  sprint_symbol+0x15/0x20
[  167.383128]  symbol_string+0x49/0x90
[  167.383128]  pointer+0x118/0x3e0
[  167.383128]  vsnprintf+0x1ec/0x4e0
[  167.383128]  vscnprintf+0xd/0x30
[  167.383129]  printk_safe_log_store+0x65/0xe0
[  167.383129]  vprintk_func+0x8d/0x100
[  167.383129]  printk+0x58/0x6f
[  167.383129]  ? PageHuge+0x6/0x40
[  167.383129]  show_ip+0x2c/0x3d
[  167.383130]  show_iret_regs+0x17/0x40
[  167.383130]  __show_regs+0x27/0x40
[  167.383130]  ? dump_stack_print_info+0x9e/0xb0
[  167.383130]  show_regs+0x3b/0x50
[  167.383130]  __die_body+0x20/0x70
[  167.383131]  __die+0x2b/0x33
[  167.383131]  no_context+0x152/0x350
[  167.383131]  __bad_area_nosemaphore+0x50/0x160
[  167.383131]  bad_area_nosemaphore+0x16/0x20
[  167.383131]  do_kern_addr_fault+0x62/0x70
[  167.383132]  exc_page_fault+0xcd/0x150
[  167.383132]  asm_exc_page_fault+0x1e/0x30
[  167.383132] RIP: 0010:PageHuge+0x6/0x40
[  167.383132] Code: c8 48 89 0e 48 39 47 08 48 0f 46 47 08 48 89 02 c3 0f 1f 00 55 be 00 04 00 00 48 89 e5 e8 72 90 2a 00 5d c3 0f 1f 44 00 00 55 <48> 8b 07 48 89 e5 a9 00 00 01 00 75 09 48 8b 47 08 83 e0 01 74 17
[  167.383133] RSP: 0018:fffffe000000b5c0 EFLAGS: 00010086
[  167.383133] RAX: 00000000fee0017b RBX: fffffca803fb8000 RCX: 0000000000000000
[  167.383133] RDX: 7fffffff011ffe84 RSI: ffff9e635b614fe8 RDI: fffffca803fb8000
[  167.383134] RBP: fffffe000000b5d8 R08: 000fffffc0000000 R09: 000000000000000d
[  167.383134] R10: 0000000000000001 R11: 00000000000011da R12: 000000000048c14f
[  167.383134] R13: fffffe000000b8c0 R14: ffff9e60955aa800 R15: ffffffffff5fd340
[  167.383134]  ? arch_perf_get_page_size+0x2e4/0x330
[  167.383135]  perf_get_page_size.part.0+0x3c/0x50
[  167.383135]  perf_prepare_sample+0x1cc/0x740
[  167.383135]  perf_event_output_forward+0x30/0x90
[  167.383135]  ? sched_clock+0x9/0x10
[  167.383135]  ? __perf_event_account_interrupt+0xcc/0xe0
[  167.383136]  __perf_event_overflow+0x57/0xf0
[  167.383136]  perf_event_overflow+0x14/0x20
[  167.383136]  __intel_pmu_pebs_event+0x2a8/0x3a0
[  167.383136]  ? get_data_src.isra.0+0xc0/0xc0
[  167.383136]  ? perf_event_stop+0xa0/0xa0
[  167.383137]  ? native_apic_mem_write+0x2/0x10
[  167.383137]  ? native_apic_mem_write+0x2/0x10
[  167.383137]  intel_pmu_drain_pebs_icl+0x1bf/0x1f0
[  167.383137]  ? get_data_src.isra.0+0xc0/0xc0
[  167.383137]  handle_pmi_common+0xb6/0x2a0
[  167.383138]  intel_pmu_handle_irq+0xca/0x170
[  167.383138]  perf_event_nmi_handler+0x2d/0x50
[  167.383138]  nmi_handle+0x5d/0x100
[  167.383138]  default_do_nmi+0x45/0x110
[  167.383138]  exc_nmi+0x15a/0x1a0
[  167.383139]  end_repeat_nmi+0x16/0x55
[  167.383139] RIP: 0010:perf_iterate_ctx+0x0/0x170
[  167.383139] Code: b0 95 01 41 89 c5 72 d9 48 c7 c7 40 a0 fb a0 e8 16 47 b8 00 b8 f4 ff ff ff e9 cb fe ff ff 66 90 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 41 57 41 56 41 55 41 54 4c 8d 67 60 53 48 83 ec 08 48
[  167.383140] RSP: 0018:ffffb51d01907ca8 EFLAGS: 00000246
[  167.383140] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[  167.383140] RDX: 0000000000000000 RSI: ffffffff9f815460 RDI: ffff9e60c5423500
[  167.383140] RBP: ffffb51d01907d08 R08: 00000026f8cdcae0 R09: 0000000000000021
[  167.383141] R10: 0000000000000000 R11: ffff9e641dc11ec8 R12: 0000000000000000
[  167.383141] R13: ffff9e641dc33300 R14: ffff9e60aaf69e00 R15: ffff9e60c5423500
[  167.383141]  ? perf_event_stop+0xa0/0xa0
[  167.383141]  ? perf_swevent_init+0x190/0x190
[  167.383142]  ? perf_swevent_init+0x190/0x190
[  167.383142]  </NMI>
[  167.383142]  ? perf_event_exec+0x1ca/0x210
[  167.383142]  begin_new_exec+0x5ba/0x980
[  167.383142]  load_elf_binary+0x6ae/0x17a0
[  167.383143]  ? __kernel_read+0x1a0/0x2d0
[  167.383143]  ? __kernel_read+0x1a0/0x2d0
[  167.383143]  bprm_execve+0x2f6/0x660
[  167.383143]  do_execveat_common.isra.0+0x189/0x1c0
[  167.383143]  __x64_sys_execve+0x37/0x50
[  167.383144]  do_syscall_64+0x38/0x90
[  167.383144]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  167.383144] RIP: 0033:0x7f2c347dae97
[  167.383144] Code: Unable to access opcode bytes at RIP 0x7f2c347dae6d.
[  167.383144] RSP: 002b:00007ffe73f584a8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  167.383145] RAX: ffffffffffffffda RBX: 00005625a990cd70 RCX: 00007f2c347dae97
[  167.383145] RDX: 00005625a990caf0 RSI: 00005625a990cd70 RDI: 00007ffe73f6069f
[  167.383145] RBP: 00007ffe73f58540 R08: 00007ffe73f584a0 R09: 00007f2c368502b0
[  167.383146] R10: 0000000000000016 R11: 0000000000000202 R12: 00005625a990caf0
[  167.383146] R13: 00005625a8c06870 R14: 0000000000000000 R15: 00007ffe73f6069f
[  167.383146] Modules linked in: snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay bnep hid_sensor_magn_3d hid_sensor_gyro_3d hid_sensor_als hid_sensor_accel_3d hid_sensor_rotation hid_sensor_incl_3d hid_sensor_trigger industrialio_triggered_buffer kfifo_buf hid_sensor_iio_common industrialio hid_sensor_custom hid_sensor_hub intel_ishtp_hid binfmt_misc spi_pxa2xx_platform dw_dmac dw_dmac_core 8250_dw mei_hdcp i2c_designware_platform i2c_designware_core intel_rapl_msr wmi_bmof intl_wmi_thunderbolt x86_pkg_temp_thermal snd_hda_intel intel_powerclamp coretemp snd_intel_dspcfg nls_iso8859_1 snd_hda_codec kvm_intel snd_hda_core snd_hwdep kvm [  167.383158]  snd_pcm irqbypass snd_seq_midi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq_midi_event snd_rawmidi aesni_intel crypto_simd cryptd snd_seq glue_helper rapl snd_seq_device intel_cstate snd_timer ofpart cmdlinepart intel_spi_pci snd e1000e intel_spi spi_nor soundcore mtd iwlmvm mei_me mei mac80211 libarc4 iwlwifi intel_lpss_pci intel_lpss cfg80211 asix usbnet mii joydev btusb btrtl btbcm btintel bluetooth ecdh_generic ecc intel_ish_ipc processor_thermal_device intel_ishtp intel_rapl_common int340x_thermal_zone thunderbolt intel_soc_dts_iosf ucsi_acpi typec_ucsi typec wmi intel_hid int3400_thermal sparse_keymap acpi_pad acpi_thermal_rel acpi_tad sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables hid_generic usbhid hid input_leds serio_raw mac_hid pinctrl_icelake
[  168.121417] ---[ end trace 7f4c6d2d09e5e90f ]---
[  168.121417] RIP: 0010:PageHuge+0x6/0x40
[  168.121417] ode: c8 48 89 0e 48 39 47 08 48 0f 46 47 08 48 89 02 c3 0f 1f 00 55 be 00 04 00 00 48 89 e5 e8 72 90 2a 00 5d c3 0f 1f 44 00 00 55 <48> 8b 07 48 89 e5 a9 00 00 01 00 75 09 48 8b 47 08 83 e0 01 74 17
[  168.121418] RSP: 0018:fffffe000000b5c0 EFLAGS: 00010086
[  168.121418] RAX: 00000000fee0017b RBX: fffffca803fb8000 RCX: 0000000000000000
[  168.121419] RDX: 7fffffff011ffe84 RSI: ffff9e635b614fe8 RDI: fffffca803fb8000
[  168.121419] RBP: fffffe000000b5d8 R08: 000fffffc0000000 R09: 000000000000000d
[  168.121419] R10: 0000000000000001 R11: 00000000000011da R12: 000000000048c14f
[  168.121419] R13: fffffe000000b8c0 R14: ffff9e60955aa800 R15: ffffffffff5fd340
[  168.121420] FS:  0000000000000000(0000) GS:ffff9e641dc00000(0000) knlGS:0000000000000000
[  168.121420] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  168.121420] CR2: 00007f2c347dae6d CR3: 000000014a0ba004 CR4: 0000000000770ef0
[  168.121420] PKRU: 55555554
[  168.121421] Kernel panic - not syncing: Fatal exception in interrupt
[  168.121520] Kernel Offset: 0x1e600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

maintainer
#18 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Liang, Kan
Quoted 17 lines · Kan

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7065

Quoted 10 lines · Peter Zijlstra
kernel/events/core.c +7 -0
@@ -7037,13 +7057,20 @@ static u64 __perf_get_page_size(struct m
  		return 0;
  	}
+	page = pte_page(*pte);
+	if (PageHuge(page)) {
+		u64 size = page_size(compound_head(page));
+		pte_unmap(pte);
+		return size;
+	}
+

The PageHuge() check for PTE crashes my machine when I did page size test. (Sorry, I didn't find the issue earlier. I just found some time to re-run the page size test.)

Quoted 2 lines

It seems we don't need the check for PTE here. The size should be always PAGE_SIZE, no? After I remove the check, everything looks good.

That's the thing, an architecture could have non-page-table aligned huge-pages. For example using 4 consecutive 4k pages to create 16k pages. In that case the above code would trigger and find a 16k compound page with HUGETLB_PAGE_DTOR (assuming it was created through hugetlbfs).

What is this page size test; I'd like to reproduce.

maintainer
#19 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Kan

kernel/events/core.c static u64 __perf_get_page_size(struct m · line 7065

Quoted 10 lines · Peter Zijlstra
kernel/events/core.c +7 -0
@@ -7037,13 +7057,20 @@ static u64 __perf_get_page_size(struct m
  		return 0;
  	}
+	page = pte_page(*pte);
+	if (PageHuge(page)) {
+		u64 size = page_size(compound_head(page));
+		pte_unmap(pte);
+		return size;
+	}
+

The PageHuge() check for PTE crashes my machine when I did page size test. (Sorry, I didn't find the issue earlier. I just found some time to re-run the page size test.)

Quoted 7 lines
Quoted 2 lines

It seems we don't need the check for PTE here. The size should be always PAGE_SIZE, no? After I remove the check, everything looks good.

That's the thing, an architecture could have non-page-table aligned huge-pages. For example using 4 consecutive 4k pages to create 16k pages. In that case the above code would trigger and find a 16k compound page with HUGETLB_PAGE_DTOR (assuming it was created through hugetlbfs).

So, from your splat:

Quoted 55 lines
[  167.383120] BUG: unable to handle page fault for address: fffffca803fb8000
[  167.383121] #PF: supervisor read access in kernel mode
[  167.383121] #PF: error_code(0x0000) - not-present page
[  167.383121] PGD 4adfc8067 P4D 4adfc8067 PUD 4adfc7067 PMD 0
[  167.383122] Oops: 0000 [#1] SMP NOPTI
[  167.383122] CPU: 0 PID: 2461 Comm: perf Not tainted 5.10.0-rc1_page_size+ #54
[  167.383123] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake U DDR4 SODIMM PD RVP TLC, BIOS ICLSFWR1.R00.3162.A00.1904162000 04/16/2019
[  167.383123] traps: PANIC: double fault, error_code: 0x0
[  167.383123] double fault: 0000 [#2] SMP NOPTI
[  167.383123] CPU: 0 PID: 2461 Comm: perf Not tainted 5.10.0-rc1_page_size+ #54
[  167.383124] Hardware name: Intel Corporation Ice Lake Client Platform/IceLake U DDR4 SODIMM PD RVP TLC, BIOS ICLSFWR1.R00.3162.A00.1904162000 04/16/2019
[  167.383124] RIP: 0010:__sprint_symbol+0xb/0x100
[  167.383124] Code: 85 e4 75 b9 48 85 c0 75 bc 49 89 d8 4c 89 e1 4c 89 fa 4c 89 f6 4c 89 ef e8 42 70 04 00 eb a6 0f 1f 44 00 00 55 48 89 e5 41 57 <41> 56 49 89 f6 41 55 41 89 cd 48 8d 4d b8 41 54 49 89 fc 53 48 63
[  167.383125] RSP: 0018:fffffe000000b000 EFLAGS: 00010046
[  167.383125] RAX: 0000000000000053 RBX: 00000000ffff0a00 RCX: 0000000000000001
[  167.383125] RDX: 0000000000000000 RSI: ffffffff9f8a6176 RDI: fffffe000000b029
[  167.383126] RBP: fffffe000000b008 R08: ffffffffa0bf8661 R09: 0000000000000001
[  167.383126] R10: 000000000000000f R11: ffff9e641dc189c8 R12: ffff9e641dc189c9
[  167.383126] R13: ffff9e641dc1a7e0 R14: ffff0a00ffffff05 R15: fffffe000000b029
[  167.383126] FS:  0000000000000000(0000) GS:ffff9e641dc00000(0000) knlGS:0000000000000000
[  167.383127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  167.383127] CR2: fffffe000000aff8 CR3: 000000014a0ba004 CR4: 0000000000770ef0
[  167.383127] PKRU: 55555554
[  167.383127] Call Trace:
[  167.383127]  <NMI>
[  167.383128]  sprint_symbol+0x15/0x20
[  167.383128]  symbol_string+0x49/0x90
[  167.383128]  pointer+0x118/0x3e0
[  167.383128]  vsnprintf+0x1ec/0x4e0
[  167.383128]  vscnprintf+0xd/0x30
[  167.383129]  printk_safe_log_store+0x65/0xe0
[  167.383129]  vprintk_func+0x8d/0x100
[  167.383129]  printk+0x58/0x6f
[  167.383129]  ? PageHuge+0x6/0x40
[  167.383129]  show_ip+0x2c/0x3d
[  167.383130]  show_iret_regs+0x17/0x40
[  167.383130]  __show_regs+0x27/0x40
[  167.383130]  ? dump_stack_print_info+0x9e/0xb0
[  167.383130]  show_regs+0x3b/0x50
[  167.383130]  __die_body+0x20/0x70
[  167.383131]  __die+0x2b/0x33
[  167.383131]  no_context+0x152/0x350
[  167.383131]  __bad_area_nosemaphore+0x50/0x160
[  167.383131]  bad_area_nosemaphore+0x16/0x20
[  167.383131]  do_kern_addr_fault+0x62/0x70
[  167.383132]  exc_page_fault+0xcd/0x150
[  167.383132]  asm_exc_page_fault+0x1e/0x30
[  167.383132] RIP: 0010:PageHuge+0x6/0x40
[  167.383132] Code: c8 48 89 0e 48 39 47 08 48 0f 46 47 08 48 89 02 c3 0f 1f 00 55 be 00 04 00 00 48 89 e5 e8 72 90 2a 00 5d c3 0f 1f 44 00 00 55 <48> 8b 07 48 89 e5 a9 00 00 01 00 75 09 48 8b 47 08 83 e0 01 74 17
[  167.383133] RSP: 0018:fffffe000000b5c0 EFLAGS: 00010086
[  167.383133] RAX: 00000000fee0017b RBX: fffffca803fb8000 RCX: 0000000000000000
[  167.383133] RDX: 7fffffff011ffe84 RSI: ffff9e635b614fe8 RDI: fffffca803fb8000
[  167.383134] RBP: fffffe000000b5d8 R08: 000fffffc0000000 R09: 000000000000000d
[  167.383134] R10: 0000000000000001 R11: 00000000000011da R12: 000000000048c14f
[  167.383134] R13: fffffe000000b8c0 R14: ffff9e60955aa800 R15: ffffffffff5fd340
Quoted 8 lines
[  167.383134]  ? arch_perf_get_page_size+0x2e4/0x330
[  167.383135]  perf_get_page_size.part.0+0x3c/0x50
[  167.383135]  perf_prepare_sample+0x1cc/0x740
[  167.383135]  perf_event_output_forward+0x30/0x90
[  167.383135]  ? sched_clock+0x9/0x10
[  167.383135]  ? __perf_event_account_interrupt+0xcc/0xe0
[  167.383136]  __perf_event_overflow+0x57/0xf0
[  167.383136]  perf_event_overflow+0x14/0x20

The faulting instruction decodes to:

48 8b 07 mov (%rdi),%rax

And the RDI value is indeed fffffca803fb8000 as per the first BUG line, which, afaict, is in a hole per x86_64/mm.rst

Trying to match the Code: to PageHuge as generate here makes this the PageCompound() test burning, not even compound_head() going bad.

must ponder more...

maintainer
#20 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 4 lines · Peter Zijlstra

Trying to match the Code: to PageHuge as generate here makes this the PageCompound() test burning, not even compound_head() going bad.

must ponder more...

Oooh.. does this help?

---
 kernel/events/core.c | 81 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 49 insertions(+), 32 deletions(-)
kernel/events/core.c +49 -32
index d2f3ca792936..3b42576c99f1 100644
@@ -7015,65 +7015,82 @@ static u64 perf_virt_to_phys(u64 virt)
  */
 __weak u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)
 {
+#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL
 	struct page *page;
-	pgd_t *pgd;
-	p4d_t *p4d;
-	pud_t *pud;
-	pmd_t *pmd;
-	pte_t *pte;
-
-	pgd = pgd_offset(mm, addr);
-	if (pgd_none(*pgd))
+	pgd_t *pgdp, pgd;
+	p4d_t *p4dp, p4d;
+	pud_t *pudp, pud;
+	pmd_t *pmdp, pmd;
+	pte_t *ptep, pte;
+
+	pgdp = pgd_offset(mm, addr);
+	pgd = READ_ONCE(*pgdp);
+	if (pgd_none(pgd))
 		return 0;
 
-	p4d = p4d_offset(pgd, addr);
-	if (!p4d_present(*p4d))
+	p4dp = p4d_offset(&pgd, addr);
+	p4d = READ_ONCE(*p4dp);
+	if (!p4d_present(p4d))
 		return 0;
 
-	if (p4d_leaf(*p4d))
+	if (p4d_leaf(p4d))
 		return 1ULL << P4D_SHIFT;
 
-	pud = pud_offset(p4d, addr);
-	if (!pud_present(*pud))
+	pudp = pud_offset(&p4d, addr);
+	pud = READ_ONCE(*pudp);
+	if (!pud_present(pud))
 		return 0;
 
-	if (pud_leaf(*pud)) {
+	if (pud_leaf(pud)) {
 #ifdef pud_page
-		page = pud_page(*pud);
-		if (PageHuge(page))
-			return page_size(compound_head(page));
+		if (!pud_devmap(pud)) {
+			page = pud_page(pud);
+			if (PageHuge(page))
+				return page_size(compound_head(page));
+		}
 #endif
 		return 1ULL << PUD_SHIFT;
 	}
 
-	pmd = pmd_offset(pud, addr);
-	if (!pmd_present(*pmd))
+	pmdp = pmd_offset(&pud, addr);
+	pmd = READ_ONCE(*pmdp);
+	if (!pmd_present(pmd))
 		return 0;
 
-	if (pmd_leaf(*pmd)) {
+	if (pmd_leaf(pmd)) {
 #ifdef pmd_page
-		page = pmd_page(*pmd);
-		if (PageHuge(page))
-			return page_size(compound_head(page));
+		if (!pmd_devmap(pmd)) {
+			page = pmd_page(pmd);
+			if (PageHuge(page))
+				return page_size(compound_head(page));
+		}
 #endif
 		return 1ULL << PMD_SHIFT;
 	}
 
-	pte = pte_offset_map(pmd, addr);
-	if (!pte_present(*pte)) {
-		pte_unmap(pte);
+	ptep = pte_offset_map(&pmd, addr);
+	pte = READ_ONCE(*ptep); // gup_get_pte()
+	if (!pte_present(pte)) {
+		pte_unmap(ptep);
 		return 0;
 	}
 
-	page = pte_page(*pte);
-	if (PageHuge(page)) {
-		u64 size = page_size(compound_head(page));
-		pte_unmap(pte);
-		return size;
+	if (!pte_devmap(pte) && !pte_special(pte)) {
+		page = pte_page(pte);
+		if (PageHuge(page)) {
+			u64 size = page_size(compound_head(page));
+			pte_unmap(ptep);
+			return size;
+		}
 	}
 
-	pte_unmap(pte);
+	pte_unmap(ptep);
 	return PAGE_SIZE;
+
+#else /* CONFIG_ARCH_HAS_PTE_SPECIAL */
+
+	return 0;
+#endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */
 }
 
 #else
#21 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7052

Quoted 12 lines · Peter Zijlstra
kernel/events/core.c +6 -3
+	if (pud_leaf(pud)) {
 #ifdef pud_page
-		page = pud_page(*pud);
-		if (PageHuge(page))
-			return page_size(compound_head(page));
+		if (!pud_devmap(pud)) {
+			page = pud_page(pud);
+			if (PageHuge(page))
+				return page_size(compound_head(page));
+		}
 #endif
 		return 1ULL << PUD_SHIFT;

This confuses me. Why only special-case hugetlbfs pages here? Should they really be treated differently from THP? If you want to consider that we might be mapping a page that's twice as big as a PUD entry and this is only half of it, then the simple way is:

	if (pud_leaf(pud)) {
#ifdef pud_page
		page = compound_head(pud_page(*pud));
		return page_size(page);
#else
		return 1ULL << PUD_SHIFT;
#endif
	}

Also, what's up with the special-casing of devmap pages here? Did the devmap people fuck up their compound pages? If so, they should fix their shit, not expect the rest of the kernel to work around this brokenness.

maintainer
#22 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 16 lines · Matthew Wilcox

kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7052

Quoted 12 lines · Peter Zijlstra
kernel/events/core.c +6 -3
+	if (pud_leaf(pud)) {
 #ifdef pud_page
-		page = pud_page(*pud);
-		if (PageHuge(page))
-			return page_size(compound_head(page));
+		if (!pud_devmap(pud)) {
+			page = pud_page(pud);
+			if (PageHuge(page))
+				return page_size(compound_head(page));
+		}
 #endif
 		return 1ULL << PUD_SHIFT;

This confuses me. Why only special-case hugetlbfs pages here? Should they really be treated differently from THP?

Do we have non-pagetable aligned THP ? I thought THP was always PUD sized.

Quoted 13 lines

If you want to consider that we might be mapping a page that's twice as big as a PUD entry and this is only half of it, then the simple way is:

	if (pud_leaf(pud)) {
#ifdef pud_page
		page = compound_head(pud_page(*pud));
		return page_size(page);
#else
		return 1ULL << PUD_SHIFT;
#endif
	}
Quoted 3 lines

Also, what's up with the special-casing of devmap pages here? Did the devmap people fuck up their compound pages? If so, they should fix their shit, not expect the rest of the kernel to work around this brokenness.

Well, the PTE code we have today (in tip/perf/core) is:

	pte = pte_offset_map(pmd, addr);
	if (!pte_present(*pte)) {
		pte_unmap(pte);
		return 0;
	}

	page = pte_page(*pte);
	if (PageHuge(page)) {
		u64 size = page_size(compound_head(page));
		pte_unmap(pte);
		return size;
	}

	pte_unmap(pte);
	return PAGE_SIZE;

and that's crashing in PageHuge()'s PageCompound() test. Clearly I should be checking pte_special() here (as well as all the READ_ONCE()s I added in the patch you just commented on). But I wasn't quite sure about devmap and paranoia won.

You're saying devmap should be valid compound pages? Then I can remove all that and only keep pte_special().

maintainer
#23 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

In reply to Matthew Wilcox
Quoted 9 lines · Matthew Wilcox

This confuses me. Why only special-case hugetlbfs pages here? Should they really be treated differently from THP? If you want to consider that we might be mapping a page that's twice as big as a PUD entry and this is only half of it, then the simple way is:

	if (pud_leaf(pud)) {
#ifdef pud_page
		page = compound_head(pud_page(*pud));
		return page_size(page);

Also; this is 'wrong'. The purpose of this function is to return the hardware TLB size of a given address. The above will return the compound size, for any random compound page, which would be myrads of reasons.

So the PageHuge() thing tells us it is a HugeTLB page and those are (barring hardware TLB promotion/demotion) guaranteed to reflect the actual TLB size.

Quoted 4 lines
#else
		return 1ULL << PUD_SHIFT;
#endif
	}
#24 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Matthew Wilcox

This confuses me. Why only special-case hugetlbfs pages here? Should they really be treated differently from THP? If you want to consider that we might be mapping a page that's twice as big as a PUD entry and this is only half of it, then the simple way is:

Quoted 8 lines
Quoted 4 lines
	if (pud_leaf(pud)) {
#ifdef pud_page
		page = compound_head(pud_page(*pud));
		return page_size(page);

Also; this is 'wrong'. The purpose of this function is to return the hardware TLB size of a given address. The above will return the compound size, for any random compound page, which would be myrads of reasons.

Oh, then the whole thing is overly-complicated. This should just be

	if (pud_leaf(pud))
		return PUD_SIZE;
maintainer
#25 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 12 lines · Matthew Wilcox
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Matthew Wilcox

This confuses me. Why only special-case hugetlbfs pages here? Should they really be treated differently from THP? If you want to consider that we might be mapping a page that's twice as big as a PUD entry and this is only half of it, then the simple way is:

	if (pud_leaf(pud)) {
#ifdef pud_page
		page = compound_head(pud_page(*pud));
		return page_size(page);
Quoted 8 lines
Quoted 3 lines

Also; this is 'wrong'. The purpose of this function is to return the hardware TLB size of a given address. The above will return the compound size, for any random compound page, which would be myrads of reasons.

Oh, then the whole thing is overly-complicated. This should just be

	if (pud_leaf(pud))
		return PUD_SIZE;

But that doesn't handle non-pagetable aligned hugetlb sizes. Granted, that's unlikely at the PUD level, but why be inconsistent..

So we really want:

	if (p*d_leaf(p*d)) {
		if (!'special') {
			page = p*d_page(p*d);
			if (PageHuge(page))
				return page_size(compound_head(page));
		}
		return P*D_SIZE;
	}

That gets us:

- regular page-table aligned large-pages
- 'funny' hugetlb sizes

The only thing it doesn't gets us is kernel usage of 'funny' sizes, which is why that function is weak (arm64, power32, sparc64 have funny sizes and at the very least arm64 uses them for kernel mappings too).

Now, when you add !PMD THP sizes (presumably for architectures that have 'funny' sizes, otherwise what's the point), then you get to add '|| PageTransHuge()' to the above PageHuge() (and fix PageTransHuge() to actually do what it claims it does).

Arguably we could fix arm64 with something like the below, but then, I'd have to audit powerpc32 and sparc64 again to see if I can make that work for them too -- not today.

---

kernel/events/core.c +5 -1
@@ -7003,6 +7003,10 @@ static u64 perf_virt_to_phys(u64 virt)
 
 #ifdef CONFIG_MMU
 
+#ifndef pte_cont
+#define pte_cont(pte)	(false)
+#endif
+
 /*
  * Return the MMU page size of a given virtual address.
  *
@@ -7077,7 +7081,7 @@ __weak u64 arch_perf_get_page_size(struc
 
 	if (!pte_devmap(pte) && !pte_special(pte)) {
 		page = pte_page(pte);
-		if (PageHuge(page)) {
+		if (PageHuge(page) || pte_cont(pte)) {
 			u64 size = page_size(compound_head(page));
 			pte_unmap(ptep);
 			return size;
#26 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 9 lines · Peter Zijlstra
Quoted 2 lines · Matthew Wilcox
	if (pud_leaf(pud))
		return PUD_SIZE;

But that doesn't handle non-pagetable aligned hugetlb sizes. Granted, that's unlikely at the PUD level, but why be inconsistent..

So we really want:

Quoted 8 lines
	if (p*d_leaf(p*d)) {
		if (!'special') {
			page = p*d_page(p*d);
			if (PageHuge(page))
				return page_size(compound_head(page));
		}
		return P*D_SIZE;
	}

Still doesn't work because pages can be mapped at funny offsets.

What we really want is for a weak definition of

unsigned long tlb_size(struct mm_struct *mm, unsigned long addr)

{
	if (p*d_leaf(p*d))
		return p*d_size(p*d);
}

then ARM can look at its special bit in the page table to determine whether this is a singleton or part of a brace of pages.

Quoted 2 lines

Now, when you add !PMD THP sizes (presumably for architectures that have 'funny' sizes, otherwise what's the point), then you get to add '||

This is the problem with all the huge page support in Linux today. It's written by people who work for hardware companies who think only about exploiting the hardware features they sell. You all ignore the very real software overhedas of trying to manage millions of pages. I see a 6% reduction in kernel overhead when running kernbench using THPs that may go as large as 256kB. On x86. Intel x86, at that.

maintainer
#27 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 19 lines · Matthew Wilcox
Quoted 9 lines · Peter Zijlstra
Quoted 2 lines · Matthew Wilcox
	if (pud_leaf(pud))
		return PUD_SIZE;

But that doesn't handle non-pagetable aligned hugetlb sizes. Granted, that's unlikely at the PUD level, but why be inconsistent..

So we really want:

Quoted 8 lines
	if (p*d_leaf(p*d)) {
		if (!'special') {
			page = p*d_page(p*d);
			if (PageHuge(page))
				return page_size(compound_head(page));
		}
		return P*D_SIZE;
	}
Quoted 1 line

Still doesn't work because pages can be mapped at funny offsets.

Wait, what?! Is there hardware that has unaligned TLB page-sizes?

Can you start a 64K page at an 8k offset? I don't think I've ever seen that. Still even with that, how would the above go wrong there? It would find the compound page covering @addr, PageHuge() (and possibly some addition arch specific condition) returns true and we get the compound size to find the hardware page size used.

Quoted 10 lines

What we really want is for a weak definition of

unsigned long tlb_size(struct mm_struct *mm, unsigned long addr)

{
	if (p*d_leaf(p*d))
		return p*d_size(p*d);
}

then ARM can look at its special bit in the page table to determine whether this is a singleton or part of a brace of pages.

That's basically what we provide. but really the only thing that's missing from this generic page walker is the ability to detect if a !PageHuge compound page is actually still a hardware page.

Quoted 9 lines
Quoted 2 lines

Now, when you add !PMD THP sizes (presumably for architectures that have 'funny' sizes, otherwise what's the point), then you get to add '||

This is the problem with all the huge page support in Linux today. It's written by people who work for hardware companies who think only about exploiting the hardware features they sell. You all ignore the very real software overhedas of trying to manage millions of pages. I see a 6% reduction in kernel overhead when running kernbench using THPs that may go as large as 256kB. On x86. Intel x86, at that.

That's a really nice improvement. However then this code doesn't care about it. Please make it possible to distinguish between THP on hardware pages vs software pages.

#28 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Matthew Wilcox
Quoted 17 lines · Peter Zijlstra
Quoted 2 lines · Matthew Wilcox
	if (pud_leaf(pud))
		return PUD_SIZE;

But that doesn't handle non-pagetable aligned hugetlb sizes. Granted, that's unlikely at the PUD level, but why be inconsistent..

So we really want:

	if (p*d_leaf(p*d)) {
		if (!'special') {
			page = p*d_page(p*d);
			if (PageHuge(page))
				return page_size(compound_head(page));
		}
		return P*D_SIZE;
	}
Quoted 3 lines
Quoted 1 line

Still doesn't work because pages can be mapped at funny offsets.

Wait, what?! Is there hardware that has unaligned TLB page-sizes?

No, you can force a 2MB page to be mapped at an address which isn't 2MB aligned.

Quoted 5 lines

Can you start a 64K page at an 8k offset? I don't think I've ever seen that. Still even with that, how would the above go wrong there? It would find the compound page covering @addr, PageHuge() (and possibly some addition arch specific condition) returns true and we get the compound size to find the hardware page size used.

On any architecture I can think of, that 2MB page will be mapped with 4kB TLB entries.

Quoted 25 lines
Quoted 10 lines

What we really want is for a weak definition of

unsigned long tlb_size(struct mm_struct *mm, unsigned long addr)

{
	if (p*d_leaf(p*d))
		return p*d_size(p*d);
}

then ARM can look at its special bit in the page table to determine whether this is a singleton or part of a brace of pages.

That's basically what we provide. but really the only thing that's missing from this generic page walker is the ability to detect if a !PageHuge compound page is actually still a hardware page.

Quoted 9 lines
Quoted 2 lines

Now, when you add !PMD THP sizes (presumably for architectures that have 'funny' sizes, otherwise what's the point), then you get to add '||

This is the problem with all the huge page support in Linux today. It's written by people who work for hardware companies who think only about exploiting the hardware features they sell. You all ignore the very real software overhedas of trying to manage millions of pages. I see a 6% reduction in kernel overhead when running kernbench using THPs that may go as large as 256kB. On x86. Intel x86, at that.

Quoted 3 lines

That's a really nice improvement. However then this code doesn't care about it. Please make it possible to distinguish between THP on hardware pages vs software pages.

That can and should be done just by looking at the page table entries. There's no need to convert it into a struct page. The CPU obviously decides what TLB entry size to use based solely on the page tables, so we can too.

maintainer
#29 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 21 lines · Matthew Wilcox
Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Matthew Wilcox
Quoted 17 lines · Peter Zijlstra
Quoted 2 lines · Matthew Wilcox
	if (pud_leaf(pud))
		return PUD_SIZE;

But that doesn't handle non-pagetable aligned hugetlb sizes. Granted, that's unlikely at the PUD level, but why be inconsistent..

So we really want:

	if (p*d_leaf(p*d)) {
		if (!'special') {
			page = p*d_page(p*d);
			if (PageHuge(page))
				return page_size(compound_head(page));
		}
		return P*D_SIZE;
	}
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Still doesn't work because pages can be mapped at funny offsets.

Wait, what?! Is there hardware that has unaligned TLB page-sizes?

No, you can force a 2MB page to be mapped at an address which isn't 2MB aligned.

But not a HugeTLB page; AFAICT mmap() will reject if you try and mmap a hugetlb page out of alignment. So what I wrote above is still valid. If PageHuge() we can be certain it is aligned properly and using a matching hardware page size.

You just don't like it because you want me to be purely page-table based.

maintainer
#30 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 2 lines · Peter Zijlstra

You just don't like it because you want me to be purely page-table based.

How's something like this then? I failed to untangle Power's many MMUs though :/

---
 arch/arm64/include/asm/pgtable.h    |   3 ++
 arch/sparc/include/asm/pgtable_64.h |  14 +++++
 arch/sparc/mm/hugetlbpage.c         |  19 ++++---
 include/linux/pgtable.h             |  16 ++++++
 kernel/events/core.c                | 100 +++++++++++++-----------------------
 5 files changed, 82 insertions(+), 70 deletions(-)
arch/arm64/include/asm/pgtable.h +3 -0
index 4ff12a7adcfd..1cd2d986b0ca 100644
@@ -503,6 +503,9 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
 				 PMD_TYPE_SECT)
 #define pmd_leaf(pmd)		pmd_sect(pmd)
 
+#define pmd_leaf_size(pmd)	(pmd_cont(pmd) ? CONT_PMD_SIZE : PMD_SIZE)
+#define pte_leaf_size(pte)	(pte_cont(pte) ? CONT_PTE_SIZE : PAGE_SIZE)
+
 #if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS < 3
 static inline bool pud_sect(pud_t pud) { return false; }
 static inline bool pud_table(pud_t pud) { return true; }
arch/sparc/include/asm/pgtable_64.h +14 -0
index 7ef6affa105e..1723e18ba89f 100644
@@ -1121,6 +1121,20 @@ extern unsigned long cmdline_memory_size;
 
 asmlinkage void do_sparc64_fault(struct pt_regs *regs);
 
+#ifdef CONFIG_HUGETLB_PAGE
+
+#define pud_leaf_size pud_leaf_size
+extern unsigned long pud_leaf_size(pud_t pud);
+
+#define pmd_leaf_size pmd_leaf_size
+extern unsigned long pmd_leaf_size(pmd_t pmd);
+
+#define pte_leaf_size pte_leaf_size
+extern unsigned long pte_leaf_size(pte_t pte);
+#endif
+
+#endif /* CONFIG_HUGETLB_PAGE */
+
 #endif /* !(__ASSEMBLY__) */
 
 #endif /* !(_SPARC64_PGTABLE_H) */
arch/sparc/mm/hugetlbpage.c +13 -6
index ec423b5f17dd..3e806b87ec19 100644
@@ -247,14 +247,17 @@ static unsigned int sun4u_huge_tte_to_shift(pte_t entry)
 	return shift;
 }
 
-static unsigned int huge_tte_to_shift(pte_t entry)
+static unsigned long __tte_to_shift(pte_t entry)
 {
-	unsigned long shift;
-
 	if (tlb_type == hypervisor)
-		shift = sun4v_huge_tte_to_shift(entry);
-	else
-		shift = sun4u_huge_tte_to_shift(entry);
+		return sun4v_huge_tte_to_shift(entry);
+
+	return sun4u_huge_tte_to_shift(entry);
+}
+
+static unsigned int huge_tte_to_shift(pte_t entry)
+{
+	unsigned long shift = __tte_to_shift(entry);
 
 	if (shift == PAGE_SHIFT)
 		WARN_ONCE(1, "tto_to_shift: invalid hugepage tte=0x%lx\n",
@@ -272,6 +275,10 @@ static unsigned long huge_tte_to_size(pte_t pte)
 	return size;
 }
 
+unsigned long pud_leaf_size(pud_t pud) { return 1UL << __tte_to_shift((pte_t)pud); }
+unsigned long pmd_leaf_size(pmd_t pmd) { return 1UL << __tte_to_shift((pte_t)pmd); }
+unsigned long pte_leaf_size(pte_t pte) { return 1UL << __tte_to_shift((pte_t)pte); }
+
 pte_t *huge_pte_alloc(struct mm_struct *mm,
 			unsigned long addr, unsigned long sz)
 {
include/linux/pgtable.h +16 -0
index 71125a4676c4..35b9da397e55 100644
@@ -1481,4 +1481,20 @@ typedef unsigned int pgtbl_mod_mask;
 #define pmd_leaf(x)	0
 #endif
 
+#ifndef pgd_leaf_size(x)
+#define pgd_leaf_size(x) PGD_SIZE
+#endif
+#ifndef p4d_leaf_size(x)
+#define p4d_leaf_size(x) P4D_SIZE
+#endif
+#ifndef pud_leaf_size(x)
+#define pud_leaf_size(x) PUD_SIZE
+#endif
+#ifndef pmd_leaf_size(x)
+#define pmd_leaf_size(x) PMD_SIZE
+#endif
+#ifndef pte_leaf_size(x)
+#define pte_leaf_size(x) PAGE_SIZE
+#endif
+
 #endif /* _LINUX_PGTABLE_H */
kernel/events/core.c +36 -64

126 lines not on this page · raw message

#31 lore raw

Re: [PATCH V9 1/4] perf/core: Add PERF_SAMPLE_DATA_PAGE_SIZE

Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

You just don't like it because you want me to be purely page-table based.

How's something like this then? I failed to untangle Power's many MMUs though :/

Looks good to me. Might want to rename

kernel/events/core.c static u64 perf_virt_to_phys(u64 virt) · line 7007

Quoted 2 lines
kernel/events/core.c +1 -1
-__weak u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)
+static u64 arch_perf_get_page_size(struct mm_struct *mm, unsigned long addr)

to perf_get_tlb_entry_size() or some such.