Skip to contentSkip to conversation

vmalloc_exec for modules and BPF programs

PATCH RESEND In review Song Liu 6 participants 29 messages Oct 31, 2022 – Nov 7, 2022

Subsystems (8)
Memory management x86 BPF x86 MM +4 more subsystems
lore
Share & tools
lore

Song Liu

author reviewer
#0 lore raw

cover vmalloc_exec for modules and BPF programs

Original email didn't make to lore, resend with a shorter CC list.

This set enables bpf programs and bpf dispatchers to share huge pages with new API:
vmalloc_exec()
vfree_exec()
vcopy_exec()

The idea is similar to Peter's suggestion in [1].

vmalloc_exec() manages a set of PMD_SIZE RO+X memory, and allocates these memory to its users. vfree_exec() is used to free memory allocated by vmalloc_exec(). vcopy_exec() is used to update memory allocated by vmalloc_exec().

Memory allocated by vmalloc_exec() is RO+X, so this doesnot violate W^X. The caller has to update the content with text_poke like mechanism. Specifically, vcopy_exec() is provided to update memory allocated by vmalloc_exec(). vcopy_exec() also makes sure the update stays in the boundary of one chunk allocated by vmalloc_exec(). Please refer to patch 1/5 for more details of

Patch 3/5 uses these new APIs in bpf program and bpf dispatcher.

Patch 4/5 and 5/5 allows static kernel text (_stext to _etext) to share PMD_SIZE pages with dynamic kernel text on x86_64. This is achieved by allocating PMD_SIZE pages to roundup(_etext, PMD_SIZE), and then use _etext to roundup(_etext, PMD_SIZE) for dynamic kernel text.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] RFC v1: https://lore.kernel.org/linux-mm/[email protected]/T/

Changes RFC v2 => PATCH v1:
1. Add vcopy_exec(), which updates memory allocated by vmalloc_exec(). It also ensures vcopy_exec() is only used to update memory from one single vmalloc_exec() call. (Christoph Hellwig)
2. Add arch_vcopy_exec() and arch_invalidate_exec() as wrapper for the text_poke() like logic.
3. Drop changes for kernel modules and focus on BPF side changes.

Changes RFC v1 => RFC v2:
1. Major rewrite of the logic of vmalloc_exec and vfree_exec. They now work fine with BPF programs (patch 1, 2, 4). But module side (patch 3) still need some work.

Song Liu (5):
vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec x86/alternative: support vmalloc_exec() and vfree_exec()
bpf: use vmalloc_exec for bpf program and bpf dispatcher
vmalloc: introduce register_text_tail_vm()
x86: use register_text_tail_vm

 arch/x86/include/asm/pgtable_64_types.h |   1 +
 arch/x86/kernel/alternative.c           |  12 +
 arch/x86/mm/init_64.c                   |   4 +-
 arch/x86/net/bpf_jit_comp.c             |  23 +-
 include/linux/bpf.h                     |   3 -
 include/linux/filter.h                  |   5 -
 include/linux/vmalloc.h                 |   9 +
 kernel/bpf/core.c                       | 180 +-----------
 kernel/bpf/dispatcher.c                 |  11 +-
 mm/nommu.c                              |   7 +
 mm/vmalloc.c                            | 351 ++++++++++++++++++++++++
 11 files changed, 404 insertions(+), 202 deletions(-)

--
2.30.2

author reviewer
#1 lore raw

PATCH RESEND 1/5 vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

vmalloc_exec is used to allocate memory to host dynamic kernel text (modules, BPF programs, etc.) with huge pages. This is similar to the proposal by Peter in [1].

A new tree of vmap_area, free_text_area_* tree, is introduced in addition to free_vmap_area_* and vmap_area_*. vmalloc_exec allocates pages from free_text_area_*. When there isn't enough space left in free_text_area_*, new PMD_SIZE page(s) is allocated from free_vmap_area_* and added to free_text_area_*. To be more accurate, the vmap_area is first added to vmap_area_* tree and then moved to free_text_area_*. This extra move simplifies the logic of vmalloc_exec.

vmap_area in free_text_area_* tree are backed with memory, but we need subtree_max_size for tree operations. Therefore, vm_struct for these vmap_area are stored in a separate list, all_text_vm.

The new tree allows separate handling of < PAGE_SIZE allocations, as current vmalloc code mostly assumes PAGE_SIZE aligned allocations. This version of vmalloc_exec can handle bpf programs, which uses 64 byte aligned allocations), and modules, which uses PAGE_SIZE aligned allocations.

Memory allocated by vmalloc_exec() is set to RO+X before returning to the caller. Therefore, the caller cannot write directly write to the memory. Instead, the caller is required to use vcopy_exec() to update the memory. For the safety and security of X memory, vcopy_exec() checks the data being updated always in the memory allocated by one vmalloc_exec() call. vcopy_exec() uses text_poke like mechanism and requires arch support. Specifically, the arch need to implement arch_vcopy_exec().

In vfree_exec(), the memory is first erased with arch_invalidate_exec(). Then, the memory is added to free_text_area_*. If this free creates big enough continuous free space (> PMD_SIZE), vfree_exec() will try to free the backing vm_struct.

[1] https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/vmalloc.h |   5 +
 mm/nommu.c              |  12 ++
 mm/vmalloc.c            | 318 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 335 insertions(+)
include/linux/vmalloc.h +5 -0
index 096d48aa3437..9b2042313c12 100644
@@ -154,6 +154,11 @@ extern void *__vmalloc_node_range(unsigned long size, unsigned long align,
 void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask,
 		int node, const void *caller) __alloc_size(1);
 void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __alloc_size(1);
+void *vmalloc_exec(unsigned long size, unsigned long align) __alloc_size(1);
+void *vcopy_exec(void *dst, void *src, size_t len);
+void vfree_exec(void *addr);
+void *arch_vcopy_exec(void *dst, void *src, size_t len);
+int arch_invalidate_exec(void *ptr, size_t len);
 
 extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2);
 extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2);
mm/nommu.c +12 -0
index 214c70e1d059..8a1317247ef0 100644
@@ -371,6 +371,18 @@ int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages,
 }
 EXPORT_SYMBOL(vm_map_pages_zero);
 
+void *vmalloc_exec(unsigned long size, unsigned long align)
+{
+	return NULL;
+}
+
+void *vcopy_exec(void *dst, void *src, size_t len)
+{
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+void vfree_exec(const void *addr) { }
+
 /*
  *  sys_brk() for the most part doesn't need the global kernel
  *  lock, except when an application is doing something nasty
mm/vmalloc.c +318 -0

336 lines not on this page · raw message

Signature · 1 line
2.30.2
#2 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 3 lines · Song Liu

vmalloc_exec is used to allocate memory to host dynamic kernel text (modules, BPF programs, etc.) with huge pages. This is similar to the proposal by Peter in [1].

This is allg reat but we need to clarify *why* we would go through the trouble. So if folks are not to excited about this series, that's probably why. IMHO it lacks substance for rationale, **and** implies a few gains without any *clear* performance metrics. I have 0 experience with mm so I'd like other's feedback on my this -- I'm just trying to do decipher rationale from prior "bpf prog pack" efforts.

I'm sensing that the cables in messaging are a bit crossed here and we need to provide a bit better full picture for rationale and this is being completely missed and this work is being undersold. If my assessment is accurate though, the bpf prog pack strategy with sharing huge pages may prove useful long term for other things than just modules / ftrace / kprobes.

I was surprised to see this entire patch series upgrade from RFC to proper PATCH form now completely fails to mention any of the original motivations behind the "BPF prog pack", which you are doing a true heroic effort to try to generalize as the problem is hard. Let me try to help with that. The rationale for the old BPF prog pack is documented as follows:

* Most BPF programs are pretty small. Allocating a hole page for each
* program is sometime a waste. Many small bpf program also adds pressure
* to instruction TLB. To solve this issue, we introduce a BPF program pack
* allocator. The prog_pack allocator uses HPAGE_PMD_SIZE page (2MB on x86)
* to host BPF programs.

Previously you have also stated in earlier versions of this patch set:

"Most BPF programs are small, but they consume a page each. For systems
with busy traffic and many BPF programs, this could also add significant pressure to instruction TLB. High iTLB pressure usually causes slow down for the whole system, which includes visible performance degradation for production workloads."

So it is implied here that one of the benefits is to help reduce iTLB misses. But that's it. We have no visible numbers to look at and for what... But reducing iTLB misses doesn't always have a complete direct correlation with improving things, but if the code change is small enough it obviously makes sense to apply. If the change is a bit more intrusive, as in this patch series a bit more rationale should be provided.

Other than the "performance aspects" of your patchset, the *main* reason I am engaged and like it is it reduces the nasty mess of semantics on dealing with special permissions on pages which we see in modules and a few other places which today completely open code it. That proves error prone and I'm glad to see efforts to generalize that nastiness. So please ensure this is added as part of the documented rationale. Even if the iTLB miss ratio improvement is not astronomical I believe that the gains in sanity on improving semantics on special pages and sharing code make it well worthwhile. The iTLB miss ratio improvement is just a small cherry on top.

Going back to performance aspects, when Linus had poked for more details about this your have elaborated further:

"we have seen direct map fragmentation causing visible
performance drop for our major services. This is the shadow production benchmark, so it is not possible to run it out of our data centers. Tracing showed that BPF program was the top trigger of these direct map splits."

And the only other metric we have is:

"For our web service production benchmark, bpf_prog_pack on 4kB pages
gives 0.5% to 0.7% more throughput than not using bpf_prog_pack."

These metrics are completely arbitrary and opaque to us. We need something tangible and reproducible and I have been suggesting that from early on...

I'm under the impression that the real missed, undocumented, major value-add here is that the old "BPF prog pack" strategy helps to reduce the direct map fragmentation caused by heavy use of the eBPF JIT programs and this in turn helps your overall random system performance (regardless of what it is you do). As I see it then the eBPF prog pack is just one strategy to try to mitigate memory fragmentation on the direct map caused by the the eBPF JIT programs, so the "slow down" your team has obvserved should be due to the eventual fragmentation caused on the direct map *while* eBPF programs get heavily used.

Mike Rapoport had presented about the Direct map fragmentation problem at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace / kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance evaluation on whether using 2M/1G pages aggressively for the kernel direct map help performance [1] ends up generally recommending huge pages. The work by Xing though was about using huge pages *alone*, not using a strategy such as in the "bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs, and that I think is the real golden nugget here.

I contend therefore that the theoretical reduction of iTLB misses by using huge pages for "bpf prog pack" is not what gets your systems to perform somehow better. It should be simply that it reduces fragmentation and *this* generally can help with performance long term. If this is accurate then let's please separate the two aspects to this.

There's two aspects to what I would like to see from a performance perspective then actually mentioned in the commit logs:

1) iTLB miss loss ratio with "bpf prog pack" or this generalized solution Vs not using it at all:

I'd tried the below but didn't see much difference:

   perf stat --repeat 10 -e dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-stores-misses,iTLB-loads,iTLB-load-misses,page-faults \
   	--pre 'modprobe -r test_bpf' \-- modprobe test_bpf

This is likely that the system I have might have a huge cache and I was not contending it with another heavy memory intensive workload. So it may be worthy to run something like the above *as* some other memory intensive benchark kicks gear in a loop. Another reason too may be that test_bpf runs tests sequentially instead of in parallel, and so it would be good to get ebpf folks's feedback as to an idea of what other things could be done to really eBPF JIT the hell out of a system. It is why I had suggested long ago maybe a custom new selftest which stresses the hell out of only eBPF JIT and had given an example multithreaded selftest.

If we ever get modules support, I was hoping to see if something like the below (if the fs module .text section does end up shared) *may* have reduce iTLB miss ratio.

   perf stat --repeat 10 -e dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-stores-misses,iTLB-loads,iTLB-load-misses,page-faults \
	--pre 'make -s mrproper defconfig' \-- make -s -j$(nproc) bzImage

Maybe something as simple as systemd-analyze may show time reduction if iTLB miss ratio is reduced by sharing most module code in a huge page.

2) Estimate in reduction on direct map fragmentation by using the "bpf prog pack" or this generalized solution:

For this I'd expect a benchmark similar to the workload you guys run or something memory intensive, as eBPF JITs are heavily used, and after a certain amount of time somehow compute how fragmented memory is. The only sensible thing I can think to measure memory fragmentation is to look at the memory compaction index /sys/kernel/debug/extfrag/extfrag_index , but I highly welcome other's ideas as I'm a mm n00b.

[0] https://lpc.events/event/11/contributions/1127/attachments/922/1792/LPC21%20Direct%20map%20management%20.pdf
[1] https://lore.kernel.org/linux-mm/[email protected]/

mm/vmalloc.c void *vmalloc(unsigned long size) · line 3366

Quoted 8 lines
mm/vmalloc.c +8 -0
+static void move_vmap_to_free_text_tree(void *addr)
+{
+	struct vmap_area *va;
+
+	/* remove from vmap_area_root */
+	spin_lock(&vmap_area_lock);
+	va = __find_vmap_area((unsigned long)addr, &vmap_area_root);
+	if (WARN_ON_ONCE(!va)) {

This seems to be hopeful but we purposely are allowing for the allocated memory to be used elsewhere are we not? Can you trigger the WARN_ON_ONCE() with stress-ng as described in commit 82dd23e84be3e ("mm/vmalloc.c:
preload a CPU with one object for split purpose") while using eBPF JIT or loading/unloading a module in a loop?

Quoted 21 lines
mm/vmalloc.c +21 -0
+void *vmalloc_exec(unsigned long size, unsigned long align)
+{
+	struct vmap_area *va, *tmp;
+	unsigned long addr;
+	enum fit_type type;
+	int ret;
+
+	va = kmem_cache_alloc_node(vmap_area_cachep, GFP_KERNEL, NUMA_NO_NODE);
+	if (unlikely(!va))
+		return NULL;
+
+again:
+	preload_this_cpu_lock(&free_text_area_lock, GFP_KERNEL, NUMA_NO_NODE);
+	tmp = find_vmap_lowest_match(&free_text_area_root, size, align, 1, false);
+
+	if (!tmp) {
+		unsigned long alloc_size;
+		void *ptr;
+
+		spin_unlock(&free_text_area_lock);
+

mm/vmalloc.c void *vmalloc(unsigned long size) · line 3428

Quoted 10 lines
mm/vmalloc.c +10 -0
+		/*
+		 * Not enough continuous space in free_text_area_root, try
+		 * allocate more memory. The memory is first added to
+		 * vmap_area_root, and then moved to free_text_area_root.
+		 */
+		alloc_size = roundup(size, PMD_SIZE * num_online_nodes());
+		ptr = __vmalloc_node_range(alloc_size, PMD_SIZE, VMALLOC_EXEC_START,
+					   VMALLOC_EXEC_END, GFP_KERNEL, PAGE_KERNEL,
+					   VM_ALLOW_HUGE_VMAP | VM_NO_GUARD,
+					   NUMA_NO_NODE, __builtin_return_address(0));

I thought Peter had suggested keeping the guard?

Once you get modules going you may want to update the comment about modules on __vmalloc_node_range().

Luis

#3 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Hi Luis,

Thanks for looping me in.

Quoted 74 lines · Luis Chamberlain
Quoted 3 lines · Song Liu

vmalloc_exec is used to allocate memory to host dynamic kernel text (modules, BPF programs, etc.) with huge pages. This is similar to the proposal by Peter in [1].

This is allg reat but we need to clarify *why* we would go through the trouble. So if folks are not to excited about this series, that's probably why. IMHO it lacks substance for rationale, **and** implies a few gains without any *clear* performance metrics. I have 0 experience with mm so I'd like other's feedback on my this -- I'm just trying to do decipher rationale from prior "bpf prog pack" efforts.

I'm sensing that the cables in messaging are a bit crossed here and we need to provide a bit better full picture for rationale and this is being completely missed and this work is being undersold. If my assessment is accurate though, the bpf prog pack strategy with sharing huge pages may prove useful long term for other things than just modules / ftrace / kprobes.

I was surprised to see this entire patch series upgrade from RFC to proper PATCH form now completely fails to mention any of the original motivations behind the "BPF prog pack", which you are doing a true heroic effort to try to generalize as the problem is hard. Let me try to help with that. The rationale for the old BPF prog pack is documented as follows:

* Most BPF programs are pretty small. Allocating a hole page for each
* program is sometime a waste. Many small bpf program also adds pressure
* to instruction TLB. To solve this issue, we introduce a BPF program pack
* allocator. The prog_pack allocator uses HPAGE_PMD_SIZE page (2MB on x86)
* to host BPF programs.

Previously you have also stated in earlier versions of this patch set:

"Most BPF programs are small, but they consume a page each. For systems
with busy traffic and many BPF programs, this could also add significant pressure to instruction TLB. High iTLB pressure usually causes slow down for the whole system, which includes visible performance degradation for production workloads."

So it is implied here that one of the benefits is to help reduce iTLB misses. But that's it. We have no visible numbers to look at and for what... But reducing iTLB misses doesn't always have a complete direct correlation with improving things, but if the code change is small enough it obviously makes sense to apply. If the change is a bit more intrusive, as in this patch series a bit more rationale should be provided.

Other than the "performance aspects" of your patchset, the *main* reason I am engaged and like it is it reduces the nasty mess of semantics on dealing with special permissions on pages which we see in modules and a few other places which today completely open code it. That proves error prone and I'm glad to see efforts to generalize that nastiness. So please ensure this is added as part of the documented rationale. Even if the iTLB miss ratio improvement is not astronomical I believe that the gains in sanity on improving semantics on special pages and sharing code make it well worthwhile. The iTLB miss ratio improvement is just a small cherry on top.

Going back to performance aspects, when Linus had poked for more details about this your have elaborated further:

"we have seen direct map fragmentation causing visible
performance drop for our major services. This is the shadow production benchmark, so it is not possible to run it out of our data centers. Tracing showed that BPF program was the top trigger of these direct map splits."

And the only other metric we have is:

"For our web service production benchmark, bpf_prog_pack on 4kB pages
gives 0.5% to 0.7% more throughput than not using bpf_prog_pack."

These metrics are completely arbitrary and opaque to us. We need something tangible and reproducible and I have been suggesting that from early on...

Quoted 9 lines

I'm under the impression that the real missed, undocumented, major value-add here is that the old "BPF prog pack" strategy helps to reduce the direct map fragmentation caused by heavy use of the eBPF JIT programs and this in turn helps your overall random system performance (regardless of what it is you do). As I see it then the eBPF prog pack is just one strategy to try to mitigate memory fragmentation on the direct map caused by the the eBPF JIT programs, so the "slow down" your team has obvserved should be due to the eventual fragmentation caused on the direct map *while* eBPF programs get heavily used.

I believe that while the eBPF prog pack is helpful in mitigation of the direct map fragmentation caused by the eBPF JIT programs, the same strategy of allocating a large page, splitting its PMD entry and then reusing the memory for smaller allocations can be (and should be) generalized to other use cases that require non-default permissions in the page table. Most prominent use cases are those that allocate memory for code, but the same approach is relevant for other cases, like secretmem or page table protection with PKS.

A while ago I've suggested to handle such caching of large pages at the page allocator level, but when we discussed it at LSF/MM/BPF, prevailing opinion was that added value does not justify changes to the page allocator and it was suggested to handle such caching elsewhere.

I had to put this project on a backburner for $VARIOUS_REASONS, but I still think that we need a generic allocator for memory with non-default permissions in the direct map and that code allocation should build on that allocator.

All that said, the direct map fragmentation problem is currently relevant only to x86 because it's the only architecture that supports splitting of the large pages in the direct map.

Quoted 9 lines

Mike Rapoport had presented about the Direct map fragmentation problem at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace / kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance evaluation on whether using 2M/1G pages aggressively for the kernel direct map help performance [1] ends up generally recommending huge pages. The work by Xing though was about using huge pages *alone*, not using a strategy such as in the "bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs, and that I think is the real golden nugget here.

Quoted 5 lines

I contend therefore that the theoretical reduction of iTLB misses by using huge pages for "bpf prog pack" is not what gets your systems to perform somehow better. It should be simply that it reduces fragmentation and *this* generally can help with performance long term. If this is accurate then let's please separate the two aspects to this.

The direct map fragmentation is the reason for higher TLB miss rate, both for iTLB and dTLB. Whenever a large page in the direct map is split, all kernel accesses via the direct map will use small pages which requires dealing with 512 page table entries instead of one for 2M range.

Since small pages in the direct map are never collapsed back to large pages, long living system that heavily uses eBPF programs will have its direct map severely fragmented, higher TLB miss rate and worse overall performance.

Quoted 5 lines

There's two aspects to what I would like to see from a performance perspective then actually mentioned in the commit logs:

1) iTLB miss loss ratio with "bpf prog pack" or this generalized solution Vs not using it at all:

...

Quoted 10 lines

2) Estimate in reduction on direct map fragmentation by using the "bpf prog pack" or this generalized solution:

For this I'd expect a benchmark similar to the workload you guys run or something memory intensive, as eBPF JITs are heavily used, and after a certain amount of time somehow compute how fragmented memory is. The only sensible thing I can think to measure memory fragmentation is to look at the memory compaction index /sys/kernel/debug/extfrag/extfrag_index , but I highly welcome other's ideas as I'm a mm n00b.

The direct map fragmentation can be tracked with

grep DirectMap /proc/meminfo
grep direct_map /proc/vmstat

and by looking at /sys/kernel/debug/page_tables/kernel

Quoted 4 lines

[0] https://lpc.events/event/11/contributions/1127/attachments/922/1792/LPC21%20Direct%20map%20management%20.pdf
[1] https://lore.kernel.org/linux-mm/[email protected]/

Luis

Signature · 2 lines
Sincerely yours,
Mike.
#4 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 98 lines · Mike Rapoport

Hi Luis,

Thanks for looping me in.

Quoted 74 lines · Luis Chamberlain
Quoted 3 lines · Song Liu

vmalloc_exec is used to allocate memory to host dynamic kernel text (modules, BPF programs, etc.) with huge pages. This is similar to the proposal by Peter in [1].

This is allg reat but we need to clarify *why* we would go through the trouble. So if folks are not to excited about this series, that's probably why. IMHO it lacks substance for rationale, **and** implies a few gains without any *clear* performance metrics. I have 0 experience with mm so I'd like other's feedback on my this -- I'm just trying to do decipher rationale from prior "bpf prog pack" efforts.

I'm sensing that the cables in messaging are a bit crossed here and we need to provide a bit better full picture for rationale and this is being completely missed and this work is being undersold. If my assessment is accurate though, the bpf prog pack strategy with sharing huge pages may prove useful long term for other things than just modules / ftrace / kprobes.

I was surprised to see this entire patch series upgrade from RFC to proper PATCH form now completely fails to mention any of the original motivations behind the "BPF prog pack", which you are doing a true heroic effort to try to generalize as the problem is hard. Let me try to help with that. The rationale for the old BPF prog pack is documented as follows:

* Most BPF programs are pretty small. Allocating a hole page for each
* program is sometime a waste. Many small bpf program also adds pressure
* to instruction TLB. To solve this issue, we introduce a BPF program pack
* allocator. The prog_pack allocator uses HPAGE_PMD_SIZE page (2MB on x86)
* to host BPF programs.

Previously you have also stated in earlier versions of this patch set:

"Most BPF programs are small, but they consume a page each. For systems
with busy traffic and many BPF programs, this could also add significant pressure to instruction TLB. High iTLB pressure usually causes slow down for the whole system, which includes visible performance degradation for production workloads."

So it is implied here that one of the benefits is to help reduce iTLB misses. But that's it. We have no visible numbers to look at and for what... But reducing iTLB misses doesn't always have a complete direct correlation with improving things, but if the code change is small enough it obviously makes sense to apply. If the change is a bit more intrusive, as in this patch series a bit more rationale should be provided.

Other than the "performance aspects" of your patchset, the *main* reason I am engaged and like it is it reduces the nasty mess of semantics on dealing with special permissions on pages which we see in modules and a few other places which today completely open code it. That proves error prone and I'm glad to see efforts to generalize that nastiness. So please ensure this is added as part of the documented rationale. Even if the iTLB miss ratio improvement is not astronomical I believe that the gains in sanity on improving semantics on special pages and sharing code make it well worthwhile. The iTLB miss ratio improvement is just a small cherry on top.

Going back to performance aspects, when Linus had poked for more details about this your have elaborated further:

"we have seen direct map fragmentation causing visible
performance drop for our major services. This is the shadow production benchmark, so it is not possible to run it out of our data centers. Tracing showed that BPF program was the top trigger of these direct map splits."

And the only other metric we have is:

"For our web service production benchmark, bpf_prog_pack on 4kB pages
gives 0.5% to 0.7% more throughput than not using bpf_prog_pack."

These metrics are completely arbitrary and opaque to us. We need something tangible and reproducible and I have been suggesting that from early on...

Quoted 9 lines

I'm under the impression that the real missed, undocumented, major value-add here is that the old "BPF prog pack" strategy helps to reduce the direct map fragmentation caused by heavy use of the eBPF JIT programs and this in turn helps your overall random system performance (regardless of what it is you do). As I see it then the eBPF prog pack is just one strategy to try to mitigate memory fragmentation on the direct map caused by the the eBPF JIT programs, so the "slow down" your team has obvserved should be due to the eventual fragmentation caused on the direct map *while* eBPF programs get heavily used.

I believe that while the eBPF prog pack is helpful in mitigation of the direct map fragmentation caused by the eBPF JIT programs, the same strategy of allocating a large page, splitting its PMD entry and then reusing the memory for smaller allocations can be (and should be) generalized to other use cases that require non-default permissions in the page table. Most prominent use cases are those that allocate memory for code, but the same approach is relevant for other cases, like secretmem or page table protection with PKS.

Quoted 4 lines

A while ago I've suggested to handle such caching of large pages at the page allocator level, but when we discussed it at LSF/MM/BPF, prevailing opinion was that added value does not justify changes to the page allocator and it was suggested to handle such caching elsewhere.

I saw that on the lwn coverage.

Quoted 4 lines

I had to put this project on a backburner for $VARIOUS_REASONS, but I still think that we need a generic allocator for memory with non-default permissions in the direct map and that code allocation should build on that allocator.

It seems this generalization of the bpf prog pack to possibly be used for modules / kprobes / ftrace is a small step in that direction.

Quoted 3 lines

All that said, the direct map fragmentation problem is currently relevant only to x86 because it's the only architecture that supports splitting of the large pages in the direct map.

I was thinking even more long term too, using this as a proof of concept. If this practice in general helps with fragmentation, could it be used for experimetnation with compound pages later, as a way to reduce possible fragmentation.

Quoted 9 lines
Quoted 9 lines

Mike Rapoport had presented about the Direct map fragmentation problem at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace / kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance evaluation on whether using 2M/1G pages aggressively for the kernel direct map help performance [1] ends up generally recommending huge pages. The work by Xing though was about using huge pages *alone*, not using a strategy such as in the "bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs, and that I think is the real golden nugget here.

Quoted 8 lines
Quoted 5 lines

I contend therefore that the theoretical reduction of iTLB misses by using huge pages for "bpf prog pack" is not what gets your systems to perform somehow better. It should be simply that it reduces fragmentation and *this* generally can help with performance long term. If this is accurate then let's please separate the two aspects to this.

The direct map fragmentation is the reason for higher TLB miss rate, both for iTLB and dTLB.

OK so then whatever benchmark is running in tandem as eBPF JIT is hammered should *also* be measured with perf for iTLB and dTLB. ie, the patch can provide such results as a justifications.

Quoted 8 lines

Whenever a large page in the direct map is split, all kernel accesses via the direct map will use small pages which requires dealing with 512 page table entries instead of one for 2M range.

Since small pages in the direct map are never collapsed back to large pages, long living system that heavily uses eBPF programs will have its direct map severely fragmented, higher TLB miss rate and worse overall performance.

Shouldn't compaction help with those situations?

Quoted 19 lines
Quoted 5 lines

There's two aspects to what I would like to see from a performance perspective then actually mentioned in the commit logs:

1) iTLB miss loss ratio with "bpf prog pack" or this generalized solution Vs not using it at all:

...

Quoted 10 lines

2) Estimate in reduction on direct map fragmentation by using the "bpf prog pack" or this generalized solution:

For this I'd expect a benchmark similar to the workload you guys run or something memory intensive, as eBPF JITs are heavily used, and after a certain amount of time somehow compute how fragmented memory is. The only sensible thing I can think to measure memory fragmentation is to look at the memory compaction index /sys/kernel/debug/extfrag/extfrag_index , but I highly welcome other's ideas as I'm a mm n00b.

Quoted 6 lines

The direct map fragmentation can be tracked with

grep DirectMap /proc/meminfo
grep direct_map /proc/vmstat

and by looking at /sys/kernel/debug/page_tables/kernel

Thanks!

Luis

#5 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 30 lines · Luis Chamberlain
Quoted 26 lines
Quoted 16 lines

Mike Rapoport had presented about the Direct map fragmentation problem
at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace /
kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance
evaluation on whether using 2M/1G pages aggressively for the kernel direct map
help performance [1] ends up generally recommending huge pages. The work by Xing
though was about using huge pages *alone*, not using a strategy such as in the
"bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs,
and that I think is the real golden nugget here.

Quoted 9 lines

I contend therefore that the theoretical reduction of iTLB misses by using
huge pages for "bpf prog pack" is not what gets your systems to perform
somehow better. It should be simply that it reduces fragmentation and
*this* generally can help with performance long term. If this is accurate
then let's please separate the two aspects to this.

Quoted 3 lines

The direct map fragmentation is the reason for higher TLB miss rate, both
for iTLB and dTLB.

Quoted 5 lines

OK so then whatever benchmark is running in tandem as eBPF JIT is hammered
should *also* be measured with perf for iTLB and dTLB. ie, the patch can
provide such results as a justifications.

Song had done some tests on the old prog pack version that to me seemed to indicate most (or possibly all) of the benefit was direct map fragmentation reduction. This was surprised me, since 2MB kernel text has shown to be beneficial.

Otherwise +1 to all these comments. This should be clear about what the benefits are. I would add, that this is also much nicer about TLB shootdowns than the existing way of loading text and saves some memory.

So I think there are sort of four areas of improvements:
1. Direct map fragmentation reduction (dTLB miss improvements). This sort of does it as a side effect in this series, and the solution Mike is talking about is a more general, probably better one.
2. 2MB mapped JITs. This is the iTLB side. I think this is a decent solution for this, but surprisingly it doesn't seem to be useful for JITs. (modules testing TBD)
3. Loading text to reused allocation with per-cpu mappings. This reduces TLB shootdowns, which are a short term load and teardown time performance drag. My understanding is this is more of a problem on bigger systems with many CPUs. This series does a decent job at this, but the solution is not compatible with modules. Maybe ok since modules don't load as often as JITs.
4. Having BPF progs share pages. This saves memory. This series could probably easily get a number for how much.

author reviewer
#6 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 47 lines · Rick P
Quoted 30 lines · Luis Chamberlain
Quoted 26 lines
Quoted 16 lines

Mike Rapoport had presented about the Direct map fragmentation problem
at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace /
kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance
evaluation on whether using 2M/1G pages aggressively for the kernel direct map
help performance [1] ends up generally recommending huge pages. The work by Xing
though was about using huge pages *alone*, not using a strategy such as in the
"bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs,
and that I think is the real golden nugget here.

Quoted 9 lines

I contend therefore that the theoretical reduction of iTLB misses by using
huge pages for "bpf prog pack" is not what gets your systems to perform
somehow better. It should be simply that it reduces fragmentation and
*this* generally can help with performance long term. If this is accurate
then let's please separate the two aspects to this.

Quoted 3 lines

The direct map fragmentation is the reason for higher TLB miss rate, both
for iTLB and dTLB.

Quoted 5 lines

OK so then whatever benchmark is running in tandem as eBPF JIT is hammered
should *also* be measured with perf for iTLB and dTLB. ie, the patch can
provide such results as a justifications.

Song had done some tests on the old prog pack version that to me seemed to indicate most (or possibly all) of the benefit was direct map fragmentation reduction. This was surprised me, since 2MB kernel text has shown to be beneficial.

Otherwise +1 to all these comments. This should be clear about what the benefits are. I would add, that this is also much nicer about TLB shootdowns than the existing way of loading text and saves some memory.

Quoted 16 lines

So I think there are sort of four areas of improvements:
1. Direct map fragmentation reduction (dTLB miss improvements). This sort of does it as a side effect in this series, and the solution Mike is talking about is a more general, probably better one.
2. 2MB mapped JITs. This is the iTLB side. I think this is a decent solution for this, but surprisingly it doesn't seem to be useful for JITs. (modules testing TBD)
3. Loading text to reused allocation with per-cpu mappings. This reduces TLB shootdowns, which are a short term load and teardown time performance drag. My understanding is this is more of a problem on bigger systems with many CPUs. This series does a decent job at this, but the solution is not compatible with modules. Maybe ok since modules don't load as often as JITs.
4. Having BPF progs share pages. This saves memory. This series could probably easily get a number for how much.

Hi Luis, Rick, and Mike,

Thanks a lot for helping me organize this information. Totally agree with all these comments. I will add more data to the next revision.

Besides the motivation improvement, could you please also share your comments on:
1. The logic/design of the vmalloc_exec() et. al. APIs;
2. The naming of these functions. Does execmem_[alloc|free|fill|cpy] (as suggested by Chritoph) sound good?

Thanks,
Song

#7 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 5 lines · Song Liu

On Thu, Nov 3, 2022 at 2:19 PM Edgecombe, Rick P <[email protected]> wrote:
Besides the motivation improvement, could you please also share your comments on:
1. The logic/design of the vmalloc_exec() et. al. APIs;

I've provided the feedback that I can so far as I'm new to mm. Best I can do is provided a better rationale so that mm folks can understand the motivation.

Quoted 2 lines

2. The naming of these functions. Does execmem_[alloc|free|fill|cpy] (as suggested by Chritoph) sound good?

That seems sensible.

There may be other users later, secmm_alloc() too then later, for instance. So we just gotta keep that in mind.

Luis

#8 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

In reply to Edgecombe, Rick P
Quoted 37 lines · Rick P
Quoted 30 lines · Luis Chamberlain
Quoted 26 lines
Quoted 16 lines

Mike Rapoport had presented about the Direct map fragmentation problem
at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace /
kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance
evaluation on whether using 2M/1G pages aggressively for the kernel direct map
help performance [1] ends up generally recommending huge pages. The work by Xing
though was about using huge pages *alone*, not using a strategy such as in the
"bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs,
and that I think is the real golden nugget here.

Quoted 9 lines

I contend therefore that the theoretical reduction of iTLB misses by using
huge pages for "bpf prog pack" is not what gets your systems to perform
somehow better. It should be simply that it reduces fragmentation and
*this* generally can help with performance long term. If this is accurate
then let's please separate the two aspects to this.

Quoted 3 lines

The direct map fragmentation is the reason for higher TLB miss rate, both
for iTLB and dTLB.

Quoted 5 lines

OK so then whatever benchmark is running in tandem as eBPF JIT is hammered
should *also* be measured with perf for iTLB and dTLB. ie, the patch can
provide such results as a justifications.

Quoted 3 lines

Song had done some tests on the old prog pack version that to me seemed to indicate most (or possibly all) of the benefit was direct map fragmentation reduction.

Matches my observations but I also provided quite a bit of hints as to *why* I think that is. I suggested lib/test_kmod.c as an example beefy multithreaded selftests which really kicks the hell out of the kernel with whatever crap you want to run. That is precicely how I uncovered some odd kmod bug lingering for years.

Quoted 9 lines

This was surprised me, since 2MB kernel text
has shown to be beneficial.

Otherwise +1 to all these comments. This should be clear about what the benefits are. I would add, that this is also much nicer about TLB shootdowns than the existing way of loading text and saves some memory.

So I think there are sort of four areas of improvements:
1. Direct map fragmentation reduction (dTLB miss improvements).

The dTLB gains should be on the benchmark which runs in tandem to the ebpf-JIT-monster-selftest, not on the ebpf-JIT-monster-selftest, right?

Quoted 6 lines

This
sort of does it as a side effect in this series, and the solution Mike is talking about is a more general, probably better one.
2. 2MB mapped JITs. This is the iTLB side. I think this is a decent solution for this, but surprisingly it doesn't seem to be useful for JITs. (modules testing TBD)

Yes I'm super eager to get this tested. In fact I wonder if one can boot Linux with less memory too...

Quoted 6 lines

3. Loading text to reused allocation with per-cpu mappings. This reduces TLB shootdowns, which are a short term load and teardown time performance drag. My understanding is this is more of a problem on bigger systems with many CPUs. This series does a decent job at this, but the solution is not compatible with modules. Maybe ok since modules don't load as often as JITs.

There are some tests like fstests which make heavy use of module removal. But as a side effect, indeed I like to reboot to have a fresh system before running fstests. I guess fstests should run with a heavily fragmented memory too as a side corner case thing.

Quoted 2 lines

4. Having BPF progs share pages. This saves memory. This series could probably easily get a number for how much.

Once that does hit modules / kprobes / ftrace, the impact is much much greater obviously.

Luis

#9 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 42 lines · Luis Chamberlain
Quoted 37 lines · Rick P
Quoted 30 lines · Luis Chamberlain
Quoted 26 lines
Quoted 16 lines

Mike Rapoport had presented about the Direct map fragmentation problem
at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace /
kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance
evaluation on whether using 2M/1G pages aggressively for the kernel direct map
help performance [1] ends up generally recommending huge pages. The work by Xing
though was about using huge pages *alone*, not using a strategy such as in the
"bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs,
and that I think is the real golden nugget here.

Quoted 9 lines

I contend therefore that the theoretical reduction of iTLB misses by using
huge pages for "bpf prog pack" is not what gets your systems to perform
somehow better. It should be simply that it reduces fragmentation and
*this* generally can help with performance long term. If this is accurate
then let's please separate the two aspects to this.

Quoted 3 lines

The direct map fragmentation is the reason for higher TLB miss rate, both
for iTLB and dTLB.

Quoted 5 lines

OK so then whatever benchmark is running in tandem as eBPF JIT is hammered
should *also* be measured with perf for iTLB and dTLB. ie, the patch can
provide such results as a justifications.

Quoted 3 lines

Song had done some tests on the old prog pack version that to me seemed to indicate most (or possibly all) of the benefit was direct map fragmentation reduction.

Quoted 5 lines

Matches my observations but I also provided quite a bit of hints as to *why* I think that is. I suggested lib/test_kmod.c as an example beefy multithreaded selftests which really kicks the hell out of the kernel with whatever crap you want to run. That is precicely how I uncovered some odd kmod bug lingering for years.

*and*, *perhaps*... it may be that you need another memory intensive benchmark to run in tandem, one which mimics the behaviour of the internal "shadow production benchmark", whatever that is.

Luis

#10 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

In reply to Luis Chamberlain
Quoted 10 lines · Luis Chamberlain
Quoted 4 lines · Mike Rapoport

I had to put this project on a backburner for $VARIOUS_REASONS, but I still think that we need a generic allocator for memory with non-default permissions in the direct map and that code allocation should build on that allocator.

It seems this generalization of the bpf prog pack to possibly be used for modules / kprobes / ftrace is a small step in that direction.

Quoted 8 lines
Quoted 3 lines

All that said, the direct map fragmentation problem is currently relevant only to x86 because it's the only architecture that supports splitting of the large pages in the direct map.

I was thinking even more long term too, using this as a proof of concept. If this practice in general helps with fragmentation, could it be used for experimetnation with compound pages later, as a way to reduce possible fragmentation.

As Rick already mentioned, these patches help with the direct map fragmentation only indirectly. With these patches memory is freed in PMD_SIZE chunks and this makes the changes to the direct map in vm_remove_mappings() to happen in in PMD_SIZE units and this is pretty much the only effect of this series on the direct map layout.

A bit unrelated, but I'm wondering now if we want to have the direct map alias of the pages allocated for code also to be read-only...

Quoted 10 lines
Quoted 8 lines

Whenever a large page in the direct map is split, all kernel accesses via the direct map will use small pages which requires dealing with 512 page table entries instead of one for 2M range.

Since small pages in the direct map are never collapsed back to large pages, long living system that heavily uses eBPF programs will have its direct map severely fragmented, higher TLB miss rate and worse overall performance.

Shouldn't compaction help with those situations?

Compaction helps to reduce fragmentation of the physical memory, it tries to bring free physical pages next to each other to create large contiguous chunks, but it does not change the virtual addresses the users of the underlying data see.

Changing permissions of a small page in the direct map causes "discontinuity" in the virtual space. E.g. if we have 2M mapped RW with a single PMD changing several page in the middle of those 2M to R+X will require to remap that range with 512 PTEs.

Quoted 3 lines

Thanks!

Luis

Signature · 2 lines
Sincerely yours,
Mike.
#11 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 20 lines · Mike Rapoport
Quoted 10 lines · Luis Chamberlain
Quoted 4 lines · Mike Rapoport

I had to put this project on a backburner for $VARIOUS_REASONS, but I still think that we need a generic allocator for memory with non-default permissions in the direct map and that code allocation should build on that allocator.

It seems this generalization of the bpf prog pack to possibly be used for modules / kprobes / ftrace is a small step in that direction.

Quoted 8 lines
Quoted 3 lines

All that said, the direct map fragmentation problem is currently relevant only to x86 because it's the only architecture that supports splitting of the large pages in the direct map.

I was thinking even more long term too, using this as a proof of concept. If this practice in general helps with fragmentation, could it be used for experimetnation with compound pages later, as a way to reduce possible fragmentation.

Quoted 5 lines

As Rick already mentioned, these patches help with the direct map fragmentation only indirectly. With these patches memory is freed in PMD_SIZE chunks and this makes the changes to the direct map in vm_remove_mappings() to happen in in PMD_SIZE units and this is pretty much the only effect of this series on the direct map layout.

I understand that is what *this* series does. I was wondering is similar scheme may be useful to study to see if it helps with aggregating say something like 32 x 64 kb for compound page allocations of order 16 (64 Kib) to see if it may help with possible fragmentation concerns for that world where that may be useful in the future (completely unrelated to page permissions).

Quoted 12 lines

A bit unrelated, but I'm wondering now if we want to have the direct map alias of the pages allocated for code also to be read-only...

Quoted 9 lines
Quoted 8 lines

Whenever a large page in the direct map is split, all kernel accesses via the direct map will use small pages which requires dealing with 512 page table entries instead of one for 2M range.

Since small pages in the direct map are never collapsed back to large pages, long living system that heavily uses eBPF programs will have its direct map severely fragmented, higher TLB miss rate and worse overall performance.

Quoted 6 lines
Quoted 1 line

Shouldn't compaction help with those situations?

Compaction helps to reduce fragmentation of the physical memory, it tries to bring free physical pages next to each other to create large contiguous chunks, but it does not change the virtual addresses the users of the underlying data see.

Sorry I understood that 'bpf prog pack' only only used *one* 2 MiB huge page for *all* eBPF JIT programs, and so the fragmentation issue prior to 'bpf prog pack' I thought was the fact that as eBPF programs are still alive, we have fragmentation. In the world without 'bpf prog pack' I thought no huge pages were used.

Quoted 4 lines

Changing permissions of a small page in the direct map causes "discontinuity" in the virtual space. E.g. if we have 2M mapped RW with a single PMD changing several page in the middle of those 2M to R+X will require to remap that range with 512 PTEs.

Makes sense, thanks.

Luis

#12 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

In reply to Luis Chamberlain

Hello,

On Wed, Nov 02, 2022 at 04:41:59PM -0700, Luis Chamberlain wrote:

... ...

Quoted 10 lines

I'm under the impression that the real missed, undocumented, major value-add here is that the old "BPF prog pack" strategy helps to reduce the direct map fragmentation caused by heavy use of the eBPF JIT programs and this in turn helps your overall random system performance (regardless of what it is you do). As I see it then the eBPF prog pack is just one strategy to try to mitigate memory fragmentation on the direct map caused by the the eBPF JIT programs, so the "slow down" your team has obvserved should be due to the eventual fragmentation caused on the direct map *while* eBPF programs get heavily used.

Quoted 8 lines

Mike Rapoport had presented about the Direct map fragmentation problem at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace / kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance evaluation on whether using 2M/1G pages aggressively for the kernel direct map help performance [1] ends up generally recommending huge pages. The work by Xing though was about using huge pages *alone*, not using a strategy such as in the "bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs, and that I think is the real golden nugget here.

I'm interested in how this patchset (further) improves direct map fragmentation so would like to evaluate it to see if my previous work to merge small mappings back in architecture layer[1] is still necessary.

I tried to apply this patchset on v6.1-rc3/2/1 and v6.0 but all failed, so I took one step back and evaluated the existing bpf_prog_pack. I'm aware of this patchset would make things even better by using order-9 page to backup the vmalloced range.

I used the sample bpf prog: sample/bpf/sockex1 because it looks easy to run, feel free to let me know a better way to evaluate this.

- In kernels before bpf_prog_pack(v5.17 and earlier), this prog would cause 3 pages to change protection to RO+X from RW+NX; And if the three pages are far apart, each would cause a level 3 split then a level 2 split; Reality is, allocated pages tend to stay close physically so actual result will not be this bad.

- After bpf_prog_pack, the load of this prog will most likely requires no new page protection change as long as the existing pack pool has space for it(the common case). The actual space required for this bpf prog that needs special protection is: 64 * 2 + 192 bytes, it would need 6144 such progs to use up the cache and trigger another 2MB alloc which can potentially cause a direct map split. 6144 seems to be pretty large number to me so I think the direct map split due to bpf is greatly improved (if not totally solved).

Here are test results on a 8G x86_64 VM.
(on x86_64, 4k is PTE mapping, 2M is PMD mapping and 1G is PUD mapping)
- v5.17
1) right after boot

$ grep Direct /proc/meminfo
DirectMap4k:       87900 kB
DirectMap2M:     5154816 kB
DirectMap1G:     5242880 kB

2) after running 512 sockex1 instances concurrently

$ grep Direct /proc/meminfo
DirectMap4k:      462684 kB
DirectMap2M:     4780032 kB
DirectMap1G:     5242880 kB
PUD mapping survived, some PMD mappings are splitted.

3) after running 1024 sockex1 instances concurrently

$ grep Direct /proc/meminfo
DirectMap4k:      884572 kB
DirectMap2M:     6455296 kB
DirectMap1G:     3145728 kB
2 PUD mappings and some PMD mappings splitted.

4) after run 2048 sockex1 instances concurrently

$ grep Direct /proc/meminfo
DirectMap4k:     1654620 kB
DirectMap2M:     6733824 kB
DirectMap1G:     2097152 kB
Another PUD mapping and some PMD mappings spliited.
At the end, 2 PUD mappings survived.

- v6.1-rc3
The direct map number doesn't change for "right after boot", "after running 512/1024/2048" sockex1 instances. I also tried running 5120 instances but it would consume all available memory when it runs about 4000 instances and even then, the direct map number doesn't change, i.e. not a single split happened. This is understandable because as I calculated above, it would need 6144 such progs to cause another alloc/split. Here is its number:

$ grep Direct /proc/meminfo
DirectMap4k:       22364 kB
DirectMap2M:     3123200 kB
DirectMap1G:     7340032 kB

Consider that production system will have memory mostly consumed and when CPU allocates pages, the page can be far apart than systems right after boot, causing more mapping split, so I also tested to firstly consume all memory to page cache by reading some sparse files and then run this test. I expect this time v5.17 would become worse. Here it is:

- v5.17
1) right after boot

$ grep Direct /proc/meminfo
DirectMap4k:       94044 kB
DirectMap2M:     4100096 kB
DirectMap1G:     6291456 kB
More mappings are in PUD for this boot.

2) after run 512 sockex1 instances concurrently

$ grep Direct /proc/meminfo
DirectMap4k:      538460 kB
DirectMap2M:     7849984 kB
DirectMap1G:     2097152 kB
4 PUD mappings and some PMD mappings are splitted this time, more than
last time.

3) after run 1024 sockex1 instances concurrently

$ grep Direct /proc/meminfo
DirectMap4k:     1083228 kB
DirectMap2M:     7305216 kB
DirectMap1G:     2097152 kB
Some PMD mappings split.

4) after running 2048 sockex1 instances concurrently

$ grep Direct /proc/meminfo
DirectMap4k:     2340700 kB
DirectMap2M:     6047744 kB
DirectMap1G:     2097152 kB
The end result is about the same as before.

- v6.1-rc3
There is no difference because I can't trigger another pack alloc before system is OOMed.

Conclusion: I think bpf_prog_pack is very good at reducing direct map fragmentation and this patchset can further improve this situation on large machines(with huge amount of memory) or with more large bpf progs loaded etc.

Some imperfect things I can think of are(not related to this patchset):
1 Once a split happened, it remains happened. This may not be a big deal now with bpf_prog_pack and this patchset because the need to allocate a new order-9 page and thus cause a potential split should happen much much less;
2 When a new order-9 page has to be allocated, there is no way to tell the allocator to allocate this order-9 page from an already splitted PUD range to avoid another PUD mapping split;
3 As Mike and others have mentioned, there are other users that can also cause direct map split.

[1]: https://lore.kernel.org/lkml/[email protected]/

Regards,
Aaron

#13 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 25 lines · Aaron Lu

Hello,

On Wed, Nov 02, 2022 at 04:41:59PM -0700, Luis Chamberlain wrote:

... ...

Quoted 10 lines

I'm under the impression that the real missed, undocumented, major value-add here is that the old "BPF prog pack" strategy helps to reduce the direct map fragmentation caused by heavy use of the eBPF JIT programs and this in turn helps your overall random system performance (regardless of what it is you do). As I see it then the eBPF prog pack is just one strategy to try to mitigate memory fragmentation on the direct map caused by the the eBPF JIT programs, so the "slow down" your team has obvserved should be due to the eventual fragmentation caused on the direct map *while* eBPF programs get heavily used.

Quoted 8 lines

Mike Rapoport had presented about the Direct map fragmentation problem at Plumbers 2021 [0], and clearly mentioned modules / BPF / ftrace / kprobes as possible sources for this. Then Xing Zhengjun's 2021 performance evaluation on whether using 2M/1G pages aggressively for the kernel direct map help performance [1] ends up generally recommending huge pages. The work by Xing though was about using huge pages *alone*, not using a strategy such as in the "bpf prog pack" to share one 2 MiB huge page for *all* small eBPF programs, and that I think is the real golden nugget here.

Quoted 3 lines

I'm interested in how this patchset (further) improves direct map fragmentation so would like to evaluate it to see if my previous work to merge small mappings back in architecture layer[1] is still necessary.

You gotta apply it to 6.0.5 which had a large change go in for eBPF which was not present on 6.0.

Quoted 4 lines

Conclusion: I think bpf_prog_pack is very good at reducing direct map fragmentation and this patchset can further improve this situation on large machines(with huge amount of memory) or with more large bpf progs loaded etc.

Fantastic. Thanks for the analysis, so yet another set of metrics which I'd hope can be applied to this patch set as this effort is generalized.

Now imagine the effort in cosnideration also of modules / ftrace / kprobes.

Quoted 5 lines

Some imperfect things I can think of are(not related to this patchset):
1 Once a split happened, it remains happened. This may not be a big deal now with bpf_prog_pack and this patchset because the need to allocate a new order-9 page and thus cause a potential split should happen much much less;

Not sure I follow, are you suggesting a order-9 (512 bytes) allocation would trigger a split of the reserved say 2 MiB huge page?

Quoted 5 lines

2 When a new order-9 page has to be allocated, there is no way to tell the allocator to allocate this order-9 page from an already splitted PUD range to avoid another PUD mapping split;
3 As Mike and others have mentioned, there are other users that can also cause direct map split.

Hence the effort to generalize.

Luis

author reviewer
#14 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

Quoted 10 lines · Luis Chamberlain
Quoted 7 lines

Some imperfect things I can think of are(not related to this patchset):
1 Once a split happened, it remains happened. This may not be a big deal now with bpf_prog_pack and this patchset because the need to allocate a new order-9 page and thus cause a potential split should happen much much less;

Not sure I follow, are you suggesting a order-9 (512 bytes) allocation would trigger a split of the reserved say 2 MiB huge page?

I think by order-9 allocation, Aaron meant 2MiB huge pages. The issue is that direct map split is one-way operation. If we set_memory_x() on one 4kB page out of a 1GB direct map, we will split it into 511x 2MiB pages and 512x 4kB pages. There is currently no way to regroup the 1GB page after set_memory_nx() on the page.

Thanks,
Song

Quoted 10 lines
Quoted 6 lines

2 When a new order-9 page has to be allocated, there is no way to tell the allocator to allocate this order-9 page from an already splitted PUD range to avoid another PUD mapping split;
3 As Mike and others have mentioned, there are other users that can also cause direct map split.

Hence the effort to generalize.

Luis

author reviewer
#15 lore raw

Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

In reply to Aaron Lu

Hi Aaron,

Quoted 1 line · Aaron Lu

[...]

Quoted 10 lines
Quoted 1 line

and that I think is the real golden nugget here.

I'm interested in how this patchset (further) improves direct map fragmentation so would like to evaluate it to see if my previous work to merge small mappings back in architecture layer[1] is still necessary.

I tried to apply this patchset on v6.1-rc3/2/1 and v6.0 but all failed, so I took one step back and evaluated the existing bpf_prog_pack. I'm aware of this patchset would make things even better by using order-9 page to backup the vmalloced range.

The patchset was based on bpf-next tree:

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/

Quoted 9 lines

I used the sample bpf prog: sample/bpf/sockex1 because it looks easy to run, feel free to let me know a better way to evaluate this.

- In kernels before bpf_prog_pack(v5.17 and earlier), this prog would cause 3 pages to change protection to RO+X from RW+NX; And if the three pages are far apart, each would cause a level 3 split then a level 2 split; Reality is, allocated pages tend to stay close physically so actual result will not be this bad.

[...]

Quoted 9 lines

- v6.1-rc3
There is no difference because I can't trigger another pack alloc before system is OOMed.

Conclusion: I think bpf_prog_pack is very good at reducing direct map fragmentation and this patchset can further improve this situation on large machines(with huge amount of memory) or with more large bpf progs loaded etc.

Thanks a lot for these experiments! I will include the data in the next version of the set.

Quoted 6 lines

Some imperfect things I can think of are(not related to this patchset):
1 Once a split happened, it remains happened. This may not be a big deal now with bpf_prog_pack and this patchset because the need to allocate a new order-9 page and thus cause a potential split should happen much much less;

I think we will need to regroup the direct map for some scenarios. But I am not aware of such workloads.

Quoted 3 lines

2 When a new order-9 page has to be allocated, there is no way to tell the allocator to allocate this order-9 page from an already splitted PUD range to avoid another PUD mapping split;

This would be a good improvement.

Quoted 2 lines

3 As Mike and others have mentioned, there are other users that can also cause direct map split.

Thanks,
Song

author reviewer
#16 lore raw

PATCH RESEND 2/5 x86/alternative: support vmalloc_exec() and vfree_exec()

In reply to Song Liu

Implement arch_vcopy_exec() and arch_invalidate_exec() to support vmalloc_exec.

arch_vcopy_exec() copies dynamic kernel text (such as BPF programs) to RO+X memory region allocated by vmalloc_exec().

arch_invalidate_exec() fills memory with 0xcc after it is released by vfree_exec().

---
 arch/x86/kernel/alternative.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/x86/kernel/alternative.c +12 -0
index 5cadcea035e0..73d89774ace3 100644
@@ -1270,6 +1270,18 @@ void *text_poke_copy(void *addr, const void *opcode, size_t len)
 	return addr;
 }
 
+void *arch_vcopy_exec(void *dst, void *src, size_t len)
+{
+	if (text_poke_copy(dst, src, len) == NULL)
+		return ERR_PTR(-EINVAL);
+	return dst;
+}
+
+int arch_invalidate_exec(void *ptr, size_t len)
+{
+	return IS_ERR_OR_NULL(text_poke_set(ptr, 0xcc, len));
+}
+
 /**
  * text_poke_set - memset into (an unused part of) RX memory
  * @addr: address to modify
Signature · 1 line
2.30.2
#17 lore raw

Re: [PATCH bpf-next v1 RESEND 2/5] x86/alternative: support vmalloc_exec() and vfree_exec()

Quoted 6 lines · Song Liu
arch/x86/kernel/alternative.c +0 -0
index 5cadcea035e0..73d89774ace3 100644

arch/x86/kernel/alternative.c void *text_poke_copy(void *addr, const void · line 1279

Quoted 10 lines
arch/x86/kernel/alternative.c +6 -0
@@ -1270,6 +1270,18 @@ void *text_poke_copy(void *addr, const void
 *opcode, size_t len)
        return addr;
 }
 
+void *arch_vcopy_exec(void *dst, void *src, size_t len)
+{
+       if (text_poke_copy(dst, src, len) == NULL)
+               return ERR_PTR(-EINVAL);
+       return dst;
+}

Except for this, there are no more users of text_poke_copy() right? Should it just be replaced with arch_vcopy_exec()?

author reviewer
#18 lore raw

Re: [PATCH bpf-next v1 RESEND 2/5] x86/alternative: support vmalloc_exec() and vfree_exec()

Quoted 12 lines · Rick P
Quoted 10 lines · Song Liu
arch/x86/kernel/alternative.c +0 -0
index 5cadcea035e0..73d89774ace3 100644
@@ -1270,6 +1270,18 @@ void *text_poke_copy(void *addr, const void
 *opcode, size_t len)
        return addr;
 }
 
Quoted 9 lines

arch/x86/kernel/alternative.c void *text_poke_copy(void *addr, const void *opcode, size_t len) · line 1278

Quoted 6 lines
arch/x86/kernel/alternative.c +6 -0
+void *arch_vcopy_exec(void *dst, void *src, size_t len)
+{
+       if (text_poke_copy(dst, src, len) == NULL)
+               return ERR_PTR(-EINVAL);
+       return dst;
+}

Except for this, there are no more users of text_poke_copy() right? Should it just be replaced with arch_vcopy_exec()?

I guess this is not really necessary, as we may have other use cases for text_poke_copy(), and the text_poke_* calls make a good API.

I won't object if folks agree removing text_poke_copy() for now is a better approach.

Thanks,
Song

author reviewer
#19 lore raw

PATCH RESEND 3/5 bpf: use vmalloc_exec for bpf program and bpf dispatcher

In reply to Song Liu

Use vmalloc_exec, vfree_exec, and vcopy_exec instead of bpf_prog_pack_alloc, bpf_prog_pack_free, and bpf_arch_text_copy.

vfree_exec doesn't require extra size information. Therefore, the free and error handling path can be simplified.

---
 arch/x86/net/bpf_jit_comp.c |  23 +----
 include/linux/bpf.h         |   3 -
 include/linux/filter.h      |   5 -
 kernel/bpf/core.c           | 180 +++---------------------------------
 kernel/bpf/dispatcher.c     |  11 +--
 5 files changed, 21 insertions(+), 201 deletions(-)
arch/x86/net/bpf_jit_comp.c +5 -18
index 36ffe67ad6e5..3db316b2f3a7 100644
@@ -228,11 +228,6 @@ static void jit_fill_hole(void *area, unsigned int size)
 	memset(area, 0xcc, size);
 }
 
-int bpf_arch_text_invalidate(void *dst, size_t len)
-{
-	return IS_ERR_OR_NULL(text_poke_set(dst, 0xcc, len));
-}
-
 struct jit_context {
 	int cleanup_addr; /* Epilogue code offset */
 
@@ -2496,11 +2491,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		if (proglen <= 0) {
 out_image:
 			image = NULL;
-			if (header) {
-				bpf_arch_text_copy(&header->size, &rw_header->size,
-						   sizeof(rw_header->size));
+			if (header)
 				bpf_jit_binary_pack_free(header, rw_header);
-			}
+
 			/* Fall back to interpreter mode */
 			prog = orig_prog;
 			if (extra_pass) {
@@ -2550,8 +2543,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		if (!prog->is_func || extra_pass) {
 			/*
 			 * bpf_jit_binary_pack_finalize fails in two scenarios:
-			 *   1) header is not pointing to proper module memory;
-			 *   2) the arch doesn't support bpf_arch_text_copy().
+			 *   1) header is not pointing to memory allocated by
+			 *      vmalloc_exec;
+			 *   2) the arch doesn't support vcopy_exec().
 			 *
 			 * Both cases are serious bugs and justify WARN_ON.
 			 */
@@ -2597,13 +2591,6 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
-void *bpf_arch_text_copy(void *dst, void *src, size_t len)
-{
-	if (text_poke_copy(dst, src, len) == NULL)
-		return ERR_PTR(-EINVAL);
-	return dst;
-}
-
 /* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */
 bool bpf_jit_supports_subprog_tailcalls(void)
 {
include/linux/bpf.h +0 -3
index 9fd68b0b3e9c..1d42f58334d0 100644
@@ -2654,9 +2654,6 @@ enum bpf_text_poke_type {
 int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
 		       void *addr1, void *addr2);
 
-void *bpf_arch_text_copy(void *dst, void *src, size_t len);
-int bpf_arch_text_invalidate(void *dst, size_t len);
-
 struct btf_id_set;
 bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
 
include/linux/filter.h +0 -5
index efc42a6e3aed..98e28126c24b 100644
@@ -1023,8 +1023,6 @@ extern long bpf_jit_limit_max;
 
 typedef void (*bpf_jit_fill_hole_t)(void *area, unsigned int size);
 
-void bpf_jit_fill_hole_with_zero(void *area, unsigned int size);
-
 struct bpf_binary_header *
 bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,
 		     unsigned int alignment,
@@ -1037,9 +1035,6 @@ void bpf_jit_free(struct bpf_prog *fp);
 struct bpf_binary_header *
 bpf_jit_binary_pack_hdr(const struct bpf_prog *fp);
 
-void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns);
-void bpf_prog_pack_free(struct bpf_binary_header *hdr);
-
 static inline bool bpf_prog_kallsyms_verify_off(const struct bpf_prog *fp)
 {
 	return list_empty(&fp->aux->ksym.lnode) ||
kernel/bpf/core.c +12 -168

246 lines not on this page · raw message

kernel/bpf/dispatcher.c +4 -7
index fa64b80b8bca..7ca8138fb8fa 100644
@@ -120,11 +120,11 @@ static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs)
 	tmp = d->num_progs ? d->rw_image + noff : NULL;
 	if (new) {
 		/* Prepare the dispatcher in d->rw_image. Then use
-		 * bpf_arch_text_copy to update d->image, which is RO+X.
+		 * vcopy_exec to update d->image, which is RO+X.
 		 */
 		if (bpf_dispatcher_prepare(d, new, tmp))
 			return;
-		if (IS_ERR(bpf_arch_text_copy(new, tmp, PAGE_SIZE / 2)))
+		if (IS_ERR(vcopy_exec(new, tmp, PAGE_SIZE / 2)))
 			return;
 	}
 
@@ -146,15 +146,12 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
 
 	mutex_lock(&d->mutex);
 	if (!d->image) {
-		d->image = bpf_prog_pack_alloc(PAGE_SIZE, bpf_jit_fill_hole_with_zero);
+		d->image = vmalloc_exec(PAGE_SIZE, PAGE_SIZE /* align */);
 		if (!d->image)
 			goto out;
 		d->rw_image = bpf_jit_alloc_exec(PAGE_SIZE);
 		if (!d->rw_image) {
-			u32 size = PAGE_SIZE;
-
-			bpf_arch_text_copy(d->image, &size, sizeof(size));
-			bpf_prog_pack_free((struct bpf_binary_header *)d->image);
+			vfree_exec((struct bpf_binary_header *)d->image);
 			d->image = NULL;
 			goto out;
 		}
Signature · 1 line
2.30.2
author reviewer
#20 lore raw

PATCH RESEND 4/5 vmalloc: introduce register_text_tail_vm()

In reply to Song Liu

Allow arch code to register some memory to be used by vmalloc_exec(). One possible use case is to allocate PMD pages for kernl text up to PMD_ALIGN(_etext), and use (_etext, PMD_ALIGN(_etext)) for vmalloc_exec. Currently, only one such region is supported.

---
 mm/vmalloc.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
mm/vmalloc.c +33 -0
index 6f4c73e67191..46f2b7e56670 100644
@@ -75,6 +75,9 @@ static const bool vmap_allow_huge = false;
 #define PMD_ALIGN(addr) ALIGN(addr, PMD_SIZE)
 #define PMD_ALIGN_DOWN(addr) ALIGN_DOWN(addr, PMD_SIZE)
 
+static struct vm_struct text_tail_vm;
+static struct vmap_area text_tail_va;
+
 bool is_vmalloc_addr(const void *x)
 {
 	unsigned long addr = (unsigned long)kasan_reset_tag(x);
@@ -653,6 +656,8 @@ int is_vmalloc_or_module_addr(const void *x)
 	unsigned long addr = (unsigned long)kasan_reset_tag(x);
 	if (addr >= MODULES_VADDR && addr < MODULES_END)
 		return 1;
+	if (addr >= text_tail_va.va_start && addr < text_tail_va.va_end)
+		return 1;
 #endif
 	return is_vmalloc_addr(x);
 }
@@ -2437,6 +2442,34 @@ static void vmap_init_free_space(void)
 	}
 }
 
+/*
+ * register_text_tail_vm() allows arch code to register memory regions
+ * for vmalloc_exec. Unlike regular memory regions used by vmalloc_exec,
+ * this region is never freed by vfree_exec.
+ *
+ * One possible use case is to allocate PMD pages for kernl text up to
+ * PMD_ALIGN(_etext), and use (_etext, PMD_ALIGN(_etext)) for vmalloc_exec.
+ */
+void register_text_tail_vm(unsigned long start, unsigned long end)
+{
+	struct vmap_area *va;
+
+	/* only support one region */
+	if (WARN_ON_ONCE(text_tail_vm.addr))
+		return;
+
+	va = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT);
+	if (WARN_ON_ONCE(!va))
+		return;
+	text_tail_vm.addr = (void *)start;
+	text_tail_vm.size = end - start;
+	text_tail_va.va_start = start;
+	text_tail_va.va_end = end;
+	text_tail_va.vm = &text_tail_vm;
+	memcpy(va, &text_tail_va, sizeof(*va));
+	insert_vmap_area_augment(va, NULL, &free_text_area_root, &free_text_area_list);
+}
+
 void __init vmalloc_init(void)
 {
 	struct vmap_area *va;
Signature · 1 line
2.30.2
author reviewer
#21 lore raw

PATCH RESEND 5/5 x86: use register_text_tail_vm

In reply to Song Liu

Allocate 2MB pages up to round_up(_etext, 2MB), and register memory [round_up(_etext, 4kb), round_up(_etext, 2MB)] with register_text_tail_vm so that we can use this part of memory for dynamic kernel text (BPF programs, etc.).

Here is an example:

[root@eth50-1 ~]# grep _etext /proc/kallsyms
ffffffff82202a08 T _etext

[root@eth50-1 ~]# grep bpf_prog_ /proc/kallsyms | tail -n 3

ffffffff8220f920 t bpf_prog_cc61a5364ac11d93_handle__sched_wakeup       [bpf]
ffffffff8220fa28 t bpf_prog_cc61a5364ac11d93_handle__sched_wakeup_new   [bpf]
ffffffff8220fad4 t bpf_prog_3bf73fa16f5e3d92_handle__sched_switch       [bpf]

[root@eth50-1 ~]# grep 0xffffffff82200000 /sys/kernel/debug/page_tables/kernel
0xffffffff82200000-0xffffffff82400000 2M ro PSE x pmd

ffffffff82200000-ffffffff82400000 is a 2MB page, serving kernel text, and bpf programs.

---
 arch/x86/include/asm/pgtable_64_types.h | 1 +
 arch/x86/mm/init_64.c                   | 4 +++-
 include/linux/vmalloc.h                 | 4 ++++
 3 files changed, 8 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable_64_types.h +1 -0
index 04f36063ad54..c0f9cceb109a 100644
@@ -101,6 +101,7 @@ extern unsigned int ptrs_per_p4d;
 #define PUD_MASK	(~(PUD_SIZE - 1))
 #define PGDIR_SIZE	(_AC(1, UL) << PGDIR_SHIFT)
 #define PGDIR_MASK	(~(PGDIR_SIZE - 1))
+#define PMD_ALIGN(x)	(((unsigned long)(x) + (PMD_SIZE - 1)) & PMD_MASK)
 
 /*
  * See Documentation/x86/x86_64/mm.rst for a description of the memory map.
arch/x86/mm/init_64.c +3 -1
index 3f040c6e5d13..5b42fc0c6099 100644
@@ -1373,7 +1373,7 @@ void mark_rodata_ro(void)
 	unsigned long start = PFN_ALIGN(_text);
 	unsigned long rodata_start = PFN_ALIGN(__start_rodata);
 	unsigned long end = (unsigned long)__end_rodata_hpage_align;
-	unsigned long text_end = PFN_ALIGN(_etext);
+	unsigned long text_end = PMD_ALIGN(_etext);
 	unsigned long rodata_end = PFN_ALIGN(__end_rodata);
 	unsigned long all_end;
 
@@ -1414,6 +1414,8 @@ void mark_rodata_ro(void)
 				(void *)rodata_end, (void *)_sdata);
 
 	debug_checkwx();
+	register_text_tail_vm(PFN_ALIGN((unsigned long)_etext),
+			      PMD_ALIGN((unsigned long)_etext));
 }
 
 int kern_addr_valid(unsigned long addr)
include/linux/vmalloc.h +4 -0
index 9b2042313c12..7365cf9c4e7f 100644
@@ -132,11 +132,15 @@ extern void vm_unmap_aliases(void);
 #ifdef CONFIG_MMU
 extern void __init vmalloc_init(void);
 extern unsigned long vmalloc_nr_pages(void);
+void register_text_tail_vm(unsigned long start, unsigned long end);
 #else
 static inline void vmalloc_init(void)
 {
 }
 static inline unsigned long vmalloc_nr_pages(void) { return 0; }
+void register_text_tail_vm(unsigned long start, unsigned long end)
+{
+}
 #endif
 
 extern void *vmalloc(unsigned long size) __alloc_size(1);
Signature · 1 line
2.30.2
#22 lore raw

Re: [PATCH bpf-next v1 RESEND 5/5] x86: use register_text_tail_vm

Quoted 55 lines · Song Liu

Allocate 2MB pages up to round_up(_etext, 2MB), and register memory [round_up(_etext, 4kb), round_up(_etext, 2MB)] with register_text_tail_vm
so that we can use this part of memory for dynamic kernel text (BPF programs, etc.).

Here is an example:

[root@eth50-1 ~]# grep _etext /proc/kallsyms
ffffffff82202a08 T _etext

[root@eth50-1 ~]# grep bpf_prog_ /proc/kallsyms | tail -n 3 ffffffff8220f920 t
bpf_prog_cc61a5364ac11d93_handle__sched_wakeup [bpf]
ffffffff8220fa28 t
bpf_prog_cc61a5364ac11d93_handle__sched_wakeup_new [bpf]
ffffffff8220fad4 t
bpf_prog_3bf73fa16f5e3d92_handle__sched_switch [bpf]

[root@eth50-1 ~]# grep 0xffffffff82200000
/sys/kernel/debug/page_tables/kernel
0xffffffff82200000-
0xffffffff82400000 2M ro PSE x pmd

ffffffff82200000-ffffffff82400000 is a 2MB page, serving kernel text, and
bpf programs.

---
 arch/x86/include/asm/pgtable_64_types.h | 1 +
 arch/x86/mm/init_64.c                   | 4 +++-
 include/linux/vmalloc.h                 | 4 ++++
 3 files changed, 8 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable_64_types.h +1 -0
index 04f36063ad54..c0f9cceb109a 100644
@@ -101,6 +101,7 @@ extern unsigned int ptrs_per_p4d;
 #define PUD_MASK	(~(PUD_SIZE - 1))
 #define PGDIR_SIZE	(_AC(1, UL) << PGDIR_SHIFT)
 #define PGDIR_MASK	(~(PGDIR_SIZE - 1))
+#define PMD_ALIGN(x)	(((unsigned long)(x) + (PMD_SIZE - 1)) &
 PMD_MASK)
 
 /*

* See Documentation/x86/x86_64/mm.rst for a description of the
memory map.

arch/x86/mm/init_64.c +0 -0
index 3f040c6e5d13..5b42fc0c6099 100644

arch/x86/mm/init_64.c void mark_rodata_ro(void) · line 1378

Quoted 7 lines
arch/x86/mm/init_64.c +1 -1
@@ -1373,7 +1373,7 @@ void mark_rodata_ro(void)
 	unsigned long start = PFN_ALIGN(_text);
 	unsigned long rodata_start = PFN_ALIGN(__start_rodata);
 	unsigned long end = (unsigned long)__end_rodata_hpage_align;
-	unsigned long text_end = PFN_ALIGN(_etext);
+	unsigned long text_end = PMD_ALIGN(_etext);
 	unsigned long rodata_end = PFN_ALIGN(__end_rodata);
 	unsigned long all_end;

Check out is_errata93(). Right now it assumes all text is between text- etext and MODULES_VADDR-MODULES_END. It's a quite old errata, but it would be nice if we had a is_text_addr() helper or something. To help keep track of the places where text might pop up.

Speaking of which, it might be nice to update Documentation/x86/x86_64/mm.rst with some hints that this area exists.

Quoted 18 lines
arch/x86/mm/init_64.c +2 -0
@@ -1414,6 +1414,8 @@ void mark_rodata_ro(void)
 				(void *)rodata_end, (void *)_sdata);
 
 	debug_checkwx();
+	register_text_tail_vm(PFN_ALIGN((unsigned long)_etext),
+			      PMD_ALIGN((unsigned long)_etext));
 }
 
 int kern_addr_valid(unsigned long addr)
include/linux/vmalloc.h +0 -0
index 9b2042313c12..7365cf9c4e7f 100644
@@ -132,11 +132,15 @@ extern void vm_unmap_aliases(void);
 #ifdef CONFIG_MMU
 extern void __init vmalloc_init(void);
 extern unsigned long vmalloc_nr_pages(void);

include/linux/vmalloc.h extern void vm_unmap_aliases(void); · line 144

Quoted 10 lines
include/linux/vmalloc.h +4 -0
+void register_text_tail_vm(unsigned long start, unsigned long end);
 #else
 static inline void vmalloc_init(void)
 {
 }
 static inline unsigned long vmalloc_nr_pages(void) { return 0; }
+void register_text_tail_vm(unsigned long start, unsigned long end)
+{
+}
 #endif

This looks like it should be in the previous patch.

Quoted 2 lines
include/linux/vmalloc.h +0 -0
 extern void *vmalloc(unsigned long size) __alloc_size(1);
author reviewer
#23 lore raw

Re: [PATCH bpf-next v1 RESEND 5/5] x86: use register_text_tail_vm

Quoted 95 lines · Rick P
Quoted 55 lines · Song Liu

Allocate 2MB pages up to round_up(_etext, 2MB), and register memory [round_up(_etext, 4kb), round_up(_etext, 2MB)] with register_text_tail_vm
so that we can use this part of memory for dynamic kernel text (BPF programs, etc.).

Here is an example:

[root@eth50-1 ~]# grep _etext /proc/kallsyms
ffffffff82202a08 T _etext

[root@eth50-1 ~]# grep bpf_prog_ /proc/kallsyms | tail -n 3 ffffffff8220f920 t
bpf_prog_cc61a5364ac11d93_handle__sched_wakeup [bpf]
ffffffff8220fa28 t
bpf_prog_cc61a5364ac11d93_handle__sched_wakeup_new [bpf]
ffffffff8220fad4 t
bpf_prog_3bf73fa16f5e3d92_handle__sched_switch [bpf]

[root@eth50-1 ~]# grep 0xffffffff82200000
/sys/kernel/debug/page_tables/kernel
0xffffffff82200000-
0xffffffff82400000 2M ro PSE x pmd

ffffffff82200000-ffffffff82400000 is a 2MB page, serving kernel text, and
bpf programs.

---
 arch/x86/include/asm/pgtable_64_types.h | 1 +
 arch/x86/mm/init_64.c                   | 4 +++-
 include/linux/vmalloc.h                 | 4 ++++
 3 files changed, 8 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable_64_types.h +1 -0
index 04f36063ad54..c0f9cceb109a 100644
@@ -101,6 +101,7 @@ extern unsigned int ptrs_per_p4d;
 #define PUD_MASK     (~(PUD_SIZE - 1))
 #define PGDIR_SIZE   (_AC(1, UL) << PGDIR_SHIFT)
 #define PGDIR_MASK   (~(PGDIR_SIZE - 1))
+#define PMD_ALIGN(x) (((unsigned long)(x) + (PMD_SIZE - 1)) &
 PMD_MASK)
 
 /*

* See Documentation/x86/x86_64/mm.rst for a description of the
memory map.

arch/x86/mm/init_64.c +0 -0
index 3f040c6e5d13..5b42fc0c6099 100644

arch/x86/mm/init_64.c void mark_rodata_ro(void) · line 1378

Quoted 7 lines
arch/x86/mm/init_64.c +1 -1
@@ -1373,7 +1373,7 @@ void mark_rodata_ro(void)
      unsigned long start = PFN_ALIGN(_text);
      unsigned long rodata_start = PFN_ALIGN(__start_rodata);
      unsigned long end = (unsigned long)__end_rodata_hpage_align;
-     unsigned long text_end = PFN_ALIGN(_etext);
+     unsigned long text_end = PMD_ALIGN(_etext);
      unsigned long rodata_end = PFN_ALIGN(__end_rodata);
      unsigned long all_end;

Check out is_errata93(). Right now it assumes all text is between text- etext and MODULES_VADDR-MODULES_END. It's a quite old errata, but it would be nice if we had a is_text_addr() helper or something. To help keep track of the places where text might pop up.

Speaking of which, it might be nice to update Documentation/x86/x86_64/mm.rst with some hints that this area exists.

Quoted 22 lines
arch/x86/mm/init_64.c +2 -0
@@ -1414,6 +1414,8 @@ void mark_rodata_ro(void)
                              (void *)rodata_end, (void *)_sdata);
 
      debug_checkwx();
+     register_text_tail_vm(PFN_ALIGN((unsigned long)_etext),
+                           PMD_ALIGN((unsigned long)_etext));
 }
 
 int kern_addr_valid(unsigned long addr)
include/linux/vmalloc.h +1 -0
index 9b2042313c12..7365cf9c4e7f 100644
@@ -132,11 +132,15 @@ extern void vm_unmap_aliases(void);
 #ifdef CONFIG_MMU
 extern void __init vmalloc_init(void);
 extern unsigned long vmalloc_nr_pages(void);
+void register_text_tail_vm(unsigned long start, unsigned long end);
 #else
 static inline void vmalloc_init(void)
 {
Quoted 8 lines

include/linux/vmalloc.h extern void vm_unmap_aliases(void); · line 144

Quoted 6 lines
include/linux/vmalloc.h +3 -0
 }
 static inline unsigned long vmalloc_nr_pages(void) { return 0; }
+void register_text_tail_vm(unsigned long start, unsigned long end)
+{
+}
 #endif

This looks like it should be in the previous patch.

Good catch! I will fix it in the next version.

Thanks,
Song

Quoted 3 lines
Quoted 3 lines
include/linux/vmalloc.h +0 -0
 extern void *vmalloc(unsigned long size) __alloc_size(1);
#24 lore raw
In reply to Song Liu
Quoted 5 lines · Song Liu

This set enables bpf programs and bpf dispatchers to share huge pages with new API:
vmalloc_exec()
vfree_exec()
vcopy_exec()

Maybe it's just me, but I don't like the names very much. They imply a slight extension to the vmalloc API, but while they use the vmalloc mechanisms internally, the API is actually quite different.

So why not something like:

execmem_alloc
execmem_free
execmem_fill or execmem_set or copy_to_execmem

?

author reviewer
#25 lore raw
Quoted 12 lines · Christoph Hellwig
Quoted 5 lines · Song Liu

This set enables bpf programs and bpf dispatchers to share huge pages with new API:
vmalloc_exec()
vfree_exec()
vcopy_exec()

Maybe it's just me, but I don't like the names very much. They imply a slight extension to the vmalloc API, but while they use the vmalloc mechanisms internally, the API is actually quite different.

Quoted 7 lines

So why not something like:

execmem_alloc
execmem_free
execmem_fill or execmem_set or copy_to_execmem

?

I don't have a strong preference on names. We can change the name to whatever we agree on.

Thanks,
Song

#26 lore raw
In reply to Song Liu
Quoted 1 line · Song Liu

Changes RFC v2 => PATCH v1:

<-- snip -->

Quoted 1 line

3. Drop changes for kernel modules and focus on BPF side changes.

Yeah because in my testing of RFCv2 things blow up pretty hard, I haven't been able to successfully even boot *any* system with those patches, virutal or bare metal.

I'm very happy to help test new iterations if you have anything.

Luis

#27 lore raw
In reply to Song Liu
Quoted 24 lines · Song Liu

This set enables bpf programs and bpf dispatchers to share huge pages with
new API:
vmalloc_exec()
vfree_exec()
vcopy_exec()

The idea is similar to Peter's suggestion in [1].

vmalloc_exec() manages a set of PMD_SIZE RO+X memory, and allocates these
memory to its users. vfree_exec() is used to free memory allocated by vmalloc_exec(). vcopy_exec() is used to update memory allocated by vmalloc_exec().

Memory allocated by vmalloc_exec() is RO+X, so this doesnot violate W^X.
The caller has to update the content with text_poke like mechanism. Specifically, vcopy_exec() is provided to update memory allocated by vmalloc_exec(). vcopy_exec() also makes sure the update stays in the boundary of one chunk allocated by vmalloc_exec(). Please refer to patch
1/5 for more details of

Quoted 8 lines

Patch 3/5 uses these new APIs in bpf program and bpf dispatcher.

Patch 4/5 and 5/5 allows static kernel text (_stext to _etext) to share
PMD_SIZE pages with dynamic kernel text on x86_64. This is achieved by
allocating PMD_SIZE pages to roundup(_etext, PMD_SIZE), and then use _etext to roundup(_etext, PMD_SIZE) for dynamic kernel text.

It might help to spell out what the benefits of this are. My understanding is that (to my surprise) we actually haven't seen a performance improvement with using 2MB pages for JITs. The main performance benefit you saw on your previous version was from reduced fragmentation of the direct map IIUC. This was from the effect of reusing the same pages for JITs so that new ones don't need to be broken.

The other benefit of this thing is reduced shootdowns. It can load a JIT with about only a local TLB flush on average, which should help really high cpu systems some unknown amount.

author reviewer
#28 lore raw
Quoted 43 lines · Rick P
Quoted 24 lines · Song Liu

This set enables bpf programs and bpf dispatchers to share huge pages with
new API:
vmalloc_exec()
vfree_exec()
vcopy_exec()

The idea is similar to Peter's suggestion in [1].

vmalloc_exec() manages a set of PMD_SIZE RO+X memory, and allocates these
memory to its users. vfree_exec() is used to free memory allocated by vmalloc_exec(). vcopy_exec() is used to update memory allocated by vmalloc_exec().

Memory allocated by vmalloc_exec() is RO+X, so this doesnot violate W^X.
The caller has to update the content with text_poke like mechanism. Specifically, vcopy_exec() is provided to update memory allocated by vmalloc_exec(). vcopy_exec() also makes sure the update stays in the boundary of one chunk allocated by vmalloc_exec(). Please refer to patch
1/5 for more details of

Quoted 8 lines

Patch 3/5 uses these new APIs in bpf program and bpf dispatcher.

Patch 4/5 and 5/5 allows static kernel text (_stext to _etext) to share
PMD_SIZE pages with dynamic kernel text on x86_64. This is achieved by
allocating PMD_SIZE pages to roundup(_etext, PMD_SIZE), and then use _etext to roundup(_etext, PMD_SIZE) for dynamic kernel text.

It might help to spell out what the benefits of this are. My understanding is that (to my surprise) we actually haven't seen a performance improvement with using 2MB pages for JITs. The main performance benefit you saw on your previous version was from reduced fragmentation of the direct map IIUC. This was from the effect of reusing the same pages for JITs so that new ones don't need to be broken.

Quoted 3 lines

The other benefit of this thing is reduced shootdowns. It can load a JIT with about only a local TLB flush on average, which should help really high cpu systems some unknown amount.

Thanks for pointing out the missing information.

I don't have a benchmark that uses very big BPF programs, so the results I have don't show much benefit from fewer iTLB misses.

Song