bpf: Add mmapable task_local storage
Subsystems (4)
cover bpf: Add mmapable task_local storage
This series adds support for mmap()ing single task_local storage mapvals into userspace. Two motivating usecases:
* sched_ext ([0]) schedulers might want to act on 'scheduling hints' provided by userspace tasks. For example, a task can tag itself as latency-sensitive but not particularly computationally intensive and BPF scheduler can use this information to make better scheduling decisions. Similarly, a database task about to start a transaction can tag itself as doing so without high overhead by writing to the mmap'd mapval. In both cases the information is task-specific and in the latter it'd be preferable to avoid incurring syscall overhead as the hint would change often.
* strobemeta ([1]) technique to read thread_local storage is used by tracing programs at Meta to annotate tracing data with task-specific metadata. For example, a multithreaded webserver with a pool of worker threads preparing responses and other threads handling request connections might want to tag threads by type, and further tag worker threads with feature flags enabled during request processing.
* The strobemeta technique predates existence of task_local storage map, instead relying on reverse-engineering thread_local storage implementation specifics. The approach enabled here avoids much of this complexity.
The general thrust of this series' implementation is "simplest thing that works". A userspace thread can mmap() a task_local storage map fd and receive the map_value corresponding to its task. In the future we can support mmap()ing in other threads' map_values via offset parameter or some other approach. Similarly, this series makes no attempt to pack multiple map_values into a userspace-mappable page - each map_value for a BPF_F_MMAPABLE task_local storage map is given its own page. For the motivating usecases above neither of those potential improvements is necessary. Patch 1's summary digs deeper into implementation details.
This series' changes to generic local_storage implementation shared by cgroup_local storage and others will make extending this support to those local storage types straightforward in the future.
Summary of patches:
* Patch 1 adds support for mmapable map_vals in generic bpf_local_storage infrastructure and uses the new feature in task_local storage
* Patch 2 adds tests
[0]: https://lore.kernel.org/bpf/[email protected]/
[1]: tools/testing/selftests/bpf/progs/strobemeta*
Dave Marchevsky (2):
bpf: Support BPF_F_MMAPABLE task_local storage selftests/bpf: Add test exercising mmapable task_local_storage
include/linux/bpf_local_storage.h | 14 +- kernel/bpf/bpf_local_storage.c | 145 +++++++++++--- kernel/bpf/bpf_task_storage.c | 35 +++- kernel/bpf/syscall.c | 2 +- .../bpf/prog_tests/task_local_storage.c | 177 ++++++++++++++++++ .../bpf/progs/task_local_storage__mmap.c | 59 ++++++ .../bpf/progs/task_local_storage__mmap.h | 7 + .../bpf/progs/task_local_storage__mmap_fail.c | 39 ++++ 8 files changed, 445 insertions(+), 33 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap.c create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap.h create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c
Signature · 1 line
2.34.1
PATCH 1/2 bpf: Support BPF_F_MMAPABLE task_local storage
This patch modifies the generic bpf_local_storage infrastructure to support mmapable map values and adds mmap() handling to task_local storage leveraging this new functionality. A userspace task which mmap's a task_local storage map will receive a pointer to the map_value corresponding to that tasks' key - mmap'ing in other tasks' mapvals is not supported in this patch.
Currently, struct bpf_local_storage_elem contains both bookkeeping information as well as a struct bpf_local_storage_data with additional bookkeeping information and the actual mapval data. We can't simply map the page containing this struct into userspace. Instead, mmapable local_storage uses bpf_local_storage_data's data field to point to the actual mapval, which is allocated separately such that it can be mmapped. Only the mapval lives on the page(s) allocated for it.
The lifetime of the actual_data mmapable region is tied to the bpf_local_storage_elem which points to it. This doesn't necessarily mean that the pages go away when the bpf_local_storage_elem is free'd - if they're mapped into some userspace process they will remain until unmapped, but are no longer the task_local storage's mapval.
Implementation details:
* A few small helpers are added to deal with bpf_local_storage_data's 'data' field having different semantics when the local_storage map is mmapable. With their help, many of the changes to existing code are purely mechanical (e.g. sdata->data becomes sdata_mapval(sdata), selem->elem_size becomes selem_bytes_used(selem)).
* The map flags are copied into bpf_local_storage_data when its containing bpf_local_storage_elem is alloc'd, since the bpf_local_storage_map associated with them may be gone when bpf_local_storage_data is free'd, and testing flags for BPF_F_MMAPABLE is necessary when free'ing to ensure that the mmapable region is free'd.
* The extra field doesn't change bpf_local_storage_elem's size. There were 48 bytes of padding after the bpf_local_storage_data field, now there are 40.
* Currently, bpf_local_storage_update always creates a new bpf_local_storage_elem for the 'updated' value - the only exception being if the map_value has a bpf_spin_lock field, in which case the spin lock is grabbed instead of the less granular bpf_local_storage lock, and the value updated in place. This inplace update behavior is desired for mmapable local_storage map_values as well, since creating a new selem would result in new mmapable pages.
* The size of the mmapable pages are accounted for when calling mem_{charge,uncharge}. If the pages are mmap'd into a userspace task mem_uncharge may be called before they actually go away.
--- include/linux/bpf_local_storage.h | 14 ++- kernel/bpf/bpf_local_storage.c | 145 ++++++++++++++++++++++++------ kernel/bpf/bpf_task_storage.c | 35 ++++++-- kernel/bpf/syscall.c | 2 +- 4 files changed, 163 insertions(+), 33 deletions(-)
include/linux/bpf_local_storage.h +13 -1
@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ; +/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ; + }; };/* Linked to bpf_local_storage and bpf_local_storage_map */
@@ -124,6 +134,8 @@ static struct bpf_local_storage_cache name = { \/* Helper functions for bpf_local_storage */ int bpf_local_storage_map_alloc_check (union bpf_attr *attr); +void *sdata_mapval (struct bpf_local_storage_data *data); +struct bpf_map * bpf_local_storage_map_alloc(union bpf_attr *attr, struct bpf_local_storage_cache *cache,
kernel/bpf/bpf_local_storage.c +120 -25
259 lines not on this page · raw message
kernel/bpf/bpf_task_storage.c +29 -6
@@ -90,6 +90,7 @@ void bpf_task_storage_free(struct task_struct *task)static void *bpf_pid_task_storage_lookup_elem (struct bpf_map *map,void *key) {struct bpf_local_storage_data *sdata; +struct bpf_local_storage_map *smap;struct task_struct *task;unsigned int f_flags;struct pid *pid;
@@ -114,7 +115,8 @@ static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key) sdata =task_storage_lookup (task, map, true);bpf_task_storage_unlock ();put_pid (pid); -return sdata ? sdata->data :NULL ; + smap = (struct bpf_local_storage_map *)map; +return sdata ?sdata_mapval (sdata) :NULL ; out:put_pid (pid);return ERR_PTR (err);
@@ -209,18 +211,19 @@ static void *__bpf_task_storage_get(struct bpf_map *map, u64 flags, gfp_t gfp_flags, bool nobusy) { struct bpf_local_storage_data *sdata; +struct bpf_local_storage_map *smap; + smap = (struct bpf_local_storage_map *)map; sdata =task_storage_lookup (task, map, nobusy);if (sdata) -return sdata->data; +return sdata_mapval (sdata);/* only allocate new storage, when the task is refcounted */ if (refcount_read (&task->usage) && (flags &BPF_LOCAL_STORAGE_GET_F_CREATE ) && nobusy) { - sdata =bpf_local_storage_update ( - task, (struct bpf_local_storage_map *)map, value, -BPF_NOEXIST , gfp_flags); -return IS_ERR (sdata) ?NULL : sdata->data; + sdata =bpf_local_storage_update (task, smap, value, +BPF_NOEXIST , gfp_flags); +return IS_ERR (sdata) ?NULL :sdata_mapval (sdata); }return NULL ;
@@ -317,6 +320,25 @@ static void task_storage_map_free(struct bpf_map *map)bpf_local_storage_map_free (map, &task_cache, &bpf_task_storage_busy); } +static int task_storage_map_mmap (struct bpf_map *map,struct vm_area_struct *vma) +{ +void *data; + +if (!(map->map_flags &BPF_F_MMAPABLE ) || vma->vm_pgoff || + (vma->vm_end - vma->vm_start) < map->value_size) +return -EINVAL ; + +WARN_ON_ONCE (!bpf_rcu_lock_held ()); +bpf_task_storage_lock (); + data =__bpf_task_storage_get (map, current,NULL ,BPF_LOCAL_STORAGE_GET_F_CREATE , +0 , true); +bpf_task_storage_unlock (); +if (!data) +return -EINVAL ; + +return remap_vmalloc_range (vma, data, vma->vm_pgoff); +} + BTF_ID_LIST_GLOBAL_SINGLE(bpf_local_storage_map_btf_id, struct, bpf_local_storage_map)const struct bpf_map_ops task_storage_map_ops = { .map_meta_equal = bpf_map_meta_equal,
@@ -331,6 +353,7 @@ const struct bpf_map_ops task_storage_map_ops = { .map_mem_usage = bpf_local_storage_map_mem_usage, .map_btf_id = &bpf_local_storage_map_btf_id[0], .map_owner_storage_ptr = task_storage_ptr, + .map_mmap = task_storage_map_mmap, };const struct bpf_func_proto bpf_task_storage_get_recur_proto = {
kernel/bpf/syscall.c +1 -1
@@ -404,7 +404,7 @@ static void bpf_map_release_memcg(struct bpf_map *map)obj_cgroup_put (map->objcg); } -static struct mem_cgroup *bpf_map_get_memcg (const struct bpf_map *map) +struct mem_cgroup *bpf_map_get_memcg (const struct bpf_map *map) {if (map->objcg)return get_mem_cgroup_from_objcg (map->objcg);
Signature · 1 line
2.34.1
Re: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
Quoted 15 lines · Dave Marchevsky
This patch modifies the generic
bpf_local_storageinfrastructure to support mmapable map values and addsmmap()handling totask_localstorage leveraging this new functionality. A userspace task which mmap's atask_localstorage map will receive a pointer to themap_valuecorresponding to that tasks' key - mmap'ing in other tasks' mapvals is not supported in this patch.Currently, struct
bpf_local_storage_elemcontains both bookkeeping information as well as a structbpf_local_storage_datawith additional bookkeeping information and the actual mapval data. We can't simply map the page containing this struct into userspace. Instead, mmapablelocal_storageuses bpf_local_storage_data's data field to point to the actual mapval, which is allocated separately such that it can be mmapped. Only the mapval lives on the page(s) allocated for it.
Quoted 5 lines
The lifetime of the
actual_datammapable region is tied to thebpf_local_storage_elemwhich points to it. This doesn't necessarily mean that the pages go away when thebpf_local_storage_elemis free'd - if they're mapped into some userspace process they will remain until unmapped, but are no longer thetask_localstorage's mapval.
Those bits look good to me. vfree() uses __free_pages(), which participates in refcounting. remap_vmalloc_range() acquires references to the individual pages, which will be dropped once the page tables disappear on munmap(). The vmalloc area doesn't need to stick around.
Martin KaFai Lau
reviewerRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave MarchevskyQuoted 9 lines · Dave Marchevsky
include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;
include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;
The pages (that can be mmap'ed later) feel like a specific kind of kptr.
Have you thought about allowing a kptr (pointing to some pages that can be mmap'ed later) to be stored as one of the members of the map's value as a kptr. bpf_local_storage_map is one of the maps that supports kptr.
struct normal_and_mmap_value {
int some_int;
int __percpu_kptr *some_cnts;
struct bpf_mmap_page __kptr *some_stats;
};struct mmap_only_value {
struct bpf_mmap_page __kptr *some_stats;
};[ ... ]
Quoted 34 lines
kernel/bpf/bpf_local_storage.c +18
-1@@ -15,7 +15,8 @@#include <linux/rcupdate_trace.h> #include <linux/rcupdate_wait.h> -#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE) +#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \ + (BPF_F_NO_PREALLOC | BPF_F_CLONE | BPF_F_MMAPABLE)static struct bpf_local_storage_map_bucket * select_bucket(struct bpf_local_storage_map *smap,@@ -24,6 +25,51 @@ select_bucket(struct bpf_local_storage_map *smap,return &smap->buckets[hash_ptr (selem, smap->bucket_log)]; } +struct mem_cgroup *bpf_map_get_memcg (const struct bpf_map *map); + +void *alloc_mmapable_selem_value (struct bpf_local_storage_map *smap) +{ +struct mem_cgroup *memcg, *old_memcg; +void *ptr; + + memcg =bpf_map_get_memcg (&smap->map); + old_memcg =set_active_memcg (memcg); + ptr =bpf_map_area_mmapable_alloc (PAGE_ALIGN (smap->map.value_size), +NUMA_NO_NODE ); +set_active_memcg (old_memcg); +mem_cgroup_put (memcg); + +return ptr; +}
[ ... ]
Quoted 4 lines
kernel/bpf/bpf_local_storage.c +0
-0@@ -76,10 +122,19 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value, bool charge_mem, gfp_t gfp_flags) { struct bpf_local_storage_elem *selem;
kernel/bpf/bpf_local_storage.c bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, · line 133
Quoted 10 lines
kernel/bpf/bpf_local_storage.c +6
-1+void *mmapable_value =NULL ; +u32 selem_mem; -if (charge_mem &&mem_charge (smap, owner, smap->elem_size)) + selem_mem =selem_bytes_used (smap); +if (charge_mem &&mem_charge (smap, owner, selem_mem))return NULL ; +if (smap->map.map_flags &BPF_F_MMAPABLE ) { + mmapable_value =alloc_mmapable_selem_value (smap);
This probably is not always safe for bpf prog to do. Leaving the gfp_flags was not used aside, the bpf local storage is moving to the bpf's memalloc because of https://lore.kernel.org/bpf/[email protected]/
Quoted 4 lines
kernel/bpf/bpf_local_storage.c +4
-0+if (!mmapable_value) + goto err_out; + } +
Re: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
Quoted 32 lines · Martin KaFai Lau
Quoted 9 lines · Dave Marchevsky
include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;The pages (that can be mmap'ed later) feel like a specific kind of kptr.
Have you thought about allowing a kptr (pointing to some pages that can be mmap'ed later) to be stored as one of the members of the map's value as a kptr.
bpf_local_storage_mapis one of the maps that supports kptr.struct normal_and_mmap_value { int some_int; int __percpu_kptr *some_cnts; struct bpf_mmap_page __kptr *some_stats; };
Quoted 6 lines
struct mmap_only_value { struct bpf_mmap_page __kptr *some_stats; };[ ... ]
This is an intriguing idea. For conciseness I'll call this specific kind of kptr 'mmapable kptrs' for the rest of this message. Below is more of a brainstorming dump than a cohesive response, separate trains of thought are separated by two newlines.
My initial thought upon seeing struct normal_and_mmap_value was to note that we currently don't support mmaping for map_value types with _any_ special fields ('special' as determined by btf_parse_fields). But IIUC you're actually talking about exposing the some_stats pointee memory via mmap, not the containing struct with kptr fields. That is, for maps that support these kptrs, mmap()ing a map with value type struct normal_and_mmap_value would return the some_stats pointer value, and likely initialize the pointer similarly to BPF_LOCAL_STORAGE_GET_F_CREATE logic in this patch. We'd only be able to support one such mmapable kptr field per mapval type, but that isn't a dealbreaker.
Some maps, like task_storage, would only support mmap() on a map_value with mmapable kptr field, as mmap()ing the mapval itself doesn't make sense or is unsafe. Seems like arraymap would do the opposite, only supporting mmap()ing the mapval itself. I'm curious if any map could feasibly support both, and if so, might have to do logic like:
if (map_val has mmapable kptr)
mmap the pointee of mmapable kptr
else
mmap the map_val itself
Which is maybe too confusing of a detail to expose to BPF program writers. Maybe a little too presumptuous and brainstorm-ey given the limited number of mmap()able maps currently, but making this a kptr type means maps should either ignore/fail if they don't support it, or have consistent semantics amongst maps that do support it.
Instead of struct bpf_mmap_page __kptr *some_stats; I'd prefer something like
struct my_type { long count; long another_count; };
struct mmap_only_value {
struct my_type __mmapable_kptr *some_stats;
};
This way the type of mmap()able field is known to BPF programs that interact with it. This is all assuming that struct bpf_mmap_page is an opaque page-sized blob of bytes.
We could then support structs like
struct mmap_value_and_lock {
struct bpf_spin_lock l;
int some_int;
struct my_type __mmapable_kptr *some_stats;
};
and have bpf_map_update_elem handler use the spin_lock instead of map-internal lock where appropriate. But no way to ensure userspace task using the mmap()ed region uses the spin_lock.
Quoted 34 lines
Quoted 31 lines
kernel/bpf/bpf_local_storage.c +17
-0@@ -15,7 +15,8 @@#include <linux/rcupdate_trace.h> #include <linux/rcupdate_wait.h> -#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE) +#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \ + (BPF_F_NO_PREALLOC | BPF_F_CLONE | BPF_F_MMAPABLE)static struct bpf_local_storage_map_bucket * select_bucket(struct bpf_local_storage_map *smap,@@ -24,6 +25,51 @@ select_bucket(struct bpf_local_storage_map *smap,return &smap->buckets[hash_ptr (selem, smap->bucket_log)]; } +struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map); + +void *alloc_mmapable_selem_value (struct bpf_local_storage_map *smap) +{ +struct mem_cgroup *memcg, *old_memcg; +void *ptr; + + memcg =bpf_map_get_memcg (&smap->map); + old_memcg =set_active_memcg (memcg); + ptr =bpf_map_area_mmapable_alloc (PAGE_ALIGN (smap->map.value_size), +NUMA_NO_NODE ); +set_active_memcg (old_memcg); +mem_cgroup_put (memcg); + +return ptr; +}[ ... ]
Quoted 15 lines
kernel/bpf/bpf_local_storage.c bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, · line 132
Quoted 12 lines
kernel/bpf/bpf_local_storage.c +5
-0@@ -76,10 +122,19 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value, bool charge_mem, gfp_t gfp_flags) { struct bpf_local_storage_elem *selem; +void *mmapable_value =NULL ; +u32 selem_mem; - if (charge_mem && mem_charge(smap, owner, smap->elem_size)) + selem_mem = selem_bytes_used(smap); +if (charge_mem &&mem_charge (smap, owner, selem_mem))return NULL ; + if (smap->map.map_flags & BPF_F_MMAPABLE) { + mmapable_value =alloc_mmapable_selem_value (smap);This probably is not always safe for bpf prog to do. Leaving the
gfp_flagswas not used aside, the bpf local storage is moving to the bpf's memalloc because of https://lore.kernel.org/bpf/[email protected]/
Minor point: alloc_mmapable_selem_value's bpf_map_area_mmapable_alloc call will always call vmalloc under the hood. vmalloc has locks as well, so your point stands.
I think I see how this ties into your 'specific kptr type' proposal above. Let me know if this sounds right: if there was a bpf_mem_alloc type focused on providing vmalloc'd mmap()able memory, we could use it here instead of raw vmalloc and avoid the lock recursion problem linked above. Such an allocator could be used in something like bpf_obj_new to create the __mmapable_kptr - either from BPF prog or mmap path before remap_vmalloc_range.
re: gfp_flags, looks like verifier is setting this param to either GFP_ATOMIC or GFP_KERNEL. Looks like we should not allow GFP_KERNEL allocs here?
Quoted 5 lines
Quoted 4 lines
kernel/bpf/bpf_local_storage.c +4
-0+if (!mmapable_value) + goto err_out; + } +
Martin KaFai Lau
reviewerRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
Quoted 39 lines · David Marchevsky
Quoted 36 lines · Martin KaFai Lau
Quoted 9 lines · Dave Marchevsky
include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;The pages (that can be mmap'ed later) feel like a specific kind of kptr.
Have you thought about allowing a kptr (pointing to some pages that can be mmap'ed later) to be stored as one of the members of the map's value as a kptr.
bpf_local_storage_mapis one of the maps that supports kptr.struct normal_and_mmap_value { int some_int; int __percpu_kptr *some_cnts; struct bpf_mmap_page __kptr *some_stats; };struct mmap_only_value { struct bpf_mmap_page __kptr *some_stats; };
Quoted 7 lines
Quoted 2 lines
[ ... ]
This is an intriguing idea. For conciseness I'll call this specific kind of kptr 'mmapable kptrs' for the rest of this message. Below is more of a brainstorming dump than a cohesive response, separate trains of thought are separated by two newlines.
Thanks for bearing with me while some ideas could be crazy. I am trying to see how this would look like for other local storage, sk and inode. Allocating a page for each sk will not be nice for server with half a million sk(s). e.g. half a million sk(s) sharing a few bandwidth policies or a few tuning parameters. Creating something mmap'able to the user space and also sharable among many sk(s) will be useful.
Quoted 13 lines
My initial thought upon seeing struct
normal_and_mmap_valuewas to note that we currently don't support mmaping formap_valuetypes with _any_ special fields ('special' as determined bybtf_parse_fields). But IIUC you're actually talking about exposing thesome_statspointee memory via mmap, not the containing struct with kptr fields. That is, for maps that support these kptrs, mmap()ing a map with value type structnormal_and_mmap_valuewould return thesome_statspointer value, and likely initialize the pointer similarly toBPF_LOCAL_STORAGE_GET_F_CREATElogic in this patch. We'd only be able to support one such mmapable kptr field per mapval type, but that isn't a dealbreaker.
Quoted 3 lines
Some maps, like
task_storage, would only supportmmap()on amap_valuewith mmapable kptr field, as mmap()ing the mapval itself doesn't make sense or is unsafe. Seems like arraymap would do the opposite, only
Changing direction a bit since arraymap is brought up. :)
arraymap supports BPF_F_MMAPABLE. If the local storage map's value can store an arraymap as kptr, the bpf prog should be able to access it as a map. More like the current map-in-map setup. The arraymap can be used as regular map in the user space also (like pinning). It may need some btf plumbing to tell the value type of the arrayamp to the verifier.
The syscall bpf_map_update_elem(task_storage_map_fd, &task_pidfd, &value, flags) can be used where the value->array_mmap initialized as an arraymap_fd. This will limit the arraymap kptr update only from the syscall side which seems to be your usecase also? Allocating the arraymap from the bpf prog side needs some thoughts and need a whitelist.
The same goes for the syscall bpf_map_lookup_elem(task_storage_map_fd, &task_pidfd, &value). The kernel can return a fd in value->array_mmap. May be we can create a libbpf helper to free the fd(s) resources held in the looked-up value by using the value's btf.
The bpf_local_storage_map side probably does not need to support mmap() then.
Quoted 103 lines
supporting mmap()ing the mapval itself. I'm curious if any map could feasibly support both, and if so, might have to do logic like:
if (
map_valhas mmapable kptr)
mmap the pointee of mmapable kptr
else
mmap themap_valitselfWhich is maybe too confusing of a detail to expose to BPF program writers. Maybe a little too presumptuous and brainstorm-ey given the limited number of mmap()able maps currently, but making this a kptr type means maps should either ignore/fail if they don't support it, or have consistent semantics amongst maps that do support it.
Instead of struct
bpf_mmap_page__kptr*some_stats; I'd prefer something likestruct
my_type{ long count; longanother_count; };struct mmap_only_value { struct my_type __mmapable_kptr *some_stats; };This way the type of mmap()able field is known to BPF programs that interact with it. This is all assuming that struct
bpf_mmap_pageis an opaque page-sized blob of bytes.We could then support structs like
struct mmap_value_and_lock { struct bpf_spin_lock l; int some_int; struct my_type __mmapable_kptr *some_stats; };and have
bpf_map_update_elemhandler use thespin_lockinstead of map-internal lock where appropriate. But no way to ensure userspace task using the mmap()ed region uses thespin_lock.Quoted 34 lines
Quoted 31 lines
kernel/bpf/bpf_local_storage.c +17
-0@@ -15,7 +15,8 @@#include <linux/rcupdate_trace.h> #include <linux/rcupdate_wait.h> -#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE) +#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \ + (BPF_F_NO_PREALLOC | BPF_F_CLONE | BPF_F_MMAPABLE)static struct bpf_local_storage_map_bucket * select_bucket(struct bpf_local_storage_map *smap,@@ -24,6 +25,51 @@ select_bucket(struct bpf_local_storage_map *smap,return &smap->buckets[hash_ptr (selem, smap->bucket_log)]; } +struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map); + +void *alloc_mmapable_selem_value (struct bpf_local_storage_map *smap) +{ +struct mem_cgroup *memcg, *old_memcg; +void *ptr; + + memcg =bpf_map_get_memcg (&smap->map); + old_memcg =set_active_memcg (memcg); + ptr =bpf_map_area_mmapable_alloc (PAGE_ALIGN (smap->map.value_size), +NUMA_NO_NODE ); +set_active_memcg (old_memcg); +mem_cgroup_put (memcg); + +return ptr; +}[ ... ]
Quoted 15 lines
kernel/bpf/bpf_local_storage.c bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, · line 132
Quoted 12 lines
kernel/bpf/bpf_local_storage.c +5
-0@@ -76,10 +122,19 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, void *value, bool charge_mem, gfp_t gfp_flags) { struct bpf_local_storage_elem *selem; +void *mmapable_value =NULL ; +u32 selem_mem; - if (charge_mem && mem_charge(smap, owner, smap->elem_size)) + selem_mem = selem_bytes_used(smap); +if (charge_mem &&mem_charge (smap, owner, selem_mem))return NULL ; + if (smap->map.map_flags & BPF_F_MMAPABLE) { + mmapable_value =alloc_mmapable_selem_value (smap);This probably is not always safe for bpf prog to do. Leaving the
gfp_flagswas not used aside, the bpf local storage is moving to the bpf's memalloc because of https://lore.kernel.org/bpf/[email protected]/Minor point: alloc_mmapable_selem_value's
bpf_map_area_mmapable_alloccall will always call vmalloc under the hood. vmalloc has locks as well, so your point stands.I think I see how this ties into your 'specific kptr type' proposal above. Let me know if this sounds right: if there was a
bpf_mem_alloctype focused on providing vmalloc'd mmap()able memory, we could use it here instead of raw vmalloc and avoid the lock recursion problem linked above. Such an allocator could be used in something likebpf_obj_newto create the__mmapable_kptr- either from BPF prog or mmap path beforeremap_vmalloc_range.
Quoted 3 lines
re:
gfp_flags, looks like verifier is setting this param to eitherGFP_ATOMICorGFP_KERNEL. Looks like we should not allowGFP_KERNELallocs here?
Going back to this patch, not sure what does it take to make bpf_mem_alloc() mmap()able. May be we can limit the blast radius for now, like limit this alloc to the user space mmap() call for now. Does it fit your use case? A whitelist for bpf prog could also be created later if needed.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
kernel/bpf/bpf_local_storage.c +4
-0+if (!mmapable_value) + goto err_out; + } +
Re: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
Quoted 87 lines · Martin KaFai Lau
Quoted 39 lines · David Marchevsky
Quoted 36 lines · Martin KaFai Lau
Quoted 9 lines · Dave Marchevsky
include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;The pages (that can be mmap'ed later) feel like a specific kind of kptr.
Have you thought about allowing a kptr (pointing to some pages that can be mmap'ed later) to be stored as one of the members of the map's value as a kptr.
bpf_local_storage_mapis one of the maps that supports kptr.struct normal_and_mmap_value { int some_int; int __percpu_kptr *some_cnts; struct bpf_mmap_page __kptr *some_stats; };struct mmap_only_value { struct bpf_mmap_page __kptr *some_stats; };Quoted 7 lines
Quoted 2 lines
[ ... ]
This is an intriguing idea. For conciseness I'll call this specific kind of kptr 'mmapable kptrs' for the rest of this message. Below is more of a brainstorming dump than a cohesive response, separate trains of thought are separated by two newlines.
Thanks for bearing with me while some ideas could be crazy. I am trying to see how this would look like for other local storage, sk and inode. Allocating a page for each sk will not be nice for server with half a million sk(s). e.g. half a million sk(s) sharing a few bandwidth policies or a few tuning parameters. Creating something mmap'able to the user space and also sharable among many sk(s) will be useful.
Quoted 13 lines
My initial thought upon seeing struct
normal_and_mmap_valuewas to note that we currently don't support mmaping formap_valuetypes with _any_ special fields ('special' as determined bybtf_parse_fields). But IIUC you're actually talking about exposing thesome_statspointee memory via mmap, not the containing struct with kptr fields. That is, for maps that support these kptrs, mmap()ing a map with value type structnormal_and_mmap_valuewould return thesome_statspointer value, and likely initialize the pointer similarly toBPF_LOCAL_STORAGE_GET_F_CREATElogic in this patch. We'd only be able to support one such mmapable kptr field per mapval type, but that isn't a dealbreaker.Quoted 3 lines
Some maps, like
task_storage, would only supportmmap()on amap_valuewith mmapable kptr field, as mmap()ing the mapval itself doesn't make sense or is unsafe. Seems like arraymap would do the opposite, onlyChanging direction a bit since arraymap is brought up. :)
arraymap supports
BPF_F_MMAPABLE. If the local storage map's value can store an arraymap as kptr, the bpf prog should be able to access it as a map. More like the current map-in-map setup. The arraymap can be used as regular map in the user space also (like pinning). It may need some btf plumbing to tell the value type of the arrayamp to the verifier.The syscall
bpf_map_update_elem(task_storage_map_fd, &task_pidfd, &value, flags)can be used where thevalue->array_mmapinitialized as anarraymap_fd. This will limit the arraymap kptr update only from the syscall side which seems to be your usecase also? Allocating the arraymap from the bpf prog side needs some thoughts and need a whitelist.
Quoted 6 lines
The same goes for the syscall
bpf_map_lookup_elem(task_storage_map_fd, &task_pidfd, &value). The kernel can return a fd invalue->array_mmap. May be we can create a libbpf helper to free the fd(s) resources held in the looked-up value by using the value's btf.The
bpf_local_storage_mapside probably does not need to supportmmap()then.
Martin,
that's an interesting idea!
I kinda like it and I think it's worth exploring further.
I think the main quirk of the proposed mmap-of-task-local-storage is using 'current' task as an implicit 'key' in task local storage map. It fits here, but I'm not sure it addresses sched-ext use case.
Tejun, David,
could you please chime in ?
Do you think mmap(..., task_local_storage_map_fd, ...) that returns a page that belongs to current task only is enough ?
If not we need to think through how to mmap local storage of other tasks. One proposal was to use pgoff to carry the key somehow like io-uring does, but if we want to generalize that the pgoff approach falls apart if we want __mmapable_kptr to work like Martin is proposing above, since the key will not fit in 64-bit of pgoff.
Maybe we need an office hours slot to discuss. This looks to be a big topic. Not sure we can converge over email.
Just getting everyone on the same page will take a lot of email reading.
Re: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
Quoted 114 lines · Alexei Starovoitov
Quoted 87 lines · Martin KaFai Lau
Quoted 39 lines · David Marchevsky
Quoted 36 lines · Martin KaFai Lau
Quoted 9 lines · Dave Marchevsky
include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;The pages (that can be mmap'ed later) feel like a specific kind of kptr.
Have you thought about allowing a kptr (pointing to some pages that can be mmap'ed later) to be stored as one of the members of the map's value as a kptr.
bpf_local_storage_mapis one of the maps that supports kptr.struct normal_and_mmap_value { int some_int; int __percpu_kptr *some_cnts; struct bpf_mmap_page __kptr *some_stats; };struct mmap_only_value { struct bpf_mmap_page __kptr *some_stats; };Quoted 7 lines
Quoted 2 lines
[ ... ]
This is an intriguing idea. For conciseness I'll call this specific kind of kptr 'mmapable kptrs' for the rest of this message. Below is more of a brainstorming dump than a cohesive response, separate trains of thought are separated by two newlines.
Thanks for bearing with me while some ideas could be crazy. I am trying to see how this would look like for other local storage, sk and inode. Allocating a page for each sk will not be nice for server with half a million sk(s). e.g. half a million sk(s) sharing a few bandwidth policies or a few tuning parameters. Creating something mmap'able to the user space and also sharable among many sk(s) will be useful.
Quoted 13 lines
My initial thought upon seeing struct
normal_and_mmap_valuewas to note that we currently don't support mmaping formap_valuetypes with _any_ special fields ('special' as determined bybtf_parse_fields). But IIUC you're actually talking about exposing thesome_statspointee memory via mmap, not the containing struct with kptr fields. That is, for maps that support these kptrs, mmap()ing a map with value type structnormal_and_mmap_valuewould return thesome_statspointer value, and likely initialize the pointer similarly toBPF_LOCAL_STORAGE_GET_F_CREATElogic in this patch. We'd only be able to support one such mmapable kptr field per mapval type, but that isn't a dealbreaker.Quoted 3 lines
Some maps, like
task_storage, would only supportmmap()on amap_valuewith mmapable kptr field, as mmap()ing the mapval itself doesn't make sense or is unsafe. Seems like arraymap would do the opposite, onlyChanging direction a bit since arraymap is brought up. :)
arraymap supports
BPF_F_MMAPABLE. If the local storage map's value can store an arraymap as kptr, the bpf prog should be able to access it as a map. More like the current map-in-map setup. The arraymap can be used as regular map in the user space also (like pinning). It may need some btf plumbing to tell the value type of the arrayamp to the verifier.The syscall
bpf_map_update_elem(task_storage_map_fd, &task_pidfd, &value, flags)can be used where thevalue->array_mmapinitialized as anarraymap_fd. This will limit the arraymap kptr update only from the syscall side which seems to be your usecase also? Allocating the arraymap from the bpf prog side needs some thoughts and need a whitelist.Quoted 6 lines
The same goes for the syscall
bpf_map_lookup_elem(task_storage_map_fd, &task_pidfd, &value). The kernel can return a fd invalue->array_mmap. May be we can create a libbpf helper to free the fd(s) resources held in the looked-up value by using the value's btf.The
bpf_local_storage_mapside probably does not need to supportmmap()then.Martin,
that's an interesting idea!
I kinda like it and I think it's worth exploring further.I think the main quirk of the proposed mmap-of-task-local-storage is using 'current' task as an implicit 'key' in task local storage map. It fits here, but I'm not sure it addresses sched-ext use case.
Tejun, David,
could you please chime in ?
Do you thinkmmap(..., task_local_storage_map_fd, ...)that returns a page that belongs to current task only is enough ?If not we need to think through how to mmap local storage of other tasks. One proposal was to use pgoff to carry the key somehow like io-uring does, but if we want to generalize that the pgoff approach falls apart if we want
__mmapable_kptrto work like Martin is proposing above, since the key will not fit in 64-bit of pgoff.
Quoted 3 lines
Maybe we need an office hours slot to discuss. This looks to be a big topic. Not sure we can converge over email.
Just getting everyone on the same page will take a lot of email reading.
Meta BPF folks were all in one place for reasons unrelated to this thread, and decided to have a design discussion regarding this mmapable task_local storage implementation and other implementation ideas discussed in this thread. I will summarize the discussion below. We didn't arrive at a confident conclusion, though, so plenty of time for others to chime in and for proper office hours discussion to happen if necessary. Below, anything attributed to a specific person is paraphrased from my notes, there will certainly be errors / misrememberings.
mmapable task_local storage design discussion 11/29
We first summarized approaches that were discussed in this thread:
1) Current implementation in this series
2) pseudo-map_in_map, kptr arraymap type:
struct mmapable_data_map {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(map_flags, BPF_F_MMAPABLE);
__uint(max_entries, 1);
__type(key, __u32);
__type(value, __u64);
};
struct my_mapval {
int whatever;
struct bpf_arraymap __kptr __arraymap_type(mmapable_data_map) *m;
/* Need special logic to support getting / updating above field from userspace (as fd) */
};3) "mmapable page" kptr type, or "mmapable kptr" tag
struct my_mapval {
int whatever;
struct bpf_mmapable_page *m;
};
struct my_mapval2 { /* Separate approach than my_mapval above */
struct bpf_spin_lock l;
int some_int;
struct my_type __mmapable_kptr *some_stats;
};Points of consideration regardless of implementation approach:
* mmap() syscall's return address must be page-aligned. If we want to reduce / eliminate wasted memory by supporting packing of multiple mapvals onto single page, need to be able to return non-page-aligned addr. Using a BPF syscall subcommand in lieu of or in addition to mmap() syscall would be a way to support this.
* Dave wants to avoid implementing packing at all, but having a reasonable path forward would be nice in case actual usecase arises.
* Andrii suggested a new actual mmap syscall supporting passing of custom params, useful for any subsystem using mmap in nontraditional ways. This was initially a response to "use offset to pass task id" discussion re: selecting non-current task.
* How orthogonal is Martin's "kptr to arraymap" suggestion from the general mmapable local_storage goal? Is there a world where we choose a different approach for this work, and then to "kptr to arraymap" independently later?
The above was mostly summary of existing discussion in this thread. Rest of discussion flowed from there.
Q&A:
- Do we need to be able to query other tasks' mapvals? (for David Vernet / Tejun Heo)
TJ had a usecase where this might've been necessary, but rewrote.
David: Being able to do this in general, aside from TJ's specific case, would be useful. David provided an example from ghOSt project - central scheduling. Another example: Folly runtime framework, farms out work to worker threads, might want to tag them.
- Which usecases actually care about avoiding syscall overhead?
Alexei: Most services that would want to use this functionality to tag their tasks don't need it, as they just set the value once.
Dave: Some tracing usecases (e.g. strobemeta) need it.
- Do we want to use mmap() in current-task-only limited way, or do BPF subcommand or something more exotic?
TJ: What if bpf subcommand returns FD that can be mmap'd. fd identifies which task_local storage elem is mmap'd. Subcommand:bpf_map_lookup_elem_fd(map *, u64 elem_id).
Alexei: Such a thing should return fd to arbitrary mapval, and should support other common ops (open, close, etc.).
David: What's the problem w/ having fd that only supports mmap for now?
TJ: 'Dangling' fds exist in some usecases already
Discussion around the bpf_map_lookup_elem_fd idea continued for quite a while. Folks liked that it avoids the "can only have one mmapable field" issue from proposal (3) above, without making any implicit assumptions.
Alexei: Can we instead have pointer to userspace blob - similar to rseq
- to avoid wasting most of page?
TJ: My instinct is to stick to something more generic, would rather pay 4k.
Discussion around userspace pointer continued for a while as well, ending in the conclusion that we should take a look at using get_user_pages, perhaps wrapping such functionality in a 'guppable' kptr type. Folks liked the 'guppable' idea as it sort-of avoids the wasted memory issue, pushing the details to userspace, and punts on working out a path forward for mmap interface, which other implementation ideas require.
Action items based on convo, priority order:
- Think more about / prototype 'guppable' kptr idea
- If the above has issues, try bpf_map_lookup_elem_fd
- If both above have issues, consider earlier approaches
I will start tackling these soon.
kernel test robot
build botRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave Marchevskykernel test robot noticed the following build warnings:
Hi Dave,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Dave-Marchevsky/bpf-Support-BPF_F_MMAPABLE-task_local-storage/20231121-020345 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20231120175925.733167-2-davemarchevsky%40fb.com patch subject: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage config: sparc-randconfig-r081-20231121 (https://download.01.org/0day-ci/archive/20231121/[email protected]/config) compiler: sparc-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231121/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> kernel/bpf/syscall.c:407:20: warning: no previous prototype for 'bpf_map_get_memcg' [-Wmissing-prototypes]
407 | struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map)
| ^~~~~~~~~~~~~~~~~
--
>> kernel/bpf/bpf_local_storage.c:30:7: warning: no previous prototype for 'alloc_mmapable_selem_value' [-Wmissing-prototypes]
30 | void *alloc_mmapable_selem_value(struct bpf_local_storage_map *smap)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
kernel/bpf/bpf_local_storage.c: In function 'bpf_selem_free_rcu':
kernel/bpf/bpf_local_storage.c:288:39: warning: variable 'smap' set but not used [-Wunused-but-set-variable]
288 | struct bpf_local_storage_map *smap;
| ^~~~vim +/bpf_map_get_memcg +407 kernel/bpf/syscall.c
406
> 407 struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map)
408 {
409 if (map->objcg)
410 return get_mem_cgroup_from_objcg(map->objcg);
411
412 return root_mem_cgroup;
413 }
414
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave Marchevskykernel test robot noticed the following build errors:
Hi Dave,
kernel test robot noticed the following build errors:
[auto build test ERROR on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Dave-Marchevsky/bpf-Support-BPF_F_MMAPABLE-task_local-storage/20231121-020345 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20231120175925.733167-2-davemarchevsky%40fb.com patch subject: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage config: i386-buildonly-randconfig-005-20231121 (https://download.01.org/0day-ci/archive/20231121/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231121/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
ld: kernel/bpf/bpf_local_storage.o: in function `alloc_mmapable_selem_value':
bpf_local_storage.c:(.text+0x207): undefined reference to `bpf_map_get_memcg'
ld: kernel/bpf/bpf_local_storage.o: in function `bpf_selem_alloc':
bpf_local_storage.c:(.text+0x410): undefined reference to `bpf_map_get_memcg'
>> ld: bpf_local_storage.c:(.text+0x543): undefined reference to `bpf_map_get_memcg'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave Marchevskykernel test robot noticed the following build warnings:
Hi Dave,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Dave-Marchevsky/bpf-Support-BPF_F_MMAPABLE-task_local-storage/20231121-020345 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20231120175925.733167-2-davemarchevsky%40fb.com patch subject: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage config: x86_64-randconfig-121-20231121 (https://download.01.org/0day-ci/archive/20231121/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231121/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> kernel/bpf/bpf_local_storage.c:30:6: sparse: sparse: symbol 'alloc_mmapable_selem_value' was not declared. Should it be static? >> kernel/bpf/bpf_local_storage.c:291:14: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct bpf_local_storage_map *smap @@ got struct bpf_local_storage_map [noderef] __rcu *smap @@
kernel/bpf/bpf_local_storage.c:291:14: sparse: expected struct bpf_local_storage_map *smap
kernel/bpf/bpf_local_storage.c:291:14: sparse: got struct bpf_local_storage_map [noderef] __rcu *smap
kernel/bpf/bpf_local_storage.c: note: in included file (through include/linux/mmzone.h, include/linux/gfp.h, include/linux/umh.h, include/linux/kmod.h, ...):
include/linux/page-flags.h:242:46: sparse: sparse: self-comparison always evaluates to false
vim +/alloc_mmapable_selem_value +30 kernel/bpf/bpf_local_storage.c
29
> 30 void *alloc_mmapable_selem_value(struct bpf_local_storage_map *smap)
31 {
32 struct mem_cgroup *memcg, *old_memcg;
33 void *ptr;
34
35 memcg = bpf_map_get_memcg(&smap->map);
36 old_memcg = set_active_memcg(memcg);
37 ptr = bpf_map_area_mmapable_alloc(PAGE_ALIGN(smap->map.value_size),
38 NUMA_NO_NODE);
39 set_active_memcg(old_memcg);
40 mem_cgroup_put(memcg);
41
42 return ptr;
43 }
44 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave MarchevskyQuoted 1 line · Dave Marchevsky
+void *alloc_mmapable_selem_value(struct
bpf_local_storage_map*smap)
should be static?
kernel/bpf/bpf_task_storage.c static void task_storage_map_free(struct bpf_map *map) · line 331
Quoted 9 lines
kernel/bpf/bpf_task_storage.c +9
-0+static int task_storage_map_mmap (struct bpf_map *map,struct vm_area_struct *vma) +{ +void *data; + +if (!(map->map_flags &BPF_F_MMAPABLE ) || vma->vm_pgoff || + (vma->vm_end - vma->vm_start) < map->value_size) +return -EINVAL ; + +WARN_ON_ONCE (!bpf_rcu_lock_held ());
why? This is mmap() syscall. What is the concern?
kernel/bpf/bpf_task_storage.c static void task_storage_map_free(struct bpf_map *map) · line 334
Quoted 3 lines
kernel/bpf/bpf_task_storage.c +3
-0+bpf_task_storage_lock (); + data =__bpf_task_storage_get (map, current,NULL ,BPF_LOCAL_STORAGE_GET_F_CREATE , +0 , true);
0 for gfp_flags? It probably should be GFP_USER?
Yonghong Song
reviewerRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave MarchevskyQuoted 68 lines · Dave Marchevsky
This patch modifies the generic
bpf_local_storageinfrastructure to support mmapable map values and addsmmap()handling totask_localstorage leveraging this new functionality. A userspace task which mmap's atask_localstorage map will receive a pointer to themap_valuecorresponding to that tasks' key - mmap'ing in other tasks' mapvals is not supported in this patch.Currently, struct
bpf_local_storage_elemcontains both bookkeeping information as well as a structbpf_local_storage_datawith additional bookkeeping information and the actual mapval data. We can't simply map the page containing this struct into userspace. Instead, mmapablelocal_storageuses bpf_local_storage_data's data field to point to the actual mapval, which is allocated separately such that it can be mmapped. Only the mapval lives on the page(s) allocated for it.The lifetime of the
actual_datammapable region is tied to thebpf_local_storage_elemwhich points to it. This doesn't necessarily mean that the pages go away when thebpf_local_storage_elemis free'd - if they're mapped into some userspace process they will remain until unmapped, but are no longer thetask_localstorage's mapval.Implementation details:
* A few small helpers are added to deal with bpf_local_storage_data's 'data' field having different semantics when the
local_storagemap is mmapable. With their help, many of the changes to existing code are purely mechanical (e.g.sdata->databecomessdata_mapval(sdata),selem->elem_sizebecomesselem_bytes_used(selem)).* The map flags are copied into
bpf_local_storage_datawhen its containingbpf_local_storage_elemis alloc'd, since thebpf_local_storage_mapassociated with them may be gone whenbpf_local_storage_datais free'd, and testing flags forBPF_F_MMAPABLEis necessary when free'ing to ensure that the mmapable region is free'd.
* The extra field doesn't change bpf_local_storage_elem's size. There were 48 bytes of padding after thebpf_local_storage_datafield, now there are 40.* Currently,
bpf_local_storage_updatealways creates a newbpf_local_storage_elemfor the 'updated' value - the only exception being if themap_valuehas abpf_spin_lockfield, in which case the spin lock is grabbed instead of the less granularbpf_local_storagelock, and the value updated in place. This inplace update behavior is desired for mmapablelocal_storagemap_valuesas well, since creating a new selem would result in new mmapable pages.* The size of the mmapable pages are accounted for when calling mem_{charge,uncharge}. If the pages are mmap'd into a userspace task
mem_unchargemay be called before they actually go away.Signed-off-by Dave Marchevsky <[email protected]>--- include/linux/bpf_local_storage.h | 14 ++- kernel/bpf/bpf_local_storage.c | 145 ++++++++++++++++++++++++------ kernel/bpf/bpf_task_storage.c | 35 ++++++-- kernel/bpf/syscall.c | 2 +- 4 files changed, 163 insertions(+), 33 deletions(-)include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;
include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;
I think we can remove __aligned(8) from 'void *actual_data __aligned(8)' in the above. There are two reasons, first, the first element in the union is aligned(8) and then the rest of union members will be at least aligned 8. second, IIUC, in the code, we have
actual_data = mmapable_value
where mmapable_value has type 'void *'. So actual_data is used to hold a pointer to mmap-ed page, so it only needs pointer type alignment which can already be achieved with 'void *'.
So we can remove __aligned(8) safely. It can also remove
an impression that 'actual_data' has to be 8-byte aligned in 32-bit system although it does not need to be just by 'actual_data' itself.
Quoted 13 lines
include/linux/bpf_local_storage.h +3
-0+ }; };/* Linked to bpf_local_storage and bpf_local_storage_map */ @@ -124,6 +134,8 @@ static struct bpf_local_storage_cache name = { \/* Helper functions for bpf_local_storage */ int bpf_local_storage_map_alloc_check (union bpf_attr *attr); +void *sdata_mapval (struct bpf_local_storage_data *data); +struct bpf_map * bpf_local_storage_map_alloc(union bpf_attr *attr, struct bpf_local_storage_cache *cache,
[...]
Yonghong Song
reviewerRe: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave MarchevskyQuoted 94 lines · Dave Marchevsky
This patch modifies the generic
bpf_local_storageinfrastructure to support mmapable map values and addsmmap()handling totask_localstorage leveraging this new functionality. A userspace task which mmap's atask_localstorage map will receive a pointer to themap_valuecorresponding to that tasks' key - mmap'ing in other tasks' mapvals is not supported in this patch.Currently, struct
bpf_local_storage_elemcontains both bookkeeping information as well as a structbpf_local_storage_datawith additional bookkeeping information and the actual mapval data. We can't simply map the page containing this struct into userspace. Instead, mmapablelocal_storageuses bpf_local_storage_data's data field to point to the actual mapval, which is allocated separately such that it can be mmapped. Only the mapval lives on the page(s) allocated for it.The lifetime of the
actual_datammapable region is tied to thebpf_local_storage_elemwhich points to it. This doesn't necessarily mean that the pages go away when thebpf_local_storage_elemis free'd - if they're mapped into some userspace process they will remain until unmapped, but are no longer thetask_localstorage's mapval.Implementation details:
* A few small helpers are added to deal with bpf_local_storage_data's 'data' field having different semantics when the
local_storagemap is mmapable. With their help, many of the changes to existing code are purely mechanical (e.g.sdata->databecomessdata_mapval(sdata),selem->elem_sizebecomesselem_bytes_used(selem)).* The map flags are copied into
bpf_local_storage_datawhen its containingbpf_local_storage_elemis alloc'd, since thebpf_local_storage_mapassociated with them may be gone whenbpf_local_storage_datais free'd, and testing flags forBPF_F_MMAPABLEis necessary when free'ing to ensure that the mmapable region is free'd.
* The extra field doesn't change bpf_local_storage_elem's size. There were 48 bytes of padding after thebpf_local_storage_datafield, now there are 40.* Currently,
bpf_local_storage_updatealways creates a newbpf_local_storage_elemfor the 'updated' value - the only exception being if themap_valuehas abpf_spin_lockfield, in which case the spin lock is grabbed instead of the less granularbpf_local_storagelock, and the value updated in place. This inplace update behavior is desired for mmapablelocal_storagemap_valuesas well, since creating a new selem would result in new mmapable pages.* The size of the mmapable pages are accounted for when calling mem_{charge,uncharge}. If the pages are mmap'd into a userspace task
mem_unchargemay be called before they actually go away.Signed-off-by Dave Marchevsky <[email protected]>--- include/linux/bpf_local_storage.h | 14 ++- kernel/bpf/bpf_local_storage.c | 145 ++++++++++++++++++++++++------ kernel/bpf/bpf_task_storage.c | 35 ++++++-- kernel/bpf/syscall.c | 2 +- 4 files changed, 163 insertions(+), 33 deletions(-)include/linux/bpf_local_storage.h +13
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ; +/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ; + }; };/* Linked to bpf_local_storage and bpf_local_storage_map */ @@ -124,6 +134,8 @@ static struct bpf_local_storage_cache name = { \/* Helper functions for bpf_local_storage */ int bpf_local_storage_map_alloc_check (union bpf_attr *attr); +void *sdata_mapval (struct bpf_local_storage_data *data); +struct bpf_map * bpf_local_storage_map_alloc(union bpf_attr *attr, struct bpf_local_storage_cache *cache,kernel/bpf/bpf_local_storage.c +0
-1--- a/kernel/bpf/bpf_local_storage.c
[...]
Quoted 20 lines
kernel/bpf/bpf_local_storage.c +2
-2@@ -583,14 +665,14 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, err =bpf_local_storage_alloc (owner, smap, selem, gfp_flags);if (err) {bpf_selem_free (selem, smap, true); -mem_uncharge (smap, owner, smap->elem_size); +mem_uncharge (smap, owner,selem_bytes_used (smap));return ERR_PTR (err); }return SDATA (selem); } -if ((map_flags &BPF_F_LOCK ) && !(map_flags &BPF_NOEXIST )) { +if (can_update_existing_selem (smap, map_flags) && !(map_flags &BPF_NOEXIST )) {/* Hoping to find an old_sdata to do inline update * such that it can avoid taking the local_storage->lock * and changing the lists. @@ -601,8 +683,13 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,if (err)return ERR_PTR (err);
kernel/bpf/bpf_local_storage.c bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, · line 692
Quoted 10 lines
kernel/bpf/bpf_local_storage.c +7
-2if (old_sdata &&selem_linked_to_storage_lockless (SELEM (old_sdata))) { -copy_map_value_locked (&smap->map, old_sdata->data, - value, false); +if (map_flags &BPF_F_LOCK ) +copy_map_value_locked (&smap->map, +sdata_mapval (old_sdata), + value, false); +else +copy_map_value (&smap->map,sdata_mapval (old_sdata), + value);
IIUC, if two 'storage_update' to the same map/key and then these two updates will be serialized due to spin_lock. How about concurrent update for mmap'ed sdata, do we need any protection here?
Quoted 34 lines
kernel/bpf/bpf_local_storage.c +7
-3return old_sdata; } }@@ -633,8 +720,8 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, goto unlock;if (old_sdata && (map_flags &BPF_F_LOCK )) { -copy_map_value_locked (&smap->map, old_sdata->data, value, - false); +copy_map_value_locked (&smap->map,sdata_mapval (old_sdata), + value, false); selem =SELEM (old_sdata); goto unlock; }@@ -656,7 +743,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap, unlock:raw_spin_unlock_irqrestore (&local_storage->lock, flags);if (alloc_selem) { -mem_uncharge (smap, owner, smap->elem_size); +mem_uncharge (smap, owner,selem_bytes_used (smap));bpf_selem_free (alloc_selem, smap, true); }return err ?ERR_PTR (err) :SDATA (selem);@@ -707,6 +794,10 @@ int bpf_local_storage_map_alloc_check(union bpf_attr *attr)if (attr->value_size >BPF_LOCAL_STORAGE_MAX_VALUE_SIZE )return -E2BIG ; +if ((attr->map_flags &BPF_F_MMAPABLE ) && + attr->map_type !=BPF_MAP_TYPE_TASK_STORAGE ) +return -EINVAL ; +return 0 ; }
[...]
Re: [PATCH v1 bpf-next 1/2] bpf: Support BPF_F_MMAPABLE task_local storage
In reply to Dave MarchevskyQuoted 22 lines · Dave Marchevsky
This patch modifies the generic
bpf_local_storageinfrastructure to support mmapable map values and addsmmap()handling totask_localstorage leveraging this new functionality. A userspace task which mmap's atask_localstorage map will receive a pointer to themap_valuecorresponding to that tasks' key - mmap'ing in other tasks' mapvals is not supported in this patch.Currently, struct
bpf_local_storage_elemcontains both bookkeeping information as well as a structbpf_local_storage_datawith additional bookkeeping information and the actual mapval data. We can't simply map the page containing this struct into userspace. Instead, mmapablelocal_storageuses bpf_local_storage_data's data field to point to the actual mapval, which is allocated separately such that it can be mmapped. Only the mapval lives on the page(s) allocated for it.The lifetime of the
actual_datammapable region is tied to thebpf_local_storage_elemwhich points to it. This doesn't necessarily mean that the pages go away when thebpf_local_storage_elemis free'd - if they're mapped into some userspace process they will remain until unmapped, but are no longer thetask_localstorage's mapval.
Quoted 7 lines
Implementation details:
* A few small helpers are added to deal with bpf_local_storage_data's 'data' field having different semantics when the
local_storagemap is mmapable. With their help, many of the changes to existing code are purely mechanical (e.g.sdata->databecomessdata_mapval(sdata),selem->elem_sizebecomesselem_bytes_used(selem)).
might be worth doing this as a pre-patch with no functional changes to make the main change a bit smaller and more focused?
Quoted 40 lines
* The map flags are copied into
bpf_local_storage_datawhen its containingbpf_local_storage_elemis alloc'd, since thebpf_local_storage_mapassociated with them may be gone whenbpf_local_storage_datais free'd, and testing flags forBPF_F_MMAPABLEis necessary when free'ing to ensure that the mmapable region is free'd.
* The extra field doesn't change bpf_local_storage_elem's size. There were 48 bytes of padding after thebpf_local_storage_datafield, now there are 40.* Currently,
bpf_local_storage_updatealways creates a newbpf_local_storage_elemfor the 'updated' value - the only exception being if themap_valuehas abpf_spin_lockfield, in which case the spin lock is grabbed instead of the less granularbpf_local_storagelock, and the value updated in place. This inplace update behavior is desired for mmapablelocal_storagemap_valuesas well, since creating a new selem would result in new mmapable pages.* The size of the mmapable pages are accounted for when calling mem_{charge,uncharge}. If the pages are mmap'd into a userspace task
mem_unchargemay be called before they actually go away.Signed-off-by Dave Marchevsky <[email protected]>--- include/linux/bpf_local_storage.h | 14 ++- kernel/bpf/bpf_local_storage.c | 145 ++++++++++++++++++++++++------ kernel/bpf/bpf_task_storage.c | 35 ++++++-- kernel/bpf/syscall.c | 2 +- 4 files changed, 163 insertions(+), 33 deletions(-)include/linux/bpf_local_storage.h +0
-1@@ -69,7 +69,17 @@ struct bpf_local_storage_data {* the number of cachelines accessed during the cache hit case. */ struct bpf_local_storage_map __rcu *smap; -u8 data[]__aligned(8) ;
include/linux/bpf_local_storage.h struct bpf_local_storage_data { · line 81
Quoted 10 lines
include/linux/bpf_local_storage.h +10
-0+/* Need to duplicate smap's map_flags as smap may be gone when +* it's time to free bpf_local_storage_data +*/ +u64 smap_map_flags; +/* If BPF_F_MMAPABLE, this is a void * to separately-alloc'd data +* Otherwise the actual mapval data lives here +*/ +union { + DECLARE_FLEX_ARRAY(u8, data) __aligned(8); +void *actual_data__aligned(8) ;
I don't know if it's the issue, but probably best to keep FLEX_ARRAY member the last even within the union, just in case if some tool doesn't handle FLEX_ARRAY not being last line number-wise?
Quoted 44 lines
include/linux/bpf_local_storage.h +3
-0+ }; };/* Linked to bpf_local_storage and bpf_local_storage_map */ @@ -124,6 +134,8 @@ static struct bpf_local_storage_cache name = { \/* Helper functions for bpf_local_storage */ int bpf_local_storage_map_alloc_check (union bpf_attr *attr); +void *sdata_mapval (struct bpf_local_storage_data *data); +struct bpf_map * bpf_local_storage_map_alloc(union bpf_attr *attr, struct bpf_local_storage_cache *cache,kernel/bpf/bpf_local_storage.c +15
-1@@ -15,7 +15,8 @@#include <linux/rcupdate_trace.h> #include <linux/rcupdate_wait.h> -#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE) +#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \ + (BPF_F_NO_PREALLOC | BPF_F_CLONE | BPF_F_MMAPABLE)static struct bpf_local_storage_map_bucket * select_bucket(struct bpf_local_storage_map *smap,@@ -24,6 +25,51 @@ select_bucket(struct bpf_local_storage_map *smap,return &smap->buckets[hash_ptr (selem, smap->bucket_log)]; } +struct mem_cgroup *bpf_map_get_memcg (const struct bpf_map *map); + +void *alloc_mmapable_selem_value (struct bpf_local_storage_map *smap) +{ +struct mem_cgroup *memcg, *old_memcg; +void *ptr; + + memcg =bpf_map_get_memcg (&smap->map); + old_memcg =set_active_memcg (memcg); + ptr =bpf_map_area_mmapable_alloc (PAGE_ALIGN (smap->map.value_size), +NUMA_NO_NODE ); +set_active_memcg (old_memcg); +mem_cgroup_put (memcg);
kernel/bpf/bpf_local_storage.c select_bucket(struct bpf_local_storage_map *smap, · line 50
Quoted 10 lines
kernel/bpf/bpf_local_storage.c +10
-0+ +return ptr; +} + +void *sdata_mapval (struct bpf_local_storage_data *data) +{ +if (data->smap_map_flags &BPF_F_MMAPABLE ) +return data->actual_data; +return &data->data; +}
given this being potentially high-frequency helper called from other .o files and it is simple, should this be a static inline in .h header instead?
Quoted 9 lines
kernel/bpf/bpf_local_storage.c +9
-0+ +static size_t sdata_data_field_size (struct bpf_local_storage_map *smap, +struct bpf_local_storage_data *data) +{ +if (smap->map.map_flags &BPF_F_MMAPABLE ) +return sizeof (void *); +return (size_t )smap->map.value_size; +} +
[...]
Quoted 13 lines
kernel/bpf/bpf_task_storage.c +1
-0@@ -90,6 +90,7 @@ void bpf_task_storage_free(struct task_struct *task)static void *bpf_pid_task_storage_lookup_elem (struct bpf_map *map,void *key) {struct bpf_local_storage_data *sdata; +struct bpf_local_storage_map *smap;struct task_struct *task;unsigned int f_flags;struct pid *pid;
kernel/bpf/bpf_task_storage.c static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key) · line 118
Quoted 5 lines
kernel/bpf/bpf_task_storage.c +1
-1@@ -114,7 +115,8 @@ static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key) sdata =task_storage_lookup (task, map, true);bpf_task_storage_unlock ();put_pid (pid); -return sdata ? sdata->data :NULL ; + smap = (struct bpf_local_storage_map *)map;
smap seems unused?
kernel/bpf/bpf_task_storage.c static void *__bpf_task_storage_get(struct bpf_map *map, · line 216
Quoted 11 lines
kernel/bpf/bpf_task_storage.c +3
-0+return sdata ?sdata_mapval (sdata) :NULL ; out:put_pid (pid);return ERR_PTR (err);@@ -209,18 +211,19 @@ static void *__bpf_task_storage_get(struct bpf_map *map, u64 flags, gfp_t gfp_flags, bool nobusy) { struct bpf_local_storage_data *sdata; +struct bpf_local_storage_map *smap; + smap = (struct bpf_local_storage_map *)map;
used much later, so maybe move it down? or just not change this part?
Quoted 19 lines
kernel/bpf/bpf_task_storage.c +4
-5sdata =task_storage_lookup (task, map, nobusy);if (sdata) -return sdata->data; +return sdata_mapval (sdata);/* only allocate new storage, when the task is refcounted */ if (refcount_read (&task->usage) && (flags &BPF_LOCAL_STORAGE_GET_F_CREATE ) && nobusy) { - sdata =bpf_local_storage_update ( - task, (struct bpf_local_storage_map *)map, value, -BPF_NOEXIST , gfp_flags); -return IS_ERR (sdata) ?NULL : sdata->data; + sdata =bpf_local_storage_update (task, smap, value, +BPF_NOEXIST , gfp_flags); +return IS_ERR (sdata) ?NULL :sdata_mapval (sdata); }return NULL ;
kernel/bpf/bpf_task_storage.c static void task_storage_map_free(struct bpf_map *map) · line 328
Quoted 9 lines
kernel/bpf/bpf_task_storage.c +6
-0@@ -317,6 +320,25 @@ static void task_storage_map_free(struct bpf_map *map)bpf_local_storage_map_free (map, &task_cache, &bpf_task_storage_busy); } +static int task_storage_map_mmap (struct bpf_map *map,struct vm_area_struct *vma) +{ +void *data; + +if (!(map->map_flags &BPF_F_MMAPABLE ) || vma->vm_pgoff || + (vma->vm_end - vma->vm_start) < map->value_size)
so we enforce that vm_pgoff is zero, that's understandable. But why disallowing mmaping only a smaller portion of map value?
Also, more importantly, I think you should reject vma->vm_end - vma->vm_start > PAGE_ALIGN(map->value_size), no?
Another question. I might have missed it, but where do we disallow mmap()'ing maps that have "special" fields in map_value, like kptrs, spin_locks, timers, etc?
Quoted 40 lines
kernel/bpf/bpf_task_storage.c +14
-0+return -EINVAL ; + +WARN_ON_ONCE (!bpf_rcu_lock_held ()); +bpf_task_storage_lock (); + data =__bpf_task_storage_get (map, current,NULL ,BPF_LOCAL_STORAGE_GET_F_CREATE , +0 , true); +bpf_task_storage_unlock (); +if (!data) +return -EINVAL ; + +return remap_vmalloc_range (vma, data, vma->vm_pgoff); +} + BTF_ID_LIST_GLOBAL_SINGLE(bpf_local_storage_map_btf_id, struct, bpf_local_storage_map)const struct bpf_map_ops task_storage_map_ops = { .map_meta_equal = bpf_map_meta_equal,@@ -331,6 +353,7 @@ const struct bpf_map_ops task_storage_map_ops = { .map_mem_usage = bpf_local_storage_map_mem_usage, .map_btf_id = &bpf_local_storage_map_btf_id[0], .map_owner_storage_ptr = task_storage_ptr, + .map_mmap = task_storage_map_mmap, };const struct bpf_func_proto bpf_task_storage_get_recur_proto = {kernel/bpf/syscall.c +1
-2@@ -404,7 +404,7 @@ static void bpf_map_release_memcg(struct bpf_map *map)obj_cgroup_put (map->objcg); } -static struct mem_cgroup *bpf_map_get_memcg (const struct bpf_map *map) +struct mem_cgroup *bpf_map_get_memcg (const struct bpf_map *map) {if (map->objcg)return get_mem_cgroup_from_objcg (map->objcg); --2.34.1
PATCH 2/2 selftests/bpf: Add test exercising mmapable task_local_storage
In reply to Dave MarchevskyThis patch tests mmapable task_local storage functionality added earlier in the series. The success tests focus on verifying correctness of the various ways of reading from and writing to mmapable task_local storage:
* Write through mmap'd region should be visible when BPF program makes bpf_task_storage_get call
* If BPF program reads-and-incrs the mapval, the new value should be visible when userspace reads from mmap'd region or does map_lookup_elem call
* If userspace does map_update_elem call, new value should be visible when userspace reads from mmap'd region or does map_lookup_elem call
* After bpf_map_delete_elem, reading from mmap'd region should still succeed, but map_lookup_elem w/o BPF_LOCAL_STORAGE_GET_F_CREATE flag should fail
* After bpf_map_delete_elem, creating a new map_val via mmap call should return a different memory region
--- .../bpf/prog_tests/task_local_storage.c | 177 ++++++++++++++++++ .../bpf/progs/task_local_storage__mmap.c | 59 ++++++ .../bpf/progs/task_local_storage__mmap.h | 7 + .../bpf/progs/task_local_storage__mmap_fail.c | 39 ++++ 4 files changed, 282 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap.c create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap.h create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +177 -0
@@ -5,14 +5,19 @@#include <unistd.h> #include <sched.h> #include <pthread.h> +#include <sys/mman.h> /* For mmap and associated flags */ #include <sys/syscall.h> /* For SYS_xxx definitions */ #include <sys/types.h> #include <test_progs.h> +#include <network_helpers.h> #include "task_local_storage_helpers.h" #include "task_local_storage.skel.h" #include "task_local_storage_exit_creds.skel.h" +#include "task_local_storage__mmap.skel.h" +#include "task_local_storage__mmap_fail.skel.h" #include "task_ls_recursion.skel.h" #include "task_storage_nodeadlock.skel.h" +#include "progs/task_local_storage__mmap.h" static void test_sys_enter_exit (void ) {
@@ -40,6 +45,173 @@ static void test_sys_enter_exit(void)task_local_storage__destroy (skel); } +static int basic_mmapable_read_write (struct task_local_storage__mmap *skel, +long *mmaped_task_local) +{ +int err; + + *mmaped_task_local =42 ; + + err =task_local_storage__mmap__attach (skel); +if (!ASSERT_OK (err,"skel_attach" )) +return -1 ; + +syscall (SYS_gettid); +ASSERT_EQ (skel->bss->mmaped_mapval,42 ,"mmaped_mapval" ); + +/* Incr from userspace should be visible when BPF prog reads */ + *mmaped_task_local = *mmaped_task_local +1 ; +syscall (SYS_gettid); +ASSERT_EQ (skel->bss->mmaped_mapval,43 ,"mmaped_mapval_user_incr" ); + +/* Incr from BPF prog should be visible from userspace */ + skel->bss->read_and_incr =1 ; +syscall (SYS_gettid); +ASSERT_EQ (skel->bss->mmaped_mapval,44 ,"mmaped_mapval_bpf_incr" ); +ASSERT_EQ (skel->bss->mmaped_mapval, *mmaped_task_local,"bpf_incr_eq" ); + skel->bss->read_and_incr =0 ; + +return 0 ; +} + +static void test_sys_enter_mmap (void ) +{ +struct task_local_storage__mmap *skel; +long *task_local, *task_local2, value; +int err, task_fd, map_fd; + + task_local = task_local2 = (long *)-1 ; + task_fd =sys_pidfd_open (getpid (),0 ); +if (!ASSERT_NEQ (task_fd,-1 ,"sys_pidfd_open" )) +return ; + + skel =task_local_storage__mmap__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open_and_load" )) { +close (task_fd); +return ; + } + + map_fd =bpf_map__fd (skel->maps.mmapable); + task_local =mmap (NULL ,sizeof (long ),PROT_READ |PROT_WRITE , +MAP_SHARED , map_fd,0 ); +if (!ASSERT_OK_PTR (task_local,"mmap_task_local_storage" )) + goto out; + + err =basic_mmapable_read_write (skel, task_local); +if (!ASSERT_OK (err,"basic_mmapable_read_write" )) + goto out; + + err =bpf_map_lookup_elem (map_fd, &task_fd, &value); +if (!ASSERT_OK (err,"bpf_map_lookup_elem" ) || + !ASSERT_EQ (value,44 ,"bpf_map_lookup_elem value" )) + goto out; + + value =148 ; +bpf_map_update_elem (map_fd, &task_fd, &value,BPF_EXIST ); +if (!ASSERT_EQ (READ_ONCE (*task_local),148 ,"mmaped_read_after_update" )) + goto out; + + err =bpf_map_lookup_elem (map_fd, &task_fd, &value); +if (!ASSERT_OK (err,"bpf_map_lookup_elem" ) || + !ASSERT_EQ (value,148 ,"bpf_map_lookup_elem value" )) + goto out; + +/* The mmapable page is not released by map_delete_elem, but no longer +* linked to local_storage +*/ + err =bpf_map_delete_elem (map_fd, &task_fd); +if (!ASSERT_OK (err,"bpf_map_delete_elem" ) || + !ASSERT_EQ (READ_ONCE (*task_local),148 ,"mmaped_read_after_delete" )) + goto out; + + err =bpf_map_lookup_elem (map_fd, &task_fd, &value); +if (!ASSERT_EQ (err, -ENOENT ,"bpf_map_lookup_elem_after_delete" )) + goto out; + +task_local_storage__mmap__destroy (skel); + +/* The mmapable page is not released when __destroy unloads the map. +* It will stick around until we munmap it +*/ + *task_local =-999 ; + +/* Although task_local's page is still around, it won't be reused */ + skel =task_local_storage__mmap__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open_and_load2" )) +return ; + + map_fd =bpf_map__fd (skel->maps.mmapable); + err =task_local_storage__mmap__attach (skel); +if (!ASSERT_OK (err,"skel_attach2" )) + goto out; + + skel->bss->read_and_incr =1 ; + skel->bss->create_flag =BPF_LOCAL_STORAGE_GET_F_CREATE ; +syscall (SYS_gettid); +ASSERT_EQ (skel->bss->mmaped_mapval,1 ,"mmaped_mapval2" ); + + skel->bss->read_and_incr =0 ; + task_local2 =mmap (NULL ,sizeof (long ),PROT_READ |PROT_WRITE , +MAP_SHARED , map_fd,0 ); +if (!ASSERT_OK_PTR (task_local,"mmap_task_local_storage2" )) + goto out; + +if (!ASSERT_NEQ (task_local, task_local2,"second_mmap_address" )) + goto out; + +ASSERT_EQ (READ_ONCE (*task_local2),1 ,"mmaped_mapval2_bpf_create_incr" ); + +out: +close (task_fd); +if (task_local >0 ) +munmap (task_local,sizeof (long )); +if (task_local2 >0 ) +munmap (task_local2,sizeof (long )); +task_local_storage__mmap__destroy (skel); +} + +static void test_sys_enter_mmap_big_mapval (void ) +{ +struct two_page_struct *task_local, value; +struct task_local_storage__mmap *skel; +int task_fd, map_fd, err; + + task_local = (struct two_page_struct *)-1 ; + task_fd =sys_pidfd_open (getpid (),0 ); +if (!ASSERT_NEQ (task_fd,-1 ,"sys_pidfd_open" )) +return ; + + skel =task_local_storage__mmap__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open_and_load" )) { +close (task_fd); +return ; + } + map_fd =bpf_map__fd (skel->maps.mmapable_two_pages); + task_local =mmap (NULL ,sizeof (struct two_page_struct ), +PROT_READ |PROT_WRITE ,MAP_SHARED , + map_fd,0 ); +if (!ASSERT_OK_PTR (task_local,"mmap_task_local_storage" )) + goto out; + + skel->bss->use_big_mapval =1 ; + err =basic_mmapable_read_write (skel, &task_local->val); +if (!ASSERT_OK (err,"basic_mmapable_read_write" )) + goto out; + + task_local->c[4096 ] ='z' ; + + err =bpf_map_lookup_elem (map_fd, &task_fd, &value); +if (!ASSERT_OK (err,"bpf_map_lookup_elem" ) || + !ASSERT_EQ (value.val,44 ,"bpf_map_lookup_elem value" )) + goto out; + +out: +close (task_fd); +if (task_local >0 ) +munmap (task_local,sizeof (struct two_page_struct )); +task_local_storage__mmap__destroy (skel); +} +static void test_exit_creds (void ) {struct task_local_storage_exit_creds *skel;
@@ -237,10 +409,15 @@ void test_task_local_storage(void) {if (test__start_subtest ("sys_enter_exit" ))test_sys_enter_exit (); +if (test__start_subtest ("sys_enter_mmap" )) +test_sys_enter_mmap (); +if (test__start_subtest ("sys_enter_mmap_big_mapval" )) +test_sys_enter_mmap_big_mapval ();if (test__start_subtest ("exit_creds" ))test_exit_creds ();if (test__start_subtest ("recursion" ))test_recursion ();if (test__start_subtest ("nodeadlock" ))test_nodeadlock (); +RUN_TESTS (task_local_storage__mmap_fail); }
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c new file +59 -0
59 lines not on this page · raw message
tools/testing/selftests/bpf/progs/task_local_storage__mmap.h new file +7 -0
@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +struct two_page_struct { +long val; +char c[4097 ]; +};
tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC | BPF_F_MMAPABLE); + __type(key, int); + __type(value, long); +} mmapable SEC(".maps"); + +__failure __msg("invalid access to map value, value_size=8 off=8 size=8") +SEC("tp_btf/sys_enter") +long BPF_PROG(fail_read_past_mapval_end, struct pt_regs *regs, long id) +{ +struct task_struct *task; +long *ptr; +long res; + + task =bpf_get_current_task_btf (); +if (!task) +return 1 ; + + ptr =bpf_task_storage_get (&mmapable, task,0 ,0 ); +if (!ptr) +return 3 ; +/* Although mmapable mapval is given an entire page, verifier shouldn't +* allow r/w past end of 'long' type +*/ + res = *(ptr +1 ); + +return res; +}
Signature · 1 line
2.34.1
Re: [PATCH v1 bpf-next 2/2] selftests/bpf: Add test exercising mmapable task_local_storage
Quoted 30 lines · Dave Marchevsky
This patch tests mmapable
task_localstorage functionality added earlier in the series. The success tests focus on verifying correctness of the various ways of reading from and writing to mmapabletask_localstorage:* Write through mmap'd region should be visible when BPF program makes
bpf_task_storage_getcall
* If BPF program reads-and-incrs the mapval, the new value should be visible when userspace reads from mmap'd region or doesmap_lookup_elemcall
* If userspace doesmap_update_elemcall, new value should be visible when userspace reads from mmap'd region or doesmap_lookup_elemcall
* Afterbpf_map_delete_elem, reading from mmap'd region should still succeed, butmap_lookup_elemw/oBPF_LOCAL_STORAGE_GET_F_CREATEflag should fail
* Afterbpf_map_delete_elem, creating a newmap_valvia mmap call should return a different memory regionSigned-off-by Dave Marchevsky <[email protected]>--- .../bpf/prog_tests/task_local_storage.c | 177 ++++++++++++++++++ .../bpf/progs/task_local_storage__mmap.c | 59 ++++++ .../bpf/progs/task_local_storage__mmap.h | 7 + .../bpf/progs/task_local_storage__mmap_fail.c | 39 ++++ 4 files changed, 282 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap.c create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap.h create mode 100644 tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c
[...]
Quoted 32 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c new file +26
-0@@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> +#include "task_local_storage__mmap.h" + +char _license[] SEC("license") = "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC | BPF_F_MMAPABLE); + __type(key, int); + __type(value, long); +} mmapable SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC | BPF_F_MMAPABLE); + __type(key, int); + __type(value, struct two_page_struct); +} mmapable_two_pages SEC(".maps"); + +long mmaped_mapval =0 ; +int read_and_incr =0 ;
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c line 36
Quoted 10 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c new file +10
-0+int create_flag =0 ; +int use_big_mapval =0 ; + +SEC("tp_btf/sys_enter") +int BPF_PROG (on_enter ,struct pt_regs *regs,long id) +{ +struct two_page_struct *big_mapval; +struct task_struct *task; +long *ptr; +
In addition to the note below about idempotency as a desired property, it might be good to *also* add tid/pid filter here to only execute the logic for our process?
Quoted 9 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c new file +9
-0+ task =bpf_get_current_task_btf (); +if (!task) +return 1 ; + +if (use_big_mapval) { + big_mapval =bpf_task_storage_get (&mmapable_two_pages, task,0 , + create_flag); +if (!big_mapval) +return 2 ;
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c line 55
Quoted 10 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c new file +10
-0+ ptr = &big_mapval->val; + }else { + ptr =bpf_task_storage_get (&mmapable, task,0 , create_flag); + } + +if (!ptr) +return 3 ; + +if (read_and_incr) + *ptr = *ptr +1 ;
this seems fragile, this program is not idempotent, which means any extra unexpected syscall might cause problem, right?
Quoted 32 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap.c new file +4
-0+ + mmaped_mapval = *ptr; +return 0 ; +}tools/testing/selftests/bpf/progs/task_local_storage__mmap.h new file +7
-0@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +struct two_page_struct { +long val; +char c[4097 ]; +};tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c new file +9
-0@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_tracing.h> +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c line 19
Quoted 10 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c new file +10
-0+ +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC | BPF_F_MMAPABLE); + __type(key, int); + __type(value, long); +} mmapable SEC(".maps"); + +__failure __msg("invalid access to map value, value_size=8 off=8 size=8") +SEC ("tp_btf/sys_enter" )
let's keep SEC() first, please move __failure and __msg below it
Quoted 23 lines
tools/testing/selftests/bpf/progs/task_local_storage__mmap_fail.c new file +20
-1+long BPF_PROG (fail_read_past_mapval_end ,struct pt_regs *regs,long id) +{ +struct task_struct *task; +long *ptr; +long res; + + task =bpf_get_current_task_btf (); +if (!task) +return 1 ; + + ptr =bpf_task_storage_get (&mmapable, task,0 ,0 ); +if (!ptr) +return 3 ; +/* Although mmapable mapval is given an entire page, verifier shouldn't +* allow r/w past end of 'long' type +*/ + res = *(ptr +1 ); + +return res; +} --2.34.1