Skip to content

bpf: Add bpf_rcu_read_lock() support

Yonghong Song bpf 10 versions – Applied

Status from this message: This series was applied to bpf/bpf-next.git (master)

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v10 Latest4 patches 4 repliesApplied 4/4
  2. v94 patches 6 repliesSuperseded
  3. v84 patches 9 repliesSuperseded
Earlier versions · 7
VersionPostedPatchesRepliesStatus
v74 patches11 repliesSuperseded
v67 patches0 repliesSuperseded
v57 patches2 repliesSuperseded
v47 patches2 repliesSuperseded
v37 patches0 repliesSuperseded
v28 patches8 repliesSuperseded
v15 patches20 repliesSuperseded

Changes in v10 from the cover letter

. if no rcu tag support in vmlinux btf, using bpf_rcu_read_lock/unlock()
will cause verification error.
. at bpf_rcu_read_unlock(), invalidate rcu ptr to PTR_UNTRUSTED
instead of SCALAR_VALUE.
. a few other comment changes and other minor changes.

Patches in v10 All 4 applied

#Subject and filesSizeRepliesApplied
1/4compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))include/linux/compiler_types.h+2 -10 repliesApplied
2/4bpf: Introduce might_sleep field in bpf_func_protoinclude/linux/bpf.h kernel/bpf/bpf_lsm.c kernel/bpf/helpers.c +2 more+14 -40 repliesApplied
3/4bpf: Add kfunc bpf_rcu_read_lock/unlock()include/linux/bpf.h include/linux/bpf_verifier.h kernel/bpf/btf.c +2 more+155 -304 repliesApplied
4/4selftests/bpf: Add tests for bpf_rcu_read_lock()tools/testing/selftests/bpf/DENYLIST.aarch64 tools/testing/selftests/bpf/DENYLIST.s390x tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c +1 more+450 -00 repliesApplied

Reviews trailers per patch

All 4 patches have a Reviewed-by or Acked-by.

PatchMartin KaFai LauKP Singh
1/4A-b A-b
2/4A-b
3/4A-b
4/4A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v10 thread

Cover letter [PATCH bpf-next v10 0/4] bpf: Add bpf_rcu_read_lock() support

Currently, without rcu attribute info in BTF, the verifier treats rcu tagged pointer as a normal pointer. This might be a problem for sleepable program where rcu_read_lock()/unlock() is not available. For example, for a sleepable fentry program, if rcu protected memory access is interleaved with a sleepable helper/kfunc, it is possible the memory access after the sleepable helper/kfunc might be invalid since the object might have been freed then. Even without
a sleepable helper/kfunc, without rcu_read_lock() protection, it is possible that the rcu protected object might be release in the middle of bpf program execution which may cause incorrect result.

To prevent above cases, enable btf_type_tag("rcu") attributes, introduce new bpf_rcu_read_lock/unlock() kfuncs and add verifier support.

In the rest of patch set, Patch 1 enabled btf_type_tag for __rcu attribute. Patche 2 added might_sleep in bpf_func_proto. Patch 3 added new bpf_rcu_read_lock/unlock() kfuncs and verifier support.
Patch 4 added some tests for these two new kfuncs.

Changelogs:
v9 -> v10:
. if no rcu tag support in vmlinux btf, using bpf_rcu_read_lock/unlock()
will cause verification error.
. at bpf_rcu_read_unlock(), invalidate rcu ptr to PTR_UNTRUSTED
instead of SCALAR_VALUE.
. a few other comment changes and other minor changes.
v8 -> v9:
. remove sleepable prog check for ld_abs/ind checking in rcu read
lock region.
. fix a test failure with gcc-compiled kernel.
. a couple of other minor fixes.
v7 -> v8:
. add might_sleep in bpf_func_proto so we can easily identify whether
a helper is sleepable or not.
. do not enforce rcu rules for sleepable, e.g., rcu dereference must
be in a bpf_rcu_read_lock region. This is to keep old code working fine.
. Mark 'b' in 'b = a->b' (b is tagged with __rcu) as MEM_RCU only if
'b = a->b' in rcu read region and 'a' is trusted. This adds safety guarantee for 'b' inside the rcu read region.
v6 -> v7:
. rebase on top of bpf-next.
. remove the patch which enables sleepable program using
cgrp_local_storage map. This is orthogonal to this patch set and will be addressed separately.
. mark the rcu pointer dereference result as UNTRUSTED if inside
a bpf_rcu_read_lock() region.
v5 -> v6:
. fix selftest prog miss_unlock which tested nested locking. . add comments in selftest prog cgrp_succ to explain how to handle
nested memory access after rcu memory load.
v4 -> v5:
. add new test to aarch64 deny list.
v3 -> v4:
. fix selftest failures when built with gcc. gcc doesn't support
btf_type_tag yet and some tests relies on that. skip these tests if vmlinux BTF does not have btf_type_tag("rcu").
v2 -> v3:
. went back to MEM_RCU approach with invalidate rcu ptr registers
at bpf_rcu_read_unlock() place.
. remove KF_RCU_LOCK/UNLOCK flag and compare btf_id at verification
time instead.
v1 -> v2:
. use kfunc instead of helper for bpf_rcu_read_lock/unlock. . not use MEM_RCU bpf_type_flag, instead use active_rcu_lock
in reg state to identify rcu ptr's.
. Add more self tests.
. add new test to s390x deny list.

Yonghong Song (4):
compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))
bpf: Introduce might_sleep field in bpf_func_proto
bpf: Add kfunc bpf_rcu_read_lock/unlock()
selftests/bpf: Add tests for bpf_rcu_read_lock()

 include/linux/bpf.h                           |   4 +
 include/linux/bpf_verifier.h                  |   5 +-
 include/linux/compiler_types.h                |   3 +-
 kernel/bpf/bpf_lsm.c                          |   6 +-
 kernel/bpf/btf.c                              |   3 +
 kernel/bpf/helpers.c                          |  14 +
 kernel/bpf/verifier.c                         | 167 ++++++++--
 kernel/trace/bpf_trace.c                      |   4 +-
 tools/testing/selftests/bpf/DENYLIST.aarch64  |   1 +
 tools/testing/selftests/bpf/DENYLIST.s390x    |   1 +
 .../selftests/bpf/prog_tests/rcu_read_lock.c  | 158 ++++++++++
 .../selftests/bpf/progs/rcu_read_lock.c       | 290 ++++++++++++++++++
 12 files changed, 621 insertions(+), 35 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c
 create mode 100644 tools/testing/selftests/bpf/progs/rcu_read_lock.c
Signature · 1 line
2.30.2