bpf: Support 64-bit pointers to kfuncs
cover bpf: Support 64-bit pointers to kfuncs
Hi,
This is the first attempt to solve the problems outlined in [1, 2, 3]. The main problem is that kfuncs in modules do not fit into bpf_insn.imm on s390x; the secondary problem is that there is a conflict between "abstract" XDP metadata function BTF ids and their "concrete" addresses.
The proposed solution is to keep fkunc BTF ids in bpf_insn.imm, and put the addresses into bpf_kfunc_desc, which does not have size restrictions.
Jiri and I discussed putting them into bpf_insn_aux_data, but at the very beginning of the implementation it became clear that this struct is freed at the end of verification and is not available during JITing.
I regtested this only on s390x, where it does not regress anything. If this approach is fine (especially w.r.t. dev-bound kfuncs), I can check the other arches.
[1] https://lore.kernel.org/bpf/Y9%2FyrKZkBK6yzXp+@krava/
[2] https://lore.kernel.org/bpf/[email protected]/
[3] https://lore.kernel.org/bpf/[email protected]/
Best regards,
Ilya
Ilya Leoshkevich (1):
bpf: Support 64-bit pointers to kfuncs
include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 23 ++++++++++--- kernel/bpf/verifier.c | 79 +++++++++++++------------------------------ 3 files changed, 45 insertions(+), 59 deletions(-)
Signature · 1 line
2.39.1
RFC PATCH 1/1 bpf: Support 64-bit pointers to kfuncs
test_ksyms_module fails to emit a kfunc call targeting a module on s390x, because the verifier stores the difference between kfunc address and __bpf_call_base in bpf_insn.imm, which is s32, and modules are roughly (1 << 42) bytes away from the kernel on s390x.
Fix by keeping BTF id in bpf_insn.imm for BPF_PSEUDO_KFUNC_CALLs, and storing the absolute address in bpf_kfunc_desc, which JITs retrieve as usual by calling bpf_jit_get_func_addr().
This also fixes the problem with XDP metadata functions outlined in the description of commit 63d7b53ab59f ("s390/bpf: Implement bpf_jit_supports_kfunc_call()") by replacing address lookups with BTF id lookups. This eliminates the inconsistency between "abstract" XDP metadata functions' BTF ids and their concrete addresses.
--- include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 23 ++++++++++--- kernel/bpf/verifier.c | 79 +++++++++++++------------------------------ 3 files changed, 45 insertions(+), 59 deletions(-)
include/linux/bpf.h +2 -0
@@ -2227,6 +2227,8 @@ bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog);const struct btf_func_model *bpf_jit_find_kfunc_model (const struct bpf_prog *prog,const struct bpf_insn *insn); +int bpf_get_kfunc_addr (const struct bpf_prog *prog,u32 func_id,u16 offset, +u8 **func_addr);struct bpf_core_ctx {struct bpf_verifier_log *log;const struct btf *btf;
kernel/bpf/core.c +19 -4
@@ -1185,10 +1185,12 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, {s16 off = insn->off;s32 imm = insn->imm; +bool fixed;u8 *addr; +int err; - *func_addr_fixed = insn->src_reg !=BPF_PSEUDO_CALL ; -if (!*func_addr_fixed) { +switch (insn->src_reg) { +case BPF_PSEUDO_CALL :/* Place-holder address till the last pass has collected * all addresses for JITed subprograms in which case we * can pick them up from prog->aux.
@@ -1200,16 +1202,29 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, addr = (u8 *)prog->aux->func[off]->bpf_func; else return -EINVAL; - }else { + fixed = false; +break ; +case 0 :/* Address of a BPF helper call. Since part of the core * kernel, it's always at a fixed location. __bpf_call_base * and the helper with imm relative to it are both in core * kernel. */ addr = (u8 *)__bpf_call_base + imm; + fixed = true; +break ; +case BPF_PSEUDO_KFUNC_CALL : + err =bpf_get_kfunc_addr (prog, imm, off, &addr); +if (err) +return err; + fixed = true; +break ; +default : +return -EINVAL ; } - *func_addr = (unsigned long )addr; + *func_addr_fixed = fixed; + *func_addr = addr;return 0 ; }
kernel/bpf/verifier.c +24 -55
156 lines not on this page · raw message
Signature · 1 line
2.39.1
kernel test robot
build botreproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…
Hi Ilya,
[FYI, it's a private test report for your RFC patch.] [auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Ilya-Leoshkevich/bpf-Support-64-bit-pointers-to-kfuncs/20230215-054702 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20230214212809.242632-2-iii%40linux.ibm.com patch subject: [PATCH RFC bpf-next 1/1] bpf: Support 64-bit pointers to kfuncs config: loongarch-allyesconfig (https://download.01.org/0day-ci/archive/20230215/[email protected]/config) compiler: loongarch64-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/5b93c73d640845b2e5ddd1b4af608f4896379002
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Ilya-Leoshkevich/bpf-Support-64-bit-pointers-to-kfuncs/20230215-054702 git checkout 5b93c73d640845b2e5ddd1b4af608f4896379002
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch SHELL=/bin/bash kernel/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
kernel/bpf/core.c: In function 'bpf_jit_get_func_addr':
>> kernel/bpf/core.c:1227:20: warning: assignment to 'u64' {aka 'long long unsigned int'} from 'u8 *' {aka 'unsigned char *'} makes integer from pointer without a cast [-Wint-conversion]
1227 | *func_addr = addr;
| ^vim +1227 kernel/bpf/core.c
1181
1182 int bpf_jit_get_func_addr(const struct bpf_prog *prog,
1183 const struct bpf_insn *insn, bool extra_pass,
1184 u64 *func_addr, bool *func_addr_fixed)
1185 {
1186 s16 off = insn->off;
1187 s32 imm = insn->imm;
1188 bool fixed;
1189 u8 *addr;
1190 int err;
1191
1192 switch (insn->src_reg) {
1193 case BPF_PSEUDO_CALL:
1194 /* Place-holder address till the last pass has collected
1195 * all addresses for JITed subprograms in which case we
1196 * can pick them up from prog->aux.
1197 */
1198 if (!extra_pass)
1199 addr = NULL;
1200 else if (prog->aux->func &&
1201 off >= 0 && off < prog->aux->func_cnt)
1202 addr = (u8 *)prog->aux->func[off]->bpf_func;
1203 else
1204 return -EINVAL;
1205 fixed = false;
1206 break;
1207 case 0:
1208 /* Address of a BPF helper call. Since part of the core
1209 * kernel, it's always at a fixed location. __bpf_call_base
1210 * and the helper with imm relative to it are both in core
1211 * kernel.
1212 */
1213 addr = (u8 *)__bpf_call_base + imm;
1214 fixed = true;
1215 break;
1216 case BPF_PSEUDO_KFUNC_CALL:
1217 err = bpf_get_kfunc_addr(prog, imm, off, &addr);
1218 if (err)
1219 return err;
1220 fixed = true;
1221 break;
1222 default:
1223 return -EINVAL;
1224 }
1225
1226 *func_addr_fixed = fixed;
> 1227 *func_addr = addr;
1228 return 0;
1229 }
1230
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests
Quoted 65 lines · Ilya Leoshkevich
test_ksyms_modulefails to emit a kfunc call targeting a module on s390x, because the verifier stores the difference between kfunc address and__bpf_call_basein bpf_insn.imm, which is s32, and modules are roughly (1 << 42) bytes away from the kernel on s390x.Fix by keeping BTF id in bpf_insn.imm for
BPF_PSEUDO_KFUNC_CALLs, and storing the absolute address inbpf_kfunc_desc, which JITs retrieve as usual by callingbpf_jit_get_func_addr().This also fixes the problem with XDP metadata functions outlined in the description of commit
63d7b53ab59f("s390/bpf: Implementbpf_jit_supports_kfunc_call()") by replacing address lookups with BTF id lookups. This eliminates the inconsistency between "abstract" XDP metadata functions' BTF ids and their concrete addresses.Signed-off-by Ilya Leoshkevich <[email protected]>--- include/linux/bpf.h | 2 ++ kernel/bpf/core.c | 23 ++++++++++--- kernel/bpf/verifier.c | 79 +++++++++++++------------------------------ 3 files changed, 45 insertions(+), 59 deletions(-)include/linux/bpf.h +2
-0@@ -2227,6 +2227,8 @@ bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog);const struct btf_func_model *bpf_jit_find_kfunc_model (const struct bpf_prog *prog,const struct bpf_insn *insn); +int bpf_get_kfunc_addr (const struct bpf_prog *prog,u32 func_id,u16 offset, +u8 **func_addr);struct bpf_core_ctx {struct
bpf_verifier_log*log;
const struct btf *btf;kernel/bpf/core.c +4
-2@@ -1185,10 +1185,12 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, {s16 off = insn->off;s32 imm = insn->imm; +bool fixed;u8 *addr; +int err; - *func_addr_fixed = insn->src_reg !=BPF_PSEUDO_CALL ; -if (!*func_addr_fixed) { +switch (insn->src_reg) { +case BPF_PSEUDO_CALL :/* Place-holder address till the last pass has collected * all addresses for JITed subprograms in which case we * can pick them up from
prog->aux.kernel/bpf/core.c +2
-1@@ -1200,16 +1202,29 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, addr = (u8 *)prog->aux->func[off]->bpf_func; else return -EINVAL; - }else { + fixed = false; +break ;
[..]
kernel/bpf/core.c int bpf_jit_get_func_addr(const struct bpf_prog *prog, · line 1207
Quoted 1 line
kernel/bpf/core.c +1
-0+case 0 :
nit: Should we define BPF_HELPER_CALL here for consistency?
Quoted 43 lines
/* Address of a BPF helper call. Since part of the core
* kernel, it's always at a fixed location.__bpf_call_base
* and the helper with imm relative to it are both in core
* kernel.
*/
addr = (u8 *)__bpf_call_base + imm;kernel/bpf/core.c +12
-1+ fixed = true; +break ; +case BPF_PSEUDO_KFUNC_CALL : + err =bpf_get_kfunc_addr (prog, imm, off, &addr); +if (err) +return err; + fixed = true; +break ; +default : +return -EINVAL ; } - *func_addr = (unsigned long )addr; + *func_addr_fixed = fixed; + *func_addr = addr;return 0 ; }kernel/bpf/verifier.c +1
-1@@ -2115,8 +2115,8 @@ static int add_subprog(struct bpf_verifier_env *env, int off)struct bpf_kfunc_desc {struct btf_func_model func_model;u32 func_id; -s32 imm;u16 offset; +unsigned long addr; };struct bpf_kfunc_btf {kernel/bpf/verifier.c +0
-0@@ -2166,6 +2166,19 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off); }
[..]
kernel/bpf/verifier.c find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) · line 2180
Quoted 13 lines
+int bpf_get_kfunc_addr(const struct
bpf_prog*prog, u32func_id, u16 offset,kernel/bpf/verifier.c +11
-0+ u8 **func_addr) +{ +const struct bpf_kfunc_desc *desc; + + desc =find_kfunc_desc (prog, func_id, offset); +if (WARN_ON_ONCE (!desc)) +return -EINVAL ; + + *func_addr = (u8 *)desc->addr; +return 0 ; +}
This function isn't doing much and has a single caller. Should we just export find_kfunc_desc?
Quoted 144 lines
kernel/bpf/verifier.c +2
-1+static struct btf *__find_kfunc_desc_btf (struct bpf_verifier_env *env,s16 offset) {@@ -2261,8 +2274,8 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)struct bpf_kfunc_desc *desc;const char *func_name;struct btf *desc_btf; -unsigned long call_imm;unsigned long addr; +void *xdp_kfunc;int err;prog_aux = env->prog->aux;kernel/bpf/verifier.c +6
-9@@ -2346,24 +2359,21 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)return -EINVAL ; } - call_imm =BPF_CALL_IMM (addr); -/* Check whether or not the relative offset overflows desc->imm */ -if ((unsigned long )(s32 )call_imm != call_imm) { -verbose (env,"address of kernel function %s is out of range\n" , - func_name); -return -EINVAL ; - } -if (bpf_dev_bound_kfunc_id (func_id)) { err =bpf_dev_bound_kfunc_check (&env->log, prog_aux);if (err)return err; + + xdp_kfunc =bpf_dev_bound_resolve_kfunc (env->prog, func_id); +if (xdp_kfunc) + addr = (unsigned long )xdp_kfunc; +/* fallback to default kfunc when not supported by netdev */ } desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; - desc->imm = call_imm; desc->offset = offset; + desc->addr = addr; err = btf_distill_func_proto(&env->log, desc_btf, func_proto, func_name,
&desc->func_model);kernel/bpf/verifier.c +0
-24@@ -2373,30 +2383,6 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)return err; } -static int kfunc_desc_cmp_by_imm (const void *a,const void *b) -{ -const struct bpf_kfunc_desc *d0 = a; -const struct bpf_kfunc_desc *d1 = b; - -if (d0->imm > d1->imm) -return 1 ; -else if (d0->imm < d1->imm) -return -1 ; -return 0 ; -} - -static void sort_kfunc_descs_by_imm (struct bpf_prog *prog) -{ -struct bpf_kfunc_desc_tab *tab; - - tab = prog->aux->kfunc_tab; -if (!tab) -return ; - -sort (tab->descs, tab->nr_descs,sizeof (tab->descs[0 ]), - kfunc_desc_cmp_by_imm,NULL ); -} -bool bpf_prog_has_kfunc_call (const struct bpf_prog *prog) {return !!prog->aux->kfunc_tab;
kernel/bpf/verifier.c +3
-2@@ -2407,14 +2393,15 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn) {const struct bpf_kfunc_desc desc = { - .imm = insn->imm, + .func_id = insn->imm, + .offset = insn->off, };const struct bpf_kfunc_desc *res;struct bpf_kfunc_desc_tab *tab; tab = prog->aux->kfunc_tab; res =bsearch (&desc, tab->descs, tab->nr_descs, -sizeof (tab->descs[0 ]), kfunc_desc_cmp_by_imm); +sizeof (tab->descs[0 ]), kfunc_desc_cmp_by_id_off);return res ? &res->func_model :NULL ;}kernel/bpf/verifier.c +0
-1@@ -16251,7 +16238,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; -void *xdp_kfunc;if (!insn->imm) {verbose(env, "invalid kernel function call not eliminated in verifier
pass\n");kernel/bpf/verifier.c +0
-14@@ -16259,20 +16245,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } *cnt =0 ; - -if (bpf_dev_bound_kfunc_id (insn->imm)) { - xdp_kfunc =bpf_dev_bound_resolve_kfunc (env->prog, insn->imm); -if (xdp_kfunc) { - insn->imm =BPF_CALL_IMM (xdp_kfunc); -return 0 ; - } - -/* fallback to default kfunc when not supported by netdev */ - } - -/* insn->imm has the btf func_id. Replace it with -* an address (relative to __bpf_call_base). -*/ desc =find_kfunc_desc (env->prog, insn->imm, insn->off);if (!desc) {verbose(env, "verifier internal error: kernel function descriptor not
found forfunc_id%u\n",kernel/bpf/verifier.c +0
-1@@ -16280,7 +16252,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -EFAULT; } - insn->imm = desc->imm;if (insn->off)return 0 ;if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl]) {kernel/bpf/verifier.c +0
-0@@ -16834,8 +16805,6 @@ static int do_misc_fixups(struct bpf_verifier_env *env) } }
[..]
kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 16837
Quoted 1 line
kernel/bpf/verifier.c +0
-1-sort_kfunc_descs_by_imm (env->prog);
If we are not doing sorting here, how does the bsearch work?
Quoted 6 lines
-
return 0;
}--
2.39.1
Quoted 79 lines · Stanislav Fomichev
Quoted 71 lines · Ilya Leoshkevich
test_ksyms_modulefails to emit a kfunc call targeting a module on s390x, because the verifier stores the difference between kfunc address and__bpf_call_basein bpf_insn.imm, which is s32, and modules
are roughly (1 << 42) bytes away from the kernel on s390x.Fix by keeping BTF id in bpf_insn.imm for
BPF_PSEUDO_KFUNC_CALLs, and storing the absolute address inbpf_kfunc_desc, which JITs retrieve
as usual by callingbpf_jit_get_func_addr().This also fixes the problem with XDP metadata functions outlined in the description of commit
63d7b53ab59f("s390/bpf: Implementbpf_jit_supports_kfunc_call()") by replacing address lookups with BTF
id lookups. This eliminates the inconsistency between "abstract" XDP
metadata functions' BTF ids and their concrete addresses.Signed-off-by Ilya Leoshkevich <[email protected]>---
include/linux/bpf.h | 2 ++
kernel/bpf/core.c | 23 ++++++++++---kernel/bpf/verifier.c| 79 +++++++++++++-------------------------
-----3 files changed, 45 insertions(+), 59 deletions(-)include/linux/bpf.h +0
-0@@ -2227,6 +2227,8 @@ bool bpf_prog_has_kfunc_call(const struct
bpf_prog
*prog);
const structbtf_func_model*bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn); +int bpf_get_kfunc_addr(const structbpf_prog*prog, u32func_id, u16
offset,
+ u8 **func_addr);
structbpf_core_ctx{
structbpf_verifier_log*log;
const struct btf *btf;kernel/bpf/core.c +0
-0@@ -1185,10 +1185,12 @@ int bpf_jit_get_func_addr(const struct
bpf_prog
*prog,
{
s16 off =insn->off;
s32 imm =insn->imm;
+ bool fixed;
u8 *addr;
+ int err;- *func_addr_fixed =
insn->src_reg!=BPF_PSEUDO_CALL; - if (!*func_addr_fixed) {
+ switch (insn->src_reg) {
+ caseBPF_PSEUDO_CALL:
/* Place-holder address till the last pass has collected
* all addresses for JITed subprograms in which case we
* can pick them up fromprog->aux.kernel/bpf/core.c +0
-0@@ -1200,16 +1202,29 @@ int bpf_jit_get_func_addr(const struct
bpf_prog
*prog,
addr = (u8 *)prog->aux->func[off]-Quoted 6 lines
Quoted 1 line
bpf_func;else
return-EINVAL;
- } else {
+ fixed = false;
+ break;
Quoted 5 lines
[..]
Quoted 1 line
+ case 0:
nit: Should we define
BPF_HELPER_CALLhere for consistency?
I think this would be good, the verifier currently uses 0 for this purpose; having a symbolic constant would surely improve readability.
Quoted 63 lines
Quoted 49 lines
/* Address of a BPF helper call. Since part of the core
* kernel, it's always at a fixed location.__bpf_call_base
* and the helper with imm relative to it are both in core
* kernel.
*/
addr = (u8 *)__bpf_call_base + imm;
+ fixed = true;
+ break;
+ caseBPF_PSEUDO_KFUNC_CALL:
+ err =bpf_get_kfunc_addr(prog, imm, off, &addr); + if (err)
+ return err;
+ fixed = true;
+ break;
+ default:
+ return-EINVAL;
}- *func_addr = (unsigned long)addr;
+ *func_addr_fixed = fixed;
+ *func_addr = addr;
return 0;
}kernel/bpf/verifier.c +0
-0@@ -2115,8 +2115,8 @@ static int add_subprog(struct
bpf_verifier_env
*env, int off)
structbpf_kfunc_desc{
structbtf_func_modelfunc_model;
u32func_id;
- s32 imm;
u16 offset;
+ unsigned long addr;
};struct
bpf_kfunc_btf{kernel/bpf/verifier.c +0
-0@@ -2166,6 +2166,19 @@ find_kfunc_desc(const struct bpf_prog *prog,u32
func_id, u16 offset)
sizeof(tab->descs[0]),kfunc_desc_cmp_by_id_off);
}[..]
Quoted 10 lines
+int bpf_get_kfunc_addr(const struct
bpf_prog*prog, u32func_id, u16
offset,
+ u8 **func_addr)kernel/bpf/verifier.c +6
-0+{ + const struct bpf_kfunc_desc *desc; + + desc = find_kfunc_desc(prog, func_id, offset); + if (WARN_ON_ONCE(!desc)) + return -EINVAL;
Quoted 8 lines
kernel/bpf/verifier.c find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) · line 2180
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+ + *func_addr = (u8 *)desc->addr; + return 0; +}This function isn't doing much and has a single caller. Should we just
exportfind_kfunc_desc?
We would have to export struct bpf_kfunc_desc as well; I thought it's better to add an extra function so that we could keep hiding the struct.
Quoted 167 lines
Quoted 163 lines
+
static struct btf *__find_kfunc_desc_btf(structbpf_verifier_env*env,
s16 offset)
{kernel/bpf/verifier.c +0
-0@@ -2261,8 +2274,8 @@ static int add_kfunc_call(struct
bpf_verifier_env
*env, u32func_id, s16 offset)
structbpf_kfunc_desc*desc;
const char*func_name;
struct btf*desc_btf;
- unsigned longcall_imm;
unsigned long addr;
+ void*xdp_kfunc;
int err;prog_aux =
env->prog->aux;kernel/bpf/verifier.c +0
-0@@ -2346,24 +2359,21 @@ static int add_kfunc_call(struct
bpf_verifier_env
*env, u32func_id, s16 offset)
return-EINVAL;
}- call_imm =
BPF_CALL_IMM(addr);
- /* Check whether or not the relative offset overflows desc-Quoted 1 line
imm */
- if ((unsigned long)(s32)call_imm !=
call_imm) { - verbose(env, "address of kernel function %s is out of range\n",
- func_name);
- return-EINVAL;
- }
-
if (bpf_dev_bound_kfunc_id(func_id)) {
err =bpf_dev_bound_kfunc_check(&env->log, prog_aux);
if (err)
return err;
+
+ xdp_kfunc =bpf_dev_bound_resolve_kfunc(env->prog, func_id);
+ if (xdp_kfunc)
+ addr = (unsigned long)xdp_kfunc; + /* fallback to default kfunc when not supported by netdev */
}desc = &tab->descs[tab->nr_descs++];
desc->func_id=func_id;
- desc->imm =call_imm;
desc->offset = offset;
+ desc->addr = addr;
err =btf_distill_func_proto(&env->log, desc_btf, func_proto, func_name, &desc->func_model);kernel/bpf/verifier.c +0
-0@@ -2373,30 +2383,6 @@ static int add_kfunc_call(struct
bpf_verifier_env
*env, u32func_id, s16 offset)
return err;
}kernel/bpf/verifier.c +0
-24-static int kfunc_desc_cmp_by_imm (const void *a,const void *b) -{ - const struct bpf_kfunc_desc *d0 = a; - const struct bpf_kfunc_desc *d1 = b; - - if (d0->imm > d1->imm) - return 1; - else if (d0->imm < d1->imm) - return -1; - return 0; -} - -static void sort_kfunc_descs_by_imm (struct bpf_prog *prog) -{ - struct bpf_kfunc_desc_tab *tab; - - tab = prog->aux->kfunc_tab; - if (!tab) - return; - - sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), - kfunc_desc_cmp_by_imm, NULL); -} -bool
bpf_prog_has_kfunc_call(const struct bpf_prog *prog){
return !!prog->aux->kfunc_tab;kernel/bpf/verifier.c +0
-0@@ -2407,14 +2393,15 @@ bpf_jit_find_kfunc_model(const struct
bpf_prog
*prog,
const structbpf_insn*insn) {
const structbpf_kfunc_descdesc = { - .imm =insn->imm,
+ .func_id =insn->imm,
+ .offset =insn->off,
};
const structbpf_kfunc_desc*res; structbpf_kfunc_desc_tab*tab;tab =
prog->aux->kfunc_tab;
res = bsearch(&desc,tab->descs,tab->nr_descs, -sizeof(tab->descs[0]),kfunc_desc_cmp_by_imm);
+sizeof(tab->descs[0]),kfunc_desc_cmp_by_id_off);return res ?
&res->func_model: NULL;
}kernel/bpf/verifier.c +0
-0@@ -16251,7 +16238,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,struct
bpf_insn*insn_buf, intinsn_idx, int *cnt)
{
const structbpf_kfunc_desc*desc;
- void*xdp_kfunc;if (!insn->imm) {
verbose(env, "invalid kernel function call not eliminated in verifier
pass\n");kernel/bpf/verifier.c +0
-9@@ -16259,20 +16245,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } *cnt = 0; - - if (bpf_dev_bound_kfunc_id(insn->imm)) { - xdp_kfunc = bpf_dev_bound_resolve_kfunc(env->prog, insn->imm); - if (xdp_kfunc) { - insn->imm = BPF_CALL_IMM(xdp_kfunc); - return 0; - } - - /* fallback to default kfunc when not supported bynetdev */
- }
-
- /*insn->immhas the btffunc_id. Replace it with - * an address (relative to__bpf_call_base).
- */
desc =find_kfunc_desc(env->prog, insn->imm, insn->off); if (!desc) {
verbose(env, "verifier internal error: kernel function descriptor not
found forfunc_id%u\n",kernel/bpf/verifier.c +0
-1@@ -16280,7 +16252,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -EFAULT; } - insn->imm = desc->imm; if (insn->off) return 0;if (
desc->func_id==
special_kfunc_list[KF_bpf_obj_new_impl]) {kernel/bpf/verifier.c +0
-0@@ -16834,8 +16805,6 @@ static int do_misc_fixups(struct
bpf_verifier_env
*env)
}
}[..]
Quoted 3 lines
Quoted 1 line
- sort_kfunc_descs_by_imm(env->prog);
If we are not doing sorting here, how does the bsearch work?
add_kfunc_call() already makes sure that kfuncs are sorted in the kfunc_desc_cmp_by_id_off() order. fixup_kfunc_call() used to put addresses into imms and re-sort based on that, however, after this patch it's not needed anymore: the code (e.g. bpf_jit_find_kfunc_model()) continues to do lookups in the kfunc_desc_cmp_by_id_off() order.
Quoted 7 lines
Quoted 6 lines
-
return 0;
}--
2.39.1
Quoted 161 lines · Ilya Leoshkevich
Quoted 79 lines · Stanislav Fomichev
Quoted 71 lines · Ilya Leoshkevich
test_ksyms_modulefails to emit a kfunc call targeting a module on s390x, because the verifier stores the difference between kfunc address and__bpf_call_basein bpf_insn.imm, which is s32, and modules
are roughly (1 << 42) bytes away from the kernel on s390x.Fix by keeping BTF id in bpf_insn.imm for
BPF_PSEUDO_KFUNC_CALLs, and storing the absolute address inbpf_kfunc_desc, which JITs retrieve
as usual by callingbpf_jit_get_func_addr().This also fixes the problem with XDP metadata functions outlined in the description of commit
63d7b53ab59f("s390/bpf: Implementbpf_jit_supports_kfunc_call()") by replacing address lookups with BTF
id lookups. This eliminates the inconsistency between "abstract" XDP
metadata functions' BTF ids and their concrete addresses.Signed-off-by Ilya Leoshkevich <[email protected]>---
include/linux/bpf.h | 2 ++
kernel/bpf/core.c | 23 ++++++++++---kernel/bpf/verifier.c| 79 +++++++++++++-------------------------
-----3 files changed, 45 insertions(+), 59 deletions(-)include/linux/bpf.h +0
-0@@ -2227,6 +2227,8 @@ bool bpf_prog_has_kfunc_call(const struct
bpf_prog
*prog);
const structbtf_func_model*bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn); +int bpf_get_kfunc_addr(const structbpf_prog*prog, u32func_id, u16
offset,
+ u8 **func_addr);
structbpf_core_ctx{
structbpf_verifier_log*log;
const struct btf *btf;kernel/bpf/core.c +0
-0@@ -1185,10 +1185,12 @@ int bpf_jit_get_func_addr(const struct
bpf_prog
*prog,
{
s16 off =insn->off;
s32 imm =insn->imm;
+ bool fixed;
u8 *addr;
+ int err;- *func_addr_fixed =
insn->src_reg!=BPF_PSEUDO_CALL; - if (!*func_addr_fixed) {
+ switch (insn->src_reg) {
+ caseBPF_PSEUDO_CALL:
/* Place-holder address till the last pass has collected
* all addresses for JITed subprograms in which case we
* can pick them up fromprog->aux.kernel/bpf/core.c +0
-0@@ -1200,16 +1202,29 @@ int bpf_jit_get_func_addr(const struct
bpf_prog
*prog,
addr = (u8 *)prog->aux->func[off]-Quoted 6 lines
Quoted 1 line
bpf_func;else
return-EINVAL;
- } else {
+ fixed = false;
+ break;Quoted 5 lines
[..]
Quoted 1 line
+ case 0:
nit: Should we define
BPF_HELPER_CALLhere for consistency?I think this would be good, the verifier currently uses 0 for this purpose; having a symbolic constant would surely improve readability.
Quoted 63 lines
Quoted 49 lines
/* Address of a BPF helper call. Since part of the core
* kernel, it's always at a fixed location.__bpf_call_base
* and the helper with imm relative to it are both in core
* kernel.
*/
addr = (u8 *)__bpf_call_base + imm;
+ fixed = true;
+ break;
+ caseBPF_PSEUDO_KFUNC_CALL:
+ err =bpf_get_kfunc_addr(prog, imm, off, &addr); + if (err)
+ return err;
+ fixed = true;
+ break;
+ default:
+ return-EINVAL;
}- *func_addr = (unsigned long)addr;
+ *func_addr_fixed = fixed;
+ *func_addr = addr;
return 0;
}kernel/bpf/verifier.c +0
-0@@ -2115,8 +2115,8 @@ static int add_subprog(struct
bpf_verifier_env
*env, int off)
structbpf_kfunc_desc{
structbtf_func_modelfunc_model;
u32func_id;
- s32 imm;
u16 offset;
+ unsigned long addr;
};struct
bpf_kfunc_btf{kernel/bpf/verifier.c +0
-0@@ -2166,6 +2166,19 @@ find_kfunc_desc(const struct bpf_prog *prog,u32
func_id, u16 offset)
sizeof(tab->descs[0]),kfunc_desc_cmp_by_id_off);
}[..]
Quoted 10 lines
+int bpf_get_kfunc_addr(const struct
bpf_prog*prog, u32func_id, u16
offset,
+ u8 **func_addr)kernel/bpf/verifier.c +6
-0+{ + const struct bpf_kfunc_desc *desc; + + desc = find_kfunc_desc(prog, func_id, offset); + if (WARN_ON_ONCE(!desc)) + return -EINVAL;Quoted 8 lines
kernel/bpf/verifier.c find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) · line 2180
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+ + *func_addr = (u8 *)desc->addr; + return 0; +}This function isn't doing much and has a single caller. Should we just
exportfind_kfunc_desc?
Quoted 3 lines
We would have to export struct
bpf_kfunc_descas well; I thought it's better to add an extra function so that we could keep hiding the struct.
Ah, good point. In this case seems ok to have this extra wrapper. On that note: what's the purpose of WARN_ON_ONCE here?
Quoted 169 lines
Quoted 169 lines
Quoted 163 lines
+
static struct btf *__find_kfunc_desc_btf(structbpf_verifier_env*env,
s16 offset)
{kernel/bpf/verifier.c +0
-0@@ -2261,8 +2274,8 @@ static int add_kfunc_call(struct
bpf_verifier_env
*env, u32func_id, s16 offset)
structbpf_kfunc_desc*desc;
const char*func_name;
struct btf*desc_btf;
- unsigned longcall_imm;
unsigned long addr;
+ void*xdp_kfunc;
int err;prog_aux =
env->prog->aux;kernel/bpf/verifier.c +0
-0@@ -2346,24 +2359,21 @@ static int add_kfunc_call(struct
bpf_verifier_env
*env, u32func_id, s16 offset)
return-EINVAL;
}- call_imm =
BPF_CALL_IMM(addr);
- /* Check whether or not the relative offset overflows desc-Quoted 1 line
imm */
- if ((unsigned long)(s32)call_imm !=
call_imm) { - verbose(env, "address of kernel function %s is out of range\n",
- func_name);
- return-EINVAL;
- }
-
if (bpf_dev_bound_kfunc_id(func_id)) {
err =bpf_dev_bound_kfunc_check(&env->log, prog_aux);
if (err)
return err;
+
+ xdp_kfunc =bpf_dev_bound_resolve_kfunc(env->prog, func_id);
+ if (xdp_kfunc)
+ addr = (unsigned long)xdp_kfunc; + /* fallback to default kfunc when not supported by netdev */
}desc = &tab->descs[tab->nr_descs++];
desc->func_id=func_id;
- desc->imm =call_imm;
desc->offset = offset;
+ desc->addr = addr;
err =btf_distill_func_proto(&env->log, desc_btf, func_proto, func_name, &desc->func_model);kernel/bpf/verifier.c +0
-0@@ -2373,30 +2383,6 @@ static int add_kfunc_call(struct
bpf_verifier_env
*env, u32func_id, s16 offset)
return err;
}kernel/bpf/verifier.c +0
-24-static int kfunc_desc_cmp_by_imm (const void *a,const void *b) -{ - const struct bpf_kfunc_desc *d0 = a; - const struct bpf_kfunc_desc *d1 = b; - - if (d0->imm > d1->imm) - return 1; - else if (d0->imm < d1->imm) - return -1; - return 0; -} - -static void sort_kfunc_descs_by_imm (struct bpf_prog *prog) -{ - struct bpf_kfunc_desc_tab *tab; - - tab = prog->aux->kfunc_tab; - if (!tab) - return; - - sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), - kfunc_desc_cmp_by_imm, NULL); -} -bool
bpf_prog_has_kfunc_call(const struct bpf_prog *prog){
return !!prog->aux->kfunc_tab;kernel/bpf/verifier.c +0
-0@@ -2407,14 +2393,15 @@ bpf_jit_find_kfunc_model(const struct
bpf_prog
*prog,
const structbpf_insn*insn) {
const structbpf_kfunc_descdesc = { - .imm =insn->imm,
+ .func_id =insn->imm,
+ .offset =insn->off,
};
const structbpf_kfunc_desc*res; structbpf_kfunc_desc_tab*tab;tab =
prog->aux->kfunc_tab;
res = bsearch(&desc,tab->descs,tab->nr_descs, -sizeof(tab->descs[0]),kfunc_desc_cmp_by_imm);
+sizeof(tab->descs[0]),kfunc_desc_cmp_by_id_off);return res ?
&res->func_model: NULL;
}kernel/bpf/verifier.c +0
-0@@ -16251,7 +16238,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,struct
bpf_insn*insn_buf, intinsn_idx, int *cnt)
{
const structbpf_kfunc_desc*desc;
- void*xdp_kfunc;if (!insn->imm) {
verbose(env, "invalid kernel function call not eliminated in verifier
pass\n");kernel/bpf/verifier.c +0
-9@@ -16259,20 +16245,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } *cnt = 0; - - if (bpf_dev_bound_kfunc_id(insn->imm)) { - xdp_kfunc = bpf_dev_bound_resolve_kfunc(env->prog, insn->imm); - if (xdp_kfunc) { - insn->imm = BPF_CALL_IMM(xdp_kfunc); - return 0; - } - - /* fallback to default kfunc when not supported bynetdev */
- }
-
- /*insn->immhas the btffunc_id. Replace it with - * an address (relative to__bpf_call_base).
- */
desc =find_kfunc_desc(env->prog, insn->imm, insn->off); if (!desc) {
verbose(env, "verifier internal error: kernel function descriptor not
found forfunc_id%u\n",kernel/bpf/verifier.c +0
-1@@ -16280,7 +16252,6 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -EFAULT; } - insn->imm = desc->imm; if (insn->off) return 0;if (
desc->func_id==
special_kfunc_list[KF_bpf_obj_new_impl]) {kernel/bpf/verifier.c +0
-0@@ -16834,8 +16805,6 @@ static int do_misc_fixups(struct
bpf_verifier_env
*env)
}
}[..]
Quoted 1 line
- sort_kfunc_descs_by_imm(env->prog);
Quoted 8 lines
Quoted 1 line
If we are not doing sorting here, how does the bsearch work?
add_kfunc_call()already makes sure that kfuncs are sorted in thekfunc_desc_cmp_by_id_off()order.fixup_kfunc_call()used to put addresses into imms and re-sort based on that, however, after this patch it's not needed anymore: the code (e.g.bpf_jit_find_kfunc_model()) continues to do lookups in thekfunc_desc_cmp_by_id_off()order.
Makes sense, thank you 👍
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
-
return 0;
}--
2.39.1
Quoted 36 lines · Stanislav Fomichev
Quoted 35 lines · Ilya Leoshkevich
Quoted 33 lines · Stanislav Fomichev
Quoted 32 lines · Ilya Leoshkevich
test_ksyms_modulefails to emit a kfunc call targeting a module on
s390x, because the verifier stores the difference between kfunc address and__bpf_call_basein bpf_insn.imm, which is s32, and modules
are roughly (1 << 42) bytes away from the kernel on s390x.Fix by keeping BTF id in bpf_insn.imm for
BPF_PSEUDO_KFUNC_CALLs,
and storing the absolute address inbpf_kfunc_desc, which JITs retrieve
as usual by callingbpf_jit_get_func_addr().This also fixes the problem with XDP metadata functions outlined in
the description of commit63d7b53ab59f("s390/bpf: Implementbpf_jit_supports_kfunc_call()") by replacing address lookups with
BTF
id lookups. This eliminates the inconsistency between "abstract"
XDP
metadata functions' BTF ids and their concrete addresses.Signed-off-by Ilya Leoshkevich <[email protected]>---
include/linux/bpf.h | 2 ++
kernel/bpf/core.c | 23 ++++++++++---kernel/bpf/verifier.c| 79 +++++++++++++---------------------
----
-----3 files changed, 45 insertions(+), 59 deletions(-)
[...]
Quoted 20 lines
Quoted 19 lines
Quoted 19 lines
Quoted 5 lines
+int bpf_get_kfunc_addr(const struct
bpf_prog*prog, u32func_id,
u16
offset,
+ u8 **func_addr)kernel/bpf/verifier.c find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) · line 2180
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+{ + const struct bpf_kfunc_desc *desc; + + desc = find_kfunc_desc(prog, func_id, offset); + if (WARN_ON_ONCE(!desc)) + return -EINVAL; + + *func_addr = (u8 *)desc->addr; + return 0; +}This function isn't doing much and has a single caller. Should we just
exportfind_kfunc_desc?
Quoted 7 lines
Quoted 4 lines
We would have to export struct
bpf_kfunc_descas well; I thought it's
better to add an extra function so that we could keep hiding the struct.Ah, good point. In this case seems ok to have this extra wrapper. On that note: what's the purpose of
WARN_ON_ONCEhere?
We can hit this only due to an internal verifier/JIT error, so it would be good to get some indication of this happening. In verifier.c we have verbose() function for that, but this function is called during JITing.
[...]
Quoted 65 lines · Ilya Leoshkevich
Quoted 36 lines · Stanislav Fomichev
Quoted 35 lines · Ilya Leoshkevich
Quoted 33 lines · Stanislav Fomichev
Quoted 32 lines · Ilya Leoshkevich
test_ksyms_modulefails to emit a kfunc call targeting a module on
s390x, because the verifier stores the difference between kfunc address and__bpf_call_basein bpf_insn.imm, which is s32, and modules
are roughly (1 << 42) bytes away from the kernel on s390x.Fix by keeping BTF id in bpf_insn.imm for
BPF_PSEUDO_KFUNC_CALLs,
and storing the absolute address inbpf_kfunc_desc, which JITs retrieve
as usual by callingbpf_jit_get_func_addr().This also fixes the problem with XDP metadata functions outlined in
the description of commit63d7b53ab59f("s390/bpf: Implementbpf_jit_supports_kfunc_call()") by replacing address lookups with
BTF
id lookups. This eliminates the inconsistency between "abstract"
XDP
metadata functions' BTF ids and their concrete addresses.Signed-off-by Ilya Leoshkevich <[email protected]>---
� include/linux/bpf.h�� |� 2 ++
� kernel/bpf/core.c���� | 23 ++++++++++---
�kernel/bpf/verifier.c| 79 +++++++++++++---------------------
----
-----
� 3 files changed, 45 insertions(+), 59 deletions(-)[...]
Quoted 25 lines
Quoted 24 lines
Quoted 11 lines
Quoted 11 lines
+int bpf_get_kfunc_addr(const struct
bpf_prog*prog, u32func_id,
u16�
offset,
+��������������������� u8 **func_addr)(quoted diff) +6
-0+{ +�������const struct bpf_kfunc_desc *desc; + +�������desc = find_kfunc_desc(prog, func_id, offset); +�������if (WARN_ON_ONCE(!desc)) +���������������return -EINVAL;Quoted 8 lines
kernel/bpf/verifier.c find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) · line 2180
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+ +�������*func_addr = (u8 *)desc->addr; +�������return 0; +}This function isn't doing much and has a single caller. Should we just
exportfind_kfunc_desc?We would have to export struct
bpf_kfunc_descas well; I thought it's
better to add an extra function so that we could keep hiding the struct.
Quoted 8 lines
Quoted 2 lines
Ah, good point. In this case seems ok to have this extra wrapper. On that note: what's the purpose of
WARN_ON_ONCEhere?We can hit this only due to an internal verifier/JIT error, so it would be good to get some indication of this happening. In
verifier.cwe haveverbose()function for that, but this function is called during JITing.[...]
From my point of view, reading the code, it makes it a bit confusing. If there is a WARN_ON_ONCE, I'm assuming it's guarding against some kind of internal inconsistency that can happen.
What kind of inconsistency is it guarding against here? We seem to have find_kfunc_desc in fixup_kfunc_call that checks the same insn->imm and returns early.
Quoted 95 lines · Stanislav Fomichev
Quoted 78 lines · Ilya Leoshkevich
Quoted 43 lines · Stanislav Fomichev
Quoted 42 lines · Ilya Leoshkevich
Quoted 40 lines · Stanislav Fomichev
Quoted 39 lines · Ilya Leoshkevich
test_ksyms_modulefails to emit a kfunc call targeting a module
on
s390x, because the verifier stores the difference between kfunc
address and__bpf_call_basein bpf_insn.imm, which is s32, and
modules
are roughly (1 << 42) bytes away from the kernel on s390x.Fix by keeping BTF id in bpf_insn.imm for
BPF_PSEUDO_KFUNC_CALLs,
and storing the absolute address inbpf_kfunc_desc, which JITs
retrieve
as usual by callingbpf_jit_get_func_addr().This also fixes the problem with XDP metadata functions outlined in
the description of commit63d7b53ab59f("s390/bpf:
Implementbpf_jit_supports_kfunc_call()") by replacing address lookups
with
BTF
id lookups. This eliminates the inconsistency between "abstract"
XDP
metadata functions' BTF ids and their concrete addresses.Signed-off-by Ilya Leoshkevich <[email protected]>---
� include/linux/bpf.h�� |� 2 ++
� kernel/bpf/core.c���� | 23 ++++++++++---
�kernel/bpf/verifier.c| 79 +++++++++++++-----------------
----
----
-----
� 3 files changed, 45 insertions(+), 59 deletions(-)[...]
Quoted 28 lines
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
+int bpf_get_kfunc_addr(const struct
bpf_prog*prog, u32func_id,
u16�
offset,
+��������������������� u8 **func_addr)(quoted diff) +7
-0+{ +�������const struct bpf_kfunc_desc *desc; + +�������desc = find_kfunc_desc(prog, func_id, offset); +�������if (WARN_ON_ONCE(!desc)) +���������������return -EINVAL; +Quoted 8 lines
Quoted 3 lines
+�������*func_addr = (u8 *)desc->addr;
+�������return 0;
+}This function isn't doing much and has a single caller. Should we
just
exportfind_kfunc_desc?Quoted 6 lines
We would have to export struct
bpf_kfunc_descas well; I thought
it's
better to add an extra function so that we could keep hiding the
struct.Quoted 2 lines
Ah, good point. In this case seems ok to have this extra wrapper. On that note: what's the purpose of
WARN_ON_ONCEhere?Quoted 8 lines
We can hit this only due to an internal verifier/JIT error, so it would
be good to get some indication of this happening. Inverifier.cwe haveverbose()function for that, but this function is called during JITing.[...]
From my point of view, reading the code, it makes it a bit confusing. If
there
is aWARN_ON_ONCE, I'm assuming it's guarding against some kind of internal
inconsistency that can happen.
Quoted 4 lines
What kind of inconsistency is it guarding against here? We seem to have
find_kfunc_descinfixup_kfunc_callthat checks the sameinsn->immand returns early.
The potential inconsistency is the situation when the check in fixup_kfunc_call() passes, and the very same check here fails.
We could say it's impossible and directly dereference desc. But if there is a bug and it still happens, the result is a panic or an oops.
Or we could still check whether it's NULL. And if we add such a check, adding a WARN_ON_ONCE() on top is a cheap way to quickly pinpoint the potential user-observable issues.
Of course, it's a defensive programming / gut feeling thing. I don't think comparing all pointers to NULL before derefencing them is a good idea. It's just that here the first NULL comparison is quite far away, and with time code that modifies kfunc_tab may be added in between, in which case this might come in handy.
In any case, I don't have a really strong opinion here, just explaining my thought process. If the consensus is to drop it, I would not mind dropping it at all.
Quoted 4 lines · Ilya Leoshkevich
In any case, I don't have a really strong opinion here, just explaining my thought process. If the consensus is to drop it, I would not mind dropping it at all.
I think doing:
+ desc = find_kfunc_desc(prog, func_id, offset);
+ if (!desc)
+ return -EFAULT;
would be enough to catch that verifier issue.
We have similar code in several places.
When we can print verbose() into the verifier log, we do. In other cases we just EFAULT which is a sign that something wrong with the verifier.
We don't use WARN in all those cases.
kernel test robot
build botreproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…
Hi Ilya,
[FYI, it's a private test report for your RFC patch.] [auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Ilya-Leoshkevich/bpf-Support-64-bit-pointers-to-kfuncs/20230215-054702 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20230214212809.242632-2-iii%40linux.ibm.com patch subject: [PATCH RFC bpf-next 1/1] bpf: Support 64-bit pointers to kfuncs config: i386-randconfig-a002-20230213 (https://download.01.org/0day-ci/archive/20230216/[email protected]/config) compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project f28c006a5895fc0e329fe15fead81e37457cb1d1)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/5b93c73d640845b2e5ddd1b4af608f4896379002
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Ilya-Leoshkevich/bpf-Support-64-bit-pointers-to-kfuncs/20230215-054702 git checkout 5b93c73d640845b2e5ddd1b4af608f4896379002
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 SHELL=/bin/bash kernel/bpf/
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> kernel/bpf/core.c:1227:13: warning: incompatible pointer to integer conversion assigning to 'u64' (aka 'unsigned long long') from 'u8 *' (aka 'unsigned char *') [-Wint-conversion]
*func_addr = addr;
^ ~~~~1 warning generated.
vim +1227 kernel/bpf/core.c
1181
1182 int bpf_jit_get_func_addr(const struct bpf_prog *prog,
1183 const struct bpf_insn *insn, bool extra_pass,
1184 u64 *func_addr, bool *func_addr_fixed)
1185 {
1186 s16 off = insn->off;
1187 s32 imm = insn->imm;
1188 bool fixed;
1189 u8 *addr;
1190 int err;
1191
1192 switch (insn->src_reg) {
1193 case BPF_PSEUDO_CALL:
1194 /* Place-holder address till the last pass has collected
1195 * all addresses for JITed subprograms in which case we
1196 * can pick them up from prog->aux.
1197 */
1198 if (!extra_pass)
1199 addr = NULL;
1200 else if (prog->aux->func &&
1201 off >= 0 && off < prog->aux->func_cnt)
1202 addr = (u8 *)prog->aux->func[off]->bpf_func;
1203 else
1204 return -EINVAL;
1205 fixed = false;
1206 break;
1207 case 0:
1208 /* Address of a BPF helper call. Since part of the core
1209 * kernel, it's always at a fixed location. __bpf_call_base
1210 * and the helper with imm relative to it are both in core
1211 * kernel.
1212 */
1213 addr = (u8 *)__bpf_call_base + imm;
1214 fixed = true;
1215 break;
1216 case BPF_PSEUDO_KFUNC_CALL:
1217 err = bpf_get_kfunc_addr(prog, imm, off, &addr);
1218 if (err)
1219 return err;
1220 fixed = true;
1221 break;
1222 default:
1223 return -EINVAL;
1224 }
1225
1226 *func_addr_fixed = fixed;
> 1227 *func_addr = addr;
1228 return 0;
1229 }
1230
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests
kernel test robot
build botreproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross #…
Hi Ilya,
[FYI, it's a private test report for your RFC patch.] [auto build test ERROR on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Ilya-Leoshkevich/bpf-Support-64-bit-pointers-to-kfuncs/20230215-054702 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20230214212809.242632-2-iii%40linux.ibm.com patch subject: [PATCH RFC bpf-next 1/1] bpf: Support 64-bit pointers to kfuncs config: loongarch-buildonly-randconfig-r003-20230214 (https://download.01.org/0day-ci/archive/20230217/[email protected]/config) compiler: loongarch64-linux-gcc (GCC) 12.1.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/5b93c73d640845b2e5ddd1b4af608f4896379002
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Ilya-Leoshkevich/bpf-Support-64-bit-pointers-to-kfuncs/20230215-054702 git checkout 5b93c73d640845b2e5ddd1b4af608f4896379002
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
kernel/bpf/core.c: In function 'bpf_jit_get_func_addr':
>> kernel/bpf/core.c:1217:23: error: implicit declaration of function 'bpf_get_kfunc_addr'; did you mean 'bpf_jit_get_func_addr'? [-Werror=implicit-function-declaration]
1217 | err = bpf_get_kfunc_addr(prog, imm, off, &addr);
| ^~~~~~~~~~~~~~~~~~
| bpf_jit_get_func_addr
kernel/bpf/core.c:1227:20: warning: assignment to 'u64' {aka 'long long unsigned int'} from 'u8 *' {aka 'unsigned char *'} makes integer from pointer without a cast [-Wint-conversion]
1227 | *func_addr = addr;
| ^ kernel/bpf/core.c: At top level:
kernel/bpf/core.c:1646:12: warning: no previous prototype for 'bpf_probe_read_kernel' [-Wmissing-prototypes]
1646 | u64 __weak bpf_probe_read_kernel(void *dst, u32 size, const void *unsafe_ptr)
| ^~~~~~~~~~~~~~~~~~~~~
kernel/bpf/core.c:2085:6: warning: no previous prototype for 'bpf_patch_call_args' [-Wmissing-prototypes]
2085 | void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth)
| ^~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +1217 kernel/bpf/core.c
1181
1182 int bpf_jit_get_func_addr(const struct bpf_prog *prog,
1183 const struct bpf_insn *insn, bool extra_pass,
1184 u64 *func_addr, bool *func_addr_fixed)
1185 {
1186 s16 off = insn->off;
1187 s32 imm = insn->imm;
1188 bool fixed;
1189 u8 *addr;
1190 int err;
1191
1192 switch (insn->src_reg) {
1193 case BPF_PSEUDO_CALL:
1194 /* Place-holder address till the last pass has collected
1195 * all addresses for JITed subprograms in which case we
1196 * can pick them up from prog->aux.
1197 */
1198 if (!extra_pass)
1199 addr = NULL;
1200 else if (prog->aux->func &&
1201 off >= 0 && off < prog->aux->func_cnt)
1202 addr = (u8 *)prog->aux->func[off]->bpf_func;
1203 else
1204 return -EINVAL;
1205 fixed = false;
1206 break;
1207 case 0:
1208 /* Address of a BPF helper call. Since part of the core
1209 * kernel, it's always at a fixed location. __bpf_call_base
1210 * and the helper with imm relative to it are both in core
1211 * kernel.
1212 */
1213 addr = (u8 *)__bpf_call_base + imm;
1214 fixed = true;
1215 break;
1216 case BPF_PSEUDO_KFUNC_CALL:
> 1217 err = bpf_get_kfunc_addr(prog, imm, off, &addr);
1218 if (err)
1219 return err;
1220 fixed = true;
1221 break;
1222 default:
1223 return -EINVAL;
1224 }
1225
1226 *func_addr_fixed = fixed;
1227 *func_addr = addr;
1228 return 0;
1229 }
1230
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests