genetlink: fix policy dump for dumps
Subsystems (2)
Jakub Kicinski
maintainerPATCH 1/1 genetlink: fix policy dump for dumps
Jonathan reports crashes when running net-next in Meta's fleet. Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.
The crash may look like this:
BUG: kernel NULL pointer dereference, address: 0000000000000cc0
RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0
ctrl_dumppolicy_start+0x19f/0x2f0
genl_start+0xe7/0x140Or we may trigger a warning:
WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80
RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80
ctrl_dumppolicy_put_op+0x214/0x360depending on what garbage we pick up from the stack.
---
CC: [email protected]
--- net/netlink/genetlink.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
net/netlink/genetlink.c +6 -6
@@ -1406,8 +1406,8 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) ctx->single_op = true; ctx->op =nla_get_u32 (tb[CTRL_ATTR_OP ]); -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , rt, &dump)) { +if (!!genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , rt, &doit) + + !!genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , rt, &dump) <1 ) {NL_SET_BAD_ATTR (cb->extack, tb[CTRL_ATTR_OP ]);return -ENOENT ; }
@@ -1551,10 +1551,10 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)if (ctx->single_op) {struct genl_split_ops doit, dumpit; -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , - ctx->rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , - ctx->rt, &dumpit)) { +if (!!genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , + ctx->rt, &doit) + + !!genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , + ctx->rt, &dumpit) <1 ) {WARN_ON (1 );return -ENOENT ; }
Signature · 1 line
2.38.1
Quoted 23 lines
-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 8, 2022 12:41 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Jakub Kicinski <[email protected]>; Jonathan Lemon <[email protected]>; Keller, Jacob E <[email protected]>
Subject: [PATCH net-next] genetlink: fix policy dump for dumpsJonathan reports crashes when running net-next in Meta's fleet. Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.
The crash may look like this:
BUG: kernel NULL pointer dereference, address: 0000000000000cc0 RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0 ctrl_dumppolicy_start+0x19f/0x2f0 genl_start+0xe7/0x140
Quoted 9 lines
Or we may trigger a warning:
WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80 RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80 ctrl_dumppolicy_put_op+0x214/0x360depending on what garbage we pick up from the stack.
Oops. Yay subtle bugs :D
Quoted 15 lines
Reported-by Jonathan Lemon <[email protected]>Fixes 26588edbef60 ("genetlink: support split policies in
ctrl_dumppolicy_put_op()")Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]--- net/netlink/genetlink.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/netlink/genetlink.c +0
-0
net/netlink/genetlink.c static int ctrl_dumppolicy_start(struct netlink_callback · line 1412
Quoted 9 lines
net/netlink/genetlink.c +2
-2@@ -1406,8 +1406,8 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) ctx->single_op = true; ctx->op =nla_get_u32 (tb[CTRL_ATTR_OP ]); -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , rt, &dump)) { +if (!!genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , rt, &doit) + + !!genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , rt, &dump) <1 ) {
A little bit tricky code here, but it makes sense. We could rewrite this to be a bit more verbose like:
doit_err = genl_get_cmd(.. GENL_CMD_CAP_DO ..); dumpit_err = genl_get_cmd(.. GENL_CMD_CAP_DUMPIT ..); if (doit_err && dumpit_err) {
...
}
That might be a bit easier to read than the !! ( ) + ( ) < 1 notation.
Either way I think it looks correct at least.
Thanks,
Jake
Quoted 21 lines
net/netlink/genetlink.c +0
-0NL_SET_BAD_ATTR (cb->extack, tb[CTRL_ATTR_OP ]);net/netlink/genetlink.c +4
-4return -ENOENT ; }@@ -1551,10 +1551,10 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)if (ctx->single_op) {struct genl_split_ops doit, dumpit; -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , - ctx->rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , - ctx->rt, &dumpit)) { +if (!!genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , + ctx->rt, &doit) + + !!genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , + ctx->rt, &dumpit) <1 ) {WARN_ON (1 );return -ENOENT ;}
--
2.38.1
Jakub Kicinski
maintainerQuoted 9 lines · Jacob E
A little bit tricky code here, but it makes sense. We could rewrite this to be a bit more verbose like:
doit_err= genl_get_cmd(..GENL_CMD_CAP_DO..);dumpit_err= genl_get_cmd(..GENL_CMD_CAP_DUMPIT..); if (doit_err&&dumpit_err) {
...
}That might be a bit easier to read than the !! ( ) + ( ) < 1 notation.
True, I should not give into the bit math temptations.
How about a helper:
net/netlink/genetlink.c +21 -9
@@ -282,6 +282,7 @@ genl_cmd_full_to_split(struct genl_split_ops *op,return 0 ; } +/* Must make sure that op is initialized to 0 on failure */ static int genl_get_cmd (u32 cmd,u8 flags,const struct genl_family *family,struct genl_split_ops *op)
@@ -302,6 +303,21 @@ genl_get_cmd(u32 cmd, u8 flags, const struct genl_family *family,return err; } +/* For policy dumping only, get ops of both do and dump. +* Fail if both are missing, genl_get_cmd() will zero-init in case of failure. +*/ +static int +genl_get_cmd_both (u32 cmd,const struct genl_family *family, +struct genl_split_ops *doit,struct genl_split_ops *dumpit) +{ +int err1, err2; + + err1 =genl_get_cmd (cmd,GENL_CMD_CAP_DO , family, doit); + err2 =genl_get_cmd (cmd,GENL_CMD_CAP_DUMP , family, dumpit); + +return err1 && err2 ? -ENOENT :0 ; +} +static bool genl_op_iter_init (const struct genl_family *family,struct genl_op_iter *iter) {
@@ -1406,10 +1422,10 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) ctx->single_op = true; ctx->op =nla_get_u32 (tb[CTRL_ATTR_OP ]); -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , rt, &dump)) { + err =genl_get_cmd_both (ctx->op, rt, &doit, &dump); +if (err) {NL_SET_BAD_ATTR (cb->extack, tb[CTRL_ATTR_OP ]); -return -ENOENT ; +return err; }if (doit.policy) {
@@ -1551,13 +1567,9 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)if (ctx->single_op) {struct genl_split_ops doit, dumpit; -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , - ctx->rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , - ctx->rt, &dumpit)) { -WARN_ON (1 ); +if (WARN_ON (genl_get_cmd_both (ctx->op, ctx->rt, + &doit, &dumpit)))return -ENOENT ; - }if (ctrl_dumppolicy_put_op (skb, cb, &doit, &dumpit))return skb->len;
Signature · 1 line
2.38.1
Quoted 20 lines
-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 8, 2022 3:44 PM
To: Keller, Jacob E <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Jonathan Lemon <[email protected]>
Subject: Re: [PATCH net-next] genetlink: fix policy dump for dumpsQuoted 1 line · Jacob E
A little bit tricky code here, but it makes sense. We could rewrite this to be a bit
more verbose like:
Quoted 7 lines
doit_err= genl_get_cmd(..GENL_CMD_CAP_DO..);dumpit_err= genl_get_cmd(..GENL_CMD_CAP_DUMPIT..); if (doit_err&&dumpit_err) {
...
}
Quoted 6 lines
Quoted 1 line
That might be a bit easier to read than the !! ( ) + ( ) < 1 notation.
True, I should not give into the bit math temptations.
How about a helper:
Much better, the little explanation about the zero init helps too.
Thanks,
Jake
Quoted 69 lines
net/netlink/genetlink.c +16
-0@@ -282,6 +282,7 @@ genl_cmd_full_to_split(struct genl_split_ops *op,return 0 ; } +/* Must make sure that op is initialized to 0 on failure */ static int genl_get_cmd (u32 cmd,u8 flags,const struct genl_family *family,struct genl_split_ops *op)@@ -302,6 +303,21 @@ genl_get_cmd(u32 cmd, u8 flags, const structgenl_family *family, return err; } +/* For policy dumping only, get ops of both do and dump. +* Fail if both are missing, genl_get_cmd() will zero-init in case of failure. +*/ +static int +genl_get_cmd_both (u32 cmd,const struct genl_family *family, +struct genl_split_ops *doit,struct genl_split_ops *dumpit) +{ +int err1, err2; + + err1 =genl_get_cmd (cmd,GENL_CMD_CAP_DO , family, doit); + err2 =genl_get_cmd (cmd,GENL_CMD_CAP_DUMP , family, dumpit); + +return err1 && err2 ? -ENOENT :0 ; +} +static bool genl_op_iter_init (const struct genl_family *family,struct genl_op_iter *iter){
net/netlink/genetlink.c +3
-3@@ -1406,10 +1422,10 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) ctx->single_op = true; ctx->op =nla_get_u32 (tb[CTRL_ATTR_OP ]); -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , rt, &dump)) { + err =genl_get_cmd_both (ctx->op, rt, &doit, &dump); +if (err) {NL_SET_BAD_ATTR (cb->extack, tb[CTRL_ATTR_OP ]); -return -ENOENT ; +return err; }net/netlink/genetlink.c +2
-6if (doit.policy) {@@ -1551,13 +1567,9 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)if (ctx->single_op) {struct genl_split_ops doit, dumpit; -if (genl_get_cmd (ctx->op,GENL_CMD_CAP_DO , - ctx->rt, &doit) && -genl_get_cmd (ctx->op,GENL_CMD_CAP_DUMP , - ctx->rt, &dumpit)) { -WARN_ON (1 ); +if (WARN_ON (genl_get_cmd_both (ctx->op, ctx->rt, + &doit, &dumpit)))return -ENOENT ; - }if (ctrl_dumppolicy_put_op (skb, cb, &doit, &dumpit))return
skb->len;
--
2.38.1
Quoted 1 line · Jakub Kicinski
Jonathan reports crashes when running net-next in Meta's fleet.
I experience these crashes too.
Quoted 19 lines
Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.
The crash may look like this:
BUG: kernel NULL pointer dereference, address: 0000000000000cc0 RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0 ctrl_dumppolicy_start+0x19f/0x2f0 genl_start+0xe7/0x140Or we may trigger a warning:
WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80 RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80 ctrl_dumppolicy_put_op+0x214/0x360depending on what garbage we pick up from the stack.
Quoted 8 lines
Reported-by Jonathan Lemon <[email protected]>Fixes 26588edbef60 ("genetlink: support split policies in ctrl_dumppolicy_put_op()")Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]--- net/netlink/genetlink.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.
Thanks
Leon Romanovsky
Tested-byQuoted 33 lines · Leon Romanovsky
Quoted 1 line · Jakub Kicinski
Jonathan reports crashes when running net-next in Meta's fleet.
I experience these crashes too.
Quoted 19 lines
Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.
The crash may look like this:
BUG: kernel NULL pointer dereference, address: 0000000000000cc0 RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0 ctrl_dumppolicy_start+0x19f/0x2f0 genl_start+0xe7/0x140Or we may trigger a warning:
WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80 RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80 ctrl_dumppolicy_put_op+0x214/0x360depending on what garbage we pick up from the stack.
Quoted 8 lines
Reported-by Jonathan Lemon <[email protected]>Fixes 26588edbef60 ("genetlink: support split policies in ctrl_dumppolicy_put_op()")Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]--- net/netlink/genetlink.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
Quoted 4 lines
I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.
Thanks,
Jakub Kicinski
maintainerQuoted 5 lines · On Wed, 9 Nov 2022 20:59:09 +0200 Leon Romanovsky wrote:
Quoted 3 lines
I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.
Tested-by Leon Romanovsky <[email protected]>
To be clear - did you test as posted or v2? Or doesn't matter? I'm wondering how applicable the tag is to v2.
Quoted 9 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 9 Nov 2022 20:59:09 +0200 Leon Romanovsky wrote:
Quoted 3 lines
I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.
Tested-by Leon Romanovsky <[email protected]>To be clear - did you test as posted or v2? Or doesn't matter? I'm wondering how applicable the tag is to v2.
FWIW: I just tested v2 and it passed our CI.
Quoted 9 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 9 Nov 2022 20:59:09 +0200 Leon Romanovsky wrote:
Quoted 3 lines
I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.
Tested-by Leon Romanovsky <[email protected]>To be clear - did you test as posted or v2? Or doesn't matter? I'm wondering how applicable the tag is to v2.
I tested this version https://lore.kernel.org/all/[email protected]
Thanks