Skip to contentSkip to conversation

genetlink: fix policy dump for dumps

PATCH In review Jakub Kicinski 4 participants 9 messages Nov 8, 2022 – Nov 10, 2022

Subsystems (2)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 genetlink: fix policy dump for dumps

Jonathan reports crashes when running net-next in Meta's fleet. Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.

The crash may look like this:

   BUG: kernel NULL pointer dereference, address: 0000000000000cc0
   RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0
     ctrl_dumppolicy_start+0x19f/0x2f0
     genl_start+0xe7/0x140

Or we may trigger a warning:

   WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80
   RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80
     ctrl_dumppolicy_put_op+0x214/0x360

depending on what garbage we pick up from the stack.

Fixes 26588edbef60 ("genetlink: support split policies in ctrl_dumppolicy_put_op()")

---
CC: [email protected]

---
 net/netlink/genetlink.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
net/netlink/genetlink.c +6 -6
index 9b7dfc45dd67..7b7bac9e7524 100644
@@ -1406,8 +1406,8 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
 		ctx->single_op = true;
 		ctx->op = nla_get_u32(tb[CTRL_ATTR_OP]);
 
-		if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO, rt, &doit) &&
-		    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP, rt, &dump)) {
+		if (!!genl_get_cmd(ctx->op, GENL_CMD_CAP_DO, rt, &doit) +
+		    !!genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP, rt, &dump) < 1) {
 			NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]);
 			return -ENOENT;
 		}
@@ -1551,10 +1551,10 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)
 		if (ctx->single_op) {
 			struct genl_split_ops doit, dumpit;
 
-			if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO,
-					 ctx->rt, &doit) &&
-			    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP,
-					 ctx->rt, &dumpit)) {
+			if (!!genl_get_cmd(ctx->op, GENL_CMD_CAP_DO,
+					   ctx->rt, &doit) +
+			    !!genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP,
+					   ctx->rt, &dumpit) < 1) {
 				WARN_ON(1);
 				return -ENOENT;
 			}
Signature · 1 line
2.38.1
#1 lore raw
Quoted 23 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 8, 2022 12:41 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Jakub Kicinski <[email protected]>; Jonathan Lemon <[email protected]>; Keller, Jacob E <[email protected]>
Subject: [PATCH net-next] genetlink: fix policy dump for dumps

Jonathan reports crashes when running net-next in Meta's fleet. Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.

The crash may look like this:

   BUG: kernel NULL pointer dereference, address: 0000000000000cc0
   RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0
     ctrl_dumppolicy_start+0x19f/0x2f0
     genl_start+0xe7/0x140
Quoted 9 lines

Or we may trigger a warning:

   WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87
netlink_policy_dump_get_policy_idx+0x79/0x80
   RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80
     ctrl_dumppolicy_put_op+0x214/0x360

depending on what garbage we pick up from the stack.

Oops. Yay subtle bugs :D

Quoted 15 lines
Fixes 26588edbef60 ("genetlink: support split policies in

ctrl_dumppolicy_put_op()")

---
CC: [email protected]

---
 net/netlink/genetlink.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
net/netlink/genetlink.c +0 -0
index 9b7dfc45dd67..7b7bac9e7524 100644

net/netlink/genetlink.c static int ctrl_dumppolicy_start(struct netlink_callback · line 1412

Quoted 9 lines
net/netlink/genetlink.c +2 -2
@@ -1406,8 +1406,8 @@ static int ctrl_dumppolicy_start(struct netlink_callback
 *cb)
 		ctx->single_op = true;
 		ctx->op = nla_get_u32(tb[CTRL_ATTR_OP]);
 
-		if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO, rt, &doit) &&
-		    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP, rt, &dump)) {
+		if (!!genl_get_cmd(ctx->op, GENL_CMD_CAP_DO, rt, &doit) +
+		    !!genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP, rt, &dump) <
 1) {

A little bit tricky code here, but it makes sense. We could rewrite this to be a bit more verbose like:

doit_err = genl_get_cmd(.. GENL_CMD_CAP_DO ..); dumpit_err = genl_get_cmd(.. GENL_CMD_CAP_DUMPIT ..); if (doit_err && dumpit_err) {
...
}

That might be a bit easier to read than the !! ( ) + ( ) < 1 notation.

Either way I think it looks correct at least.

Thanks,
Jake

Quoted 21 lines
net/netlink/genetlink.c +0 -0
 			NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]);
net/netlink/genetlink.c +4 -4
 			return -ENOENT;
 		}
@@ -1551,10 +1551,10 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct
 netlink_callback *cb)
 		if (ctx->single_op) {
 			struct genl_split_ops doit, dumpit;
 
-			if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO,
-					 ctx->rt, &doit) &&
-			    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP,
-					 ctx->rt, &dumpit)) {
+			if (!!genl_get_cmd(ctx->op, GENL_CMD_CAP_DO,
+					   ctx->rt, &doit) +
+			    !!genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP,
+					   ctx->rt, &dumpit) < 1) {
 				WARN_ON(1);
 				return -ENOENT;

}
--
2.38.1

author maintainer
#2 lore raw
Quoted 9 lines · Jacob E

A little bit tricky code here, but it makes sense. We could rewrite this to be a bit more verbose like:

doit_err = genl_get_cmd(.. GENL_CMD_CAP_DO ..); dumpit_err = genl_get_cmd(.. GENL_CMD_CAP_DUMPIT ..); if (doit_err && dumpit_err) {
...
}

That might be a bit easier to read than the !! ( ) + ( ) < 1 notation.

True, I should not give into the bit math temptations.

How about a helper:

net/netlink/genetlink.c +21 -9
index 9b7dfc45dd67..600993c80050 100644
@@ -282,6 +282,7 @@ genl_cmd_full_to_split(struct genl_split_ops *op,
 	return 0;
 }
 
+/* Must make sure that op is initialized to 0 on failure */
 static int
 genl_get_cmd(u32 cmd, u8 flags, const struct genl_family *family,
 	     struct genl_split_ops *op)
@@ -302,6 +303,21 @@ genl_get_cmd(u32 cmd, u8 flags, const struct genl_family *family,
 	return err;
 }
 
+/* For policy dumping only, get ops of both do and dump.
+ * Fail if both are missing, genl_get_cmd() will zero-init in case of failure.
+ */
+static int
+genl_get_cmd_both(u32 cmd, const struct genl_family *family,
+		  struct genl_split_ops *doit, struct genl_split_ops *dumpit)
+{
+	int err1, err2;
+
+	err1 = genl_get_cmd(cmd, GENL_CMD_CAP_DO, family, doit);
+	err2 = genl_get_cmd(cmd, GENL_CMD_CAP_DUMP, family, dumpit);
+
+	return err1 && err2 ? -ENOENT : 0;
+}
+
 static bool
 genl_op_iter_init(const struct genl_family *family, struct genl_op_iter *iter)
 {
@@ -1406,10 +1422,10 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
 		ctx->single_op = true;
 		ctx->op = nla_get_u32(tb[CTRL_ATTR_OP]);
 
-		if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO, rt, &doit) &&
-		    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP, rt, &dump)) {
+		err = genl_get_cmd_both(ctx->op, rt, &doit, &dump);
+		if (err) {
 			NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]);
-			return -ENOENT;
+			return err;
 		}
 
 		if (doit.policy) {
@@ -1551,13 +1567,9 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)
 		if (ctx->single_op) {
 			struct genl_split_ops doit, dumpit;
 
-			if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO,
-					 ctx->rt, &doit) &&
-			    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP,
-					 ctx->rt, &dumpit)) {
-				WARN_ON(1);
+			if (WARN_ON(genl_get_cmd_both(ctx->op, ctx->rt,
+						      &doit, &dumpit)))
 				return -ENOENT;
-			}
 
 			if (ctrl_dumppolicy_put_op(skb, cb, &doit, &dumpit))
 				return skb->len;
Signature · 1 line
2.38.1
#3 lore raw
Quoted 20 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 8, 2022 3:44 PM
To: Keller, Jacob E <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Jonathan Lemon <[email protected]>
Subject: Re: [PATCH net-next] genetlink: fix policy dump for dumps

Quoted 1 line · Jacob E

A little bit tricky code here, but it makes sense. We could rewrite this to be a bit

more verbose like:

Quoted 7 lines

doit_err = genl_get_cmd(.. GENL_CMD_CAP_DO ..); dumpit_err = genl_get_cmd(.. GENL_CMD_CAP_DUMPIT ..); if (doit_err && dumpit_err) {
...
}

Quoted 6 lines
Quoted 1 line

That might be a bit easier to read than the !! ( ) + ( ) < 1 notation.

True, I should not give into the bit math temptations.

How about a helper:

Much better, the little explanation about the zero init helps too.

Thanks,
Jake

Quoted 69 lines
net/netlink/genetlink.c +16 -0
index 9b7dfc45dd67..600993c80050 100644
@@ -282,6 +282,7 @@ genl_cmd_full_to_split(struct genl_split_ops *op,
 	return 0;
 }
 
+/* Must make sure that op is initialized to 0 on failure */
 static int
 genl_get_cmd(u32 cmd, u8 flags, const struct genl_family *family,
 	     struct genl_split_ops *op)
@@ -302,6 +303,21 @@ genl_get_cmd(u32 cmd, u8 flags, const struct
 genl_family *family,
 	return err;
 }
 
+/* For policy dumping only, get ops of both do and dump.
+ * Fail if both are missing, genl_get_cmd() will zero-init in case of failure.
+ */
+static int
+genl_get_cmd_both(u32 cmd, const struct genl_family *family,
+		  struct genl_split_ops *doit, struct genl_split_ops *dumpit)
+{
+	int err1, err2;
+
+	err1 = genl_get_cmd(cmd, GENL_CMD_CAP_DO, family, doit);
+	err2 = genl_get_cmd(cmd, GENL_CMD_CAP_DUMP, family, dumpit);
+
+	return err1 && err2 ? -ENOENT : 0;
+}
+
 static bool
 genl_op_iter_init(const struct genl_family *family, struct genl_op_iter *iter)

{

net/netlink/genetlink.c +3 -3
@@ -1406,10 +1422,10 @@ static int ctrl_dumppolicy_start(struct
 netlink_callback *cb)
 		ctx->single_op = true;
 		ctx->op = nla_get_u32(tb[CTRL_ATTR_OP]);
 
-		if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO, rt, &doit) &&
-		    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP, rt, &dump)) {
+		err = genl_get_cmd_both(ctx->op, rt, &doit, &dump);
+		if (err) {
 			NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]);
-			return -ENOENT;
+			return err;
 		}
 
net/netlink/genetlink.c +2 -6
 		if (doit.policy) {
@@ -1551,13 +1567,9 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct
 netlink_callback *cb)
 		if (ctx->single_op) {
 			struct genl_split_ops doit, dumpit;
 
-			if (genl_get_cmd(ctx->op, GENL_CMD_CAP_DO,
-					 ctx->rt, &doit) &&
-			    genl_get_cmd(ctx->op, GENL_CMD_CAP_DUMP,
-					 ctx->rt, &dumpit)) {
-				WARN_ON(1);
+			if (WARN_ON(genl_get_cmd_both(ctx->op, ctx->rt,
+						      &doit, &dumpit)))
 				return -ENOENT;
-			}
 
 			if (ctrl_dumppolicy_put_op(skb, cb, &doit, &dumpit))

return skb->len;
--
2.38.1

#4 lore raw
In reply to Jakub Kicinski
Quoted 1 line · Jakub Kicinski

Jonathan reports crashes when running net-next in Meta's fleet.

I experience these crashes too.

Quoted 19 lines

Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.

The crash may look like this:

   BUG: kernel NULL pointer dereference, address: 0000000000000cc0
   RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0
     ctrl_dumppolicy_start+0x19f/0x2f0
     genl_start+0xe7/0x140

Or we may trigger a warning:

   WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80
   RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80
     ctrl_dumppolicy_put_op+0x214/0x360

depending on what garbage we pick up from the stack.

Quoted 8 lines
Fixes 26588edbef60 ("genetlink: support split policies in ctrl_dumppolicy_put_op()")

---
CC: [email protected]

---
 net/netlink/genetlink.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.

Thanks

Tested-by
#5 lore raw
Quoted 33 lines · Leon Romanovsky
Quoted 1 line · Jakub Kicinski

Jonathan reports crashes when running net-next in Meta's fleet.

I experience these crashes too.

Quoted 19 lines

Stats collection uses ethtool -I which does a per-op policy dump to check if stats are supported. We don't initialize the dumpit information if doit succeeds due to evaluation short-circuiting.

The crash may look like this:

   BUG: kernel NULL pointer dereference, address: 0000000000000cc0
   RIP: 0010:netlink_policy_dump_add_policy+0x174/0x2a0
     ctrl_dumppolicy_start+0x19f/0x2f0
     genl_start+0xe7/0x140

Or we may trigger a warning:

   WARNING: CPU: 1 PID: 785 at net/netlink/policy.c:87 netlink_policy_dump_get_policy_idx+0x79/0x80
   RIP: 0010:netlink_policy_dump_get_policy_idx+0x79/0x80
     ctrl_dumppolicy_put_op+0x214/0x360

depending on what garbage we pick up from the stack.

Quoted 8 lines
Fixes 26588edbef60 ("genetlink: support split policies in ctrl_dumppolicy_put_op()")

---
CC: [email protected]

---
 net/netlink/genetlink.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
Quoted 4 lines

I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.

Thanks,

author maintainer
#6 lore raw
Quoted 5 lines · On Wed, 9 Nov 2022 20:59:09 +0200 Leon Romanovsky wrote:
Quoted 3 lines

I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.

To be clear - did you test as posted or v2? Or doesn't matter? I'm wondering how applicable the tag is to v2.

#7 lore raw
Quoted 9 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 9 Nov 2022 20:59:09 +0200 Leon Romanovsky wrote:
Quoted 3 lines

I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.

To be clear - did you test as posted or v2? Or doesn't matter? I'm wondering how applicable the tag is to v2.

FWIW: I just tested v2 and it passed our CI.

#8 lore raw
In reply to Jakub Kicinski
Quoted 9 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 9 Nov 2022 20:59:09 +0200 Leon Romanovsky wrote:
Quoted 3 lines

I added your updated patch to my CI run. Unfortunately, the regression system is overloaded due to nightly regression so won't be able to get results in sensible time frame.

To be clear - did you test as posted or v2? Or doesn't matter? I'm wondering how applicable the tag is to v2.

I tested this version https://lore.kernel.org/all/[email protected]

Thanks