Skip to contentSkip to conversation

tipc: Don't using smp_processor_id() in preemptible code

PATCH In review [email protected] 3 participants 6 messages Jul 14, 2020 – Jul 15, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 tipc: Don't using smp_processor_id() in preemptible code

From: Zhang Qiang <[email protected]>

CPU: 0 PID: 6801 Comm: syz-executor201 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x18f/0x20d lib/dump_stack.c:118
 check_preemption_disabled+0x128/0x130 lib/smp_processor_id.c:48
 tipc_aead_tfm_next net/tipc/crypto.c:402 [inline]
 tipc_aead_encrypt net/tipc/crypto.c:639 [inline]
 tipc_crypto_xmit+0x80a/0x2790 net/tipc/crypto.c:1605
 tipc_bearer_xmit_skb+0x180/0x3f0 net/tipc/bearer.c:523
 tipc_enable_bearer+0xb1d/0xdc0 net/tipc/bearer.c:331
 __tipc_nl_bearer_enable+0x2bf/0x390 net/tipc/bearer.c:995
 __tipc_nl_compat_doit net/tipc/netlink_compat.c:361 [inline]
 tipc_nl_compat_doit+0x440/0x640 net/tipc/netlink_compat.c:383
 tipc_nl_compat_handle net/tipc/netlink_compat.c:1268 [inline]
 tipc_nl_compat_recv+0x4ef/0xb40 net/tipc/netlink_compat.c:1311
 genl_family_rcv_msg_doit net/netlink/genetlink.c:669 [inline]
 genl_family_rcv_msg net/netlink/genetlink.c:714 [inline]
 genl_rcv_msg+0x61d/0x980 net/netlink/genetlink.c:731
 netlink_rcv_skb+0x15a/0x430 net/netlink/af_netlink.c:2469
 genl_rcv+0x24/0x40 net/netlink/genetlink.c:742
 netlink_unicast_kernel net/netlink/af_netlink.c:1303 [inline]
 netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1329
 netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1918
 sock_sendmsg_nosec net/socket.c:652 [inline]
 sock_sendmsg+0xcf/0x120 net/socket.c:672
 ____sys_sendmsg+0x6e8/0x810 net/socket.c:2352
 ___sys_sendmsg+0xf3/0x170 net/socket.c:2406
 __sys_sendmsg+0xe5/0x1b0 net/socket.c:2439
 do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x4476a9
Code: Bad RIP value.
RSP: 002b:00007fff2b6d5168 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 000000000000
Fixes fc1b6d6de2208 ("tipc: introduce TIPC encryption & authentication")
Reported-by [email protected]

---
v1->v2:
add fixes tags.

 net/tipc/crypto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/crypto.c +2 -1
index 8c47ded2edb6..520af0afe1b3 100644
@@ -399,9 +399,10 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-	struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+	struct tipc_tfm **tfm_entry = get_cpu_ptr(aead->tfm_entry);
 
 	*tfm_entry = list_next_entry(*tfm_entry, list);
+	put_cpu_ptr(tfm_entry);
 	return (*tfm_entry)->tfm;
 }
 
Signature · 1 line
2.24.1
#1 lore raw
Quoted 23 lines · On 7/14/20 1:05 AM, [email protected] wrote:

From: Zhang Qiang <[email protected]>

CPU: 0 PID: 6801 Comm: syz-executor201 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine,

BIOS Google 01/01/2011

Fixes fc1b6d6de2208 ("tipc: introduce TIPC encryption & authentication")
Reported-by [email protected]

---
v1->v2:
add fixes tags.

 net/tipc/crypto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/crypto.c +0 -0
index 8c47ded2edb6..520af0afe1b3 100644
@@ -399,9 +399,10 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */

net/tipc/crypto.c static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val) · line 408

Quoted 10 lines
net/tipc/crypto.c +2 -1
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-	struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+	struct tipc_tfm **tfm_entry = get_cpu_ptr(aead->tfm_entry);
 
 	*tfm_entry = list_next_entry(*tfm_entry, list);
+	put_cpu_ptr(tfm_entry);
 	return (*tfm_entry)->tfm;
 }
 

You have not explained why this was safe.

This seems to hide a real bug.

Presumably callers of this function should have disable preemption, and maybe interrupts as well.

Right after put_cpu_ptr(tfm_entry), this thread could migrate to another cpu, and still access data owned by the old cpu.

#2 lore raw

________________________________________
发件人: Eric Dumazet <[email protected]>
发送时间: 2020年7月14日 22:15
收件人: Zhang, Qiang; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Xue, Ying 抄送: [email protected]; [email protected]; [email protected]
主题: Re: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

Quoted 34 lines · On 7/14/20 1:05 AM, [email protected] wrote:

From: Zhang Qiang <[email protected]>

CPU: 0 PID: 6801 Comm: syz-executor201 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine,

BIOS Google 01/01/2011

Fixes fc1b6d6de2208 ("tipc: introduce TIPC encryption & authentication")
Reported-by [email protected]

---
v1->v2:
add fixes tags.

 net/tipc/crypto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/crypto.c +2 -1
index 8c47ded2edb6..520af0afe1b3 100644
@@ -399,9 +399,10 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-     struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+     struct tipc_tfm **tfm_entry = get_cpu_ptr(aead->tfm_entry);
 
      *tfm_entry = list_next_entry(*tfm_entry, list);
+     put_cpu_ptr(tfm_entry);
      return (*tfm_entry)->tfm;
 }
 
Quoted 8 lines

You have not explained why this was safe.

This seems to hide a real bug.

Presumably callers of this function should have disable preemption, and maybe > interrupts as well.

Right after put_cpu_ptr(tfm_entry), this thread could migrate to another cpu, >and still access data owned by the old cpu.

Thanks for you suggest, I will check code again.

#3 lore raw
Quoted 59 lines

-----Original Message-----
From: Zhang, Qiang <[email protected]>
Sent: Wednesday, July 15, 2020 9:13 AM
To: Eric Dumazet <[email protected]>; [email protected]; [email protected]; [email protected]; Tuong Tong Lien <[email protected]>; Xue, Ying <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: 回复: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

________________________________________
发件人: Eric Dumazet <[email protected]>
发送时间: 2020年7月14日 22:15
收件人: Zhang, Qiang; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Xue, Ying
抄送: [email protected]; [email protected]; [email protected] 主题: Re: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

Quoted 36 lines · On 7/14/20 1:05 AM, [email protected] wrote:

From: Zhang Qiang <[email protected]>

CPU: 0 PID: 6801 Comm: syz-executor201 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine,

BIOS Google 01/01/2011

Fixes fc1b6d6de2208 ("tipc: introduce TIPC encryption & authentication")
Reported-by [email protected]

---
v1->v2:
add fixes tags.

 net/tipc/crypto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/crypto.c +2 -1
index 8c47ded2edb6..520af0afe1b3 100644
@@ -399,9 +399,10 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-     struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+     struct tipc_tfm **tfm_entry = get_cpu_ptr(aead->tfm_entry);
 
      *tfm_entry = list_next_entry(*tfm_entry, list);
+     put_cpu_ptr(tfm_entry);
      return (*tfm_entry)->tfm;
 }
 

You have not explained why this was safe.

Quoted 9 lines
Quoted 6 lines

This seems to hide a real bug.

Presumably callers of this function should have disable preemption, and maybe > interrupts as well.

Right after put_cpu_ptr(tfm_entry), this thread could migrate to another cpu, >and still access data owned by the old cpu.

Thanks for you suggest, I will check code again.

Actually, last week I sent a similar patch to tipc-discussion which covers the case as well (there is also another place causing the same issue...). If you don't mind, you can take a look at below (just copied/pasted).

BR/Tuong

-----Original Message-----
From: Tuong Tong Lien <[email protected]>
Sent: Friday, July 10, 2020 5:11 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: tipc-dek <[email protected]>
Subject: [PATCH RFC 1/5] tipc: fix using smp_processor_id() in preemptible

The 'this_cpu_ptr()' is used to obtain the AEAD key' TFM on the current CPU for encryption, however the execution can be preemptible since it's actually user-space context, so the 'using smp_processor_id() in preemptible' has been observed.

We fix the issue by using the 'get/put_cpu_ptr()' API which consists of a 'preempt_disable()' instead.

---
 net/tipc/crypto.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
net/tipc/crypto.c +9 -3
index c8c47fc72653..1827ce4fac5d 100644
@@ -326,7 +326,8 @@ static void tipc_aead_free(struct rcu_head *rp)
 	if (aead->cloned) {
 		tipc_aead_put(aead->cloned);
 	} else {
-		head = *this_cpu_ptr(aead->tfm_entry);
+		head = *get_cpu_ptr(aead->tfm_entry);
+		put_cpu_ptr(aead->tfm_entry);
 		list_for_each_entry_safe(tfm_entry, tmp, &head->list, list) {
 			crypto_free_aead(tfm_entry->tfm);
 			list_del(&tfm_entry->list);
@@ -399,10 +400,15 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-	struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+	struct tipc_tfm **tfm_entry;
+	struct crypto_aead *tfm;
 
+	tfm_entry = get_cpu_ptr(aead->tfm_entry);
 	*tfm_entry = list_next_entry(*tfm_entry, list);
-	return (*tfm_entry)->tfm;
+	tfm = (*tfm_entry)->tfm;
+	put_cpu_ptr(tfm_entry);
+
+	return tfm;
 }
 
 /**
Signature · 1 line
2.13.7
#4 lore raw

________________________________________
发件人: Tuong Tong Lien <[email protected]>
发送时间: 2020年7月15日 11:53
收件人: Zhang, Qiang; Eric Dumazet; [email protected]; [email protected]; [email protected]; Xue, Ying 抄送: [email protected]; [email protected]; [email protected] 主题: RE: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

Quoted 64 lines

-----Original Message-----
From: Zhang, Qiang <[email protected]>
Sent: Wednesday, July 15, 2020 9:13 AM
To: Eric Dumazet <[email protected]>; [email protected]; [email protected]; [email protected]; Tuong Tong Lien <[email protected]>; Xue, Ying <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: 回复: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

________________________________________
发件人: Eric Dumazet <[email protected]>
发送时间: 2020年7月14日 22:15
收件人: Zhang, Qiang; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Xue, Ying
抄送: [email protected]; [email protected]; [email protected] 主题: Re: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

Quoted 34 lines · On 7/14/20 1:05 AM, [email protected] wrote:

From: Zhang Qiang <[email protected]>

CPU: 0 PID: 6801 Comm: syz-executor201 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine,

BIOS Google 01/01/2011

Fixes fc1b6d6de2208 ("tipc: introduce TIPC encryption & authentication")
Reported-by [email protected]

---
v1->v2:
add fixes tags.

 net/tipc/crypto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/crypto.c +2 -1
index 8c47ded2edb6..520af0afe1b3 100644
@@ -399,9 +399,10 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-     struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+     struct tipc_tfm **tfm_entry = get_cpu_ptr(aead->tfm_entry);
 
      *tfm_entry = list_next_entry(*tfm_entry, list);
+     put_cpu_ptr(tfm_entry);
      return (*tfm_entry)->tfm;
 }
 
Quoted 8 lines

You have not explained why this was safe.

This seems to hide a real bug.

Presumably callers of this function should have disable preemption, and maybe > interrupts as well.

Right after put_cpu_ptr(tfm_entry), this thread could migrate to another cpu, >and still access data owned by the old cpu.

Quoted 8 lines

Thanks for you suggest, I will check code again.

Actually, last week I sent a similar patch to tipc-discussion which covers the case as well (there is also another place causing the same issue...). If you don't mind, you can take a look at below (just copied/pasted).

BR/Tuong

Hi Tuong Tong Lien

The tipc_aead_free is RCU callback, this func is called in softirq context which preemption has been banned
so should not add preempt_disable/enable.

thanks
Zhang Qiang

Quoted 5 lines

-----Original Message-----
From: Tuong Tong Lien <[email protected]>
Sent: Friday, July 10, 2020 5:11 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: tipc-dek <[email protected]>

Subject: [PATCH RFC 1/5] tipc: fix using smp_processor_id() in preemptible

Quoted 2 lines

The 'this_cpu_ptr()' is used to obtain the AEAD key' TFM on the current

CPU for encryption, however the execution can be preemptible since it's actually user-space context, so the 'using smp_processor_id() in preemptible' has been observed.

We fix the issue by using the 'get/put_cpu_ptr()' API which consists of a 'preempt_disable()' instead.

---
 net/tipc/crypto.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
net/tipc/crypto.c +9 -3
index c8c47fc72653..1827ce4fac5d 100644
@@ -326,7 +326,8 @@ static void tipc_aead_free(struct rcu_head *rp)
        if (aead->cloned) {
                tipc_aead_put(aead->cloned);
        } else {
-               head = *this_cpu_ptr(aead->tfm_entry);
+               head = *get_cpu_ptr(aead->tfm_entry);
+               put_cpu_ptr(aead->tfm_entry);
                list_for_each_entry_safe(tfm_entry, tmp, &head->list, list) {
                        crypto_free_aead(tfm_entry->tfm);
                        list_del(&tfm_entry->list);
@@ -399,10 +400,15 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-       struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+       struct tipc_tfm **tfm_entry;
+       struct crypto_aead *tfm;
 
+       tfm_entry = get_cpu_ptr(aead->tfm_entry);
        *tfm_entry = list_next_entry(*tfm_entry, list);
-       return (*tfm_entry)->tfm;
+       tfm = (*tfm_entry)->tfm;
+       put_cpu_ptr(tfm_entry);
+
+       return tfm;
 }
 
 /**

--
2.13.7

#5 lore raw

________________________________________
发件人: Zhang, Qiang <[email protected]>
发送时间: 2020年7月15日 13:27
收件人: Tuong Tong Lien; Eric Dumazet; [email protected]; [email protected]; [email protected]; Xue, Ying 抄送: [email protected]; [email protected]; [email protected] 主题: 回复: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

________________________________________
发件人: Tuong Tong Lien <[email protected]>
发送时间: 2020年7月15日 11:53
收件人: Zhang, Qiang; Eric Dumazet; [email protected]; [email protected]; [email protected]; Xue, Ying 抄送: [email protected]; [email protected]; [email protected] 主题: RE: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

Quoted 74 lines

-----Original Message-----
From: Zhang, Qiang <[email protected]>
Sent: Wednesday, July 15, 2020 9:13 AM
To: Eric Dumazet <[email protected]>; [email protected]; [email protected]; [email protected]; Tuong Tong Lien <[email protected]>; Xue, Ying <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: 回复: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

________________________________________
发件人: Eric Dumazet <[email protected]>
发送时间: 2020年7月14日 22:15
收件人: Zhang, Qiang; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Xue, Ying
抄送: [email protected]; [email protected]; [email protected] 主题: Re: [PATCH v2] tipc: Don't using smp_processor_id() in preemptible code

Quoted 34 lines · On 7/14/20 1:05 AM, [email protected] wrote:

From: Zhang Qiang <[email protected]>

CPU: 0 PID: 6801 Comm: syz-executor201 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine,

BIOS Google 01/01/2011

Fixes fc1b6d6de2208 ("tipc: introduce TIPC encryption & authentication")
Reported-by [email protected]

---
v1->v2:
add fixes tags.

 net/tipc/crypto.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/tipc/crypto.c +2 -1
index 8c47ded2edb6..520af0afe1b3 100644
@@ -399,9 +399,10 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-     struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+     struct tipc_tfm **tfm_entry = get_cpu_ptr(aead->tfm_entry);
 
      *tfm_entry = list_next_entry(*tfm_entry, list);
+     put_cpu_ptr(tfm_entry);
      return (*tfm_entry)->tfm;
 }
 
Quoted 8 lines

You have not explained why this was safe.

This seems to hide a real bug.

Presumably callers of this function should have disable preemption, and maybe > interrupts as well.

Right after put_cpu_ptr(tfm_entry), this thread could migrate to another cpu, >and still access data owned by the old cpu.

Thanks for you suggest, I will check code again.

Actually, last week I sent a similar patch to tipc-discussion which covers the case as well (there is also another place causing the same issue...). If you don't mind, you can take a look at below (just copied/pasted).

BR/Tuong

Quoted 8 lines

Hi Tuong Tong Lien

The tipc_aead_free is RCU callback, this func is called in softirq context which preemption has been banned
so should not add preempt_disable/enable.

thanks
Zhang Qiang

sorry there are some questions in my reply. the tipc_aead_free function may also be called in the thread context. if enable CONFIG_RCU_BOOST

Quoted 5 lines

-----Original Message-----
From: Tuong Tong Lien <[email protected]>
Sent: Friday, July 10, 2020 5:11 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: tipc-dek <[email protected]>

Subject: [PATCH RFC 1/5] tipc: fix using smp_processor_id() in preemptible

Quoted 2 lines

The 'this_cpu_ptr()' is used to obtain the AEAD key' TFM on the current

CPU for encryption, however the execution can be preemptible since it's actually user-space context, so the 'using smp_processor_id() in preemptible' has been observed.

We fix the issue by using the 'get/put_cpu_ptr()' API which consists of a 'preempt_disable()' instead.

---
 net/tipc/crypto.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
net/tipc/crypto.c +9 -3
index c8c47fc72653..1827ce4fac5d 100644
@@ -326,7 +326,8 @@ static void tipc_aead_free(struct rcu_head *rp)
        if (aead->cloned) {
                tipc_aead_put(aead->cloned);
        } else {
-               head = *this_cpu_ptr(aead->tfm_entry);
+               head = *get_cpu_ptr(aead->tfm_entry);
+               put_cpu_ptr(aead->tfm_entry);
                list_for_each_entry_safe(tfm_entry, tmp, &head->list, list) {
                        crypto_free_aead(tfm_entry->tfm);
                        list_del(&tfm_entry->list);
@@ -399,10 +400,15 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
  */
 static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
 {
-       struct tipc_tfm **tfm_entry = this_cpu_ptr(aead->tfm_entry);
+       struct tipc_tfm **tfm_entry;
+       struct crypto_aead *tfm;
 
+       tfm_entry = get_cpu_ptr(aead->tfm_entry);
        *tfm_entry = list_next_entry(*tfm_entry, list);
-       return (*tfm_entry)->tfm;
+       tfm = (*tfm_entry)->tfm;
+       put_cpu_ptr(tfm_entry);
+
+       return tfm;
 }
 
 /**

--
2.13.7