l2tp: Serialize access to sk_user_data with sk_callback_lock
Subsystems (3)
PATCH v4 1/1 l2tp: Serialize access to sk_user_data with sk_callback_lock
sk->sk_user_data has multiple users, which are not compatible with each other. Writers must synchronize by grabbing the sk->sk_callback_lock.
l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the sk_callback_lock also inside the sk_destruct callback overridden by l2tp, even though there should be no refs allowing access to the sock at the time when sk_destruct gets called.
v4:
- serialize write to sk_user_data in l2tp sk_destruct
v3:
- switch from sock lock to sk_callback_lock
- document write-protection for sk_user_data
v2:
- update Fixes to point to origin of the bug
- use real names in Reported/Tested-by tags
---
This took me forever. Sorry about that.
include/net/sock.h | 2 +- net/l2tp/l2tp_core.c | 19 +++++++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-)
include/net/sock.h +1 -1
@@ -323,7 +323,7 @@ struct sk_filter;* @sk_tskey: counter to disambiguate concurrent tstamp requests * @sk_zckey: counter to order MSG_ZEROCOPY notifications * @sk_socket: Identd and reporting IO signals -* @sk_user_data: RPC layer private data +* @sk_user_data: RPC layer private data. Write-protected by @sk_callback_lock. * @sk_frag: cached page frag * @sk_peek_off: current peek_offset value * @sk_send_head: front of stuff to transmit
net/l2tp/l2tp_core.c +13 -6
@@ -1150,8 +1150,10 @@ static void l2tp_tunnel_destruct(struct sock *sk) }/* Remove hooks into tunnel socket */ +write_lock_bh (&sk->sk_callback_lock); sk->sk_destruct = tunnel->old_sk_destruct; sk->sk_user_data =NULL ; +write_unlock_bh (&sk->sk_callback_lock);/* Call the original destructor */ if (sk->sk_destruct)
@@ -1469,16 +1471,18 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, sock =sockfd_lookup (tunnel->fd, &ret);if (!sock) goto err; - - ret =l2tp_validate_socket (sock->sk, net, tunnel->encap); -if (ret <0 ) - goto err_sock; } + sk = sock->sk; +write_lock (&sk->sk_callback_lock); + + ret =l2tp_validate_socket (sk, net, tunnel->encap); +if (ret <0 ) + goto err_sock; + tunnel->l2tp_net = net; pn =l2tp_pernet (net); - sk = sock->sk;sock_hold (sk); tunnel->sock = sk;
@@ -1504,7 +1508,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,setup_udp_tunnel_sock (net, sock, &udp_cfg); }else { - sk->sk_user_data = tunnel; +rcu_assign_sk_user_data (sk, tunnel); } tunnel->old_sk_destruct = sk->sk_destruct;
@@ -1518,6 +1522,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,if (tunnel->fd >=0 )sockfd_put (sock); +write_unlock (&sk->sk_callback_lock);return 0 ; err_sock:
@@ -1525,6 +1530,8 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,sock_release (sock); elsesockfd_put (sock ); + +write_unlock (&sk->sk_callback_lock); err:return ret; }
Signature · 1 line
2.38.1
Quoted 13 lines · Jakub Sitnicki
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.v4:
- serialize write tosk_user_datain l2tpsk_destruct
Quoted 7 lines
v3:
- switch from sock lock tosk_callback_lock
- document write-protection forsk_user_datav2:
- update Fixes to point to origin of the bug
- use real names in Reported/Tested-by tags
LGTM, thanks Jakub.
Quoted 94 lines
Cc Tom Parkin <[email protected]>Fixes 3557baabf280 ("[L2TP]: PPP over L2TP driver core")Reported-by Haowei Yan <[email protected]>Signed-off-by Jakub Sitnicki <[email protected]>---
This took me forever. Sorry about that.
include/net/sock.h | 2 +- net/l2tp/l2tp_core.c | 19 +++++++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-)include/net/sock.h +1
-1@@ -323,7 +323,7 @@ struct sk_filter;* @sk_tskey: counter to disambiguate concurrent tstamp requests * @sk_zckey: counter to order MSG_ZEROCOPY notifications * @sk_socket: Identd and reporting IO signals -* @sk_user_data: RPC layer private data +* @sk_user_data: RPC layer private data. Write-protected by @sk_callback_lock. * @sk_frag: cached page frag * @sk_peek_off: current peek_offset value * @sk_send_head: front of stuff to transmit net/l2tp/l2tp_core.c +13
-6@@ -1150,8 +1150,10 @@ static void l2tp_tunnel_destruct(struct sock *sk) }/* Remove hooks into tunnel socket */ +write_lock_bh (&sk->sk_callback_lock); sk->sk_destruct = tunnel->old_sk_destruct; sk->sk_user_data =NULL ; +write_unlock_bh (&sk->sk_callback_lock);/* Call the original destructor */ if (sk->sk_destruct)@@ -1469,16 +1471,18 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, sock =sockfd_lookup (tunnel->fd, &ret);if (!sock) goto err; - - ret =l2tp_validate_socket (sock->sk, net, tunnel->encap); -if (ret <0 ) - goto err_sock; } + sk = sock->sk; +write_lock (&sk->sk_callback_lock); + + ret =l2tp_validate_socket (sk, net, tunnel->encap); +if (ret <0 ) + goto err_sock; + tunnel->l2tp_net = net; pn =l2tp_pernet (net); - sk = sock->sk;sock_hold (sk); tunnel->sock = sk;@@ -1504,7 +1508,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,setup_udp_tunnel_sock (net, sock, &udp_cfg); }else { - sk->sk_user_data = tunnel; +rcu_assign_sk_user_data (sk, tunnel); } tunnel->old_sk_destruct = sk->sk_destruct;@@ -1518,6 +1522,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,if (tunnel->fd >=0 )sockfd_put (sock); +write_unlock (&sk->sk_callback_lock);return 0 ; err_sock:@@ -1525,6 +1530,8 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,sock_release (sock); elsesockfd_put (sock ); + +write_unlock (&sk->sk_callback_lock); err:return ret; }Signature · 1 line
2.38.1
Signature · 4 lines
Tom Parkin Katalix Systems Ltd https://katalix.com Catalysts for your Embedded Linux software development
This patch was applied to netdev/net.git (master) by David S. Miller <[email protected]>:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:
Quoted 11 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access to sk_user_data with sk_callback_lock
https://git.kernel.org/netdev/net/c/b68777d54fac
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.
Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54fac
I guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lock always needs _bh safety.
I will send something like:
net/l2tp/l2tp_core.c +0 -0
100644
net/l2tp/l2tp_core.c +0 -0
@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel
*tunnel, struct net *net,
}
sk = sock->sk;- write_lock(&sk->sk_callback_lock);
+ write_lock_bh(&sk->sk_callback_lock);
ret = l2tp_validate_socket(sk, net, tunnel->encap);
if (ret < 0)
@@ -1522,7 +1522,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel
*tunnel, struct net *net,
if (tunnel->fd >= 0)
sockfd_put(sock);- write_unlock(&sk->sk_callback_lock);
+ write_unlock_bh(&sk->sk_callback_lock);
return 0; err_sock:
@@ -1531,7 +1531,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net,
else
sockfd_put(sock);- write_unlock(&sk->sk_callback_lock);
+ write_unlock_bh(&sk->sk_callback_lock);
err:
return ret;
}
Quoted 39 lines · Eric Dumazet
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54facI guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lockalways needs_bhsafety.I will send something like:
net/l2tp/l2tp_core.c +0
-0100644
net/l2tp/l2tp_core.c int l2tp_tunnel_register(struct l2tp_tunnel · line 1478
Quoted 6 lines
net/l2tp/l2tp_core.c +1
-1@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, } sk = sock->sk; -write_lock (&sk->sk_callback_lock); +write_lock_bh (&sk->sk_callback_lock);
Unfortunately this might still not work, because setup_udp_tunnel_sock->udp_encap_enable() probably could sleep in static_branch_inc() ?
I will release the syzbot report, and let you folks work on a fix, thanks.
Quoted 51 lines · Eric Dumazet
Quoted 39 lines · Eric Dumazet
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54facI guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lockalways needs_bhsafety.I will send something like:
net/l2tp/l2tp_core.c +0
-0100644
net/l2tp/l2tp_core.c int l2tp_tunnel_register(struct l2tp_tunnel · line 1478
Quoted 6 lines
net/l2tp/l2tp_core.c +1
-1@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, } sk = sock->sk; -write_lock (&sk->sk_callback_lock); +write_lock_bh (&sk->sk_callback_lock);Unfortunately this might still not work, because
setup_udp_tunnel_sock->udp_encap_enable()probably could sleep instatic_branch_inc()?
Quoted 1 line
I will release the syzbot report, and let you folks work on a fix, thanks.
Ah, the problem is with pppol2tp_connect racing with itself. Thanks for the syzbot report. I will take a look. I live for debugging deadlocks :-)
Quoted 53 lines · Jakub Sitnicki
Quoted 51 lines · Eric Dumazet
Quoted 39 lines · Eric Dumazet
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54facI guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lockalways needs_bhsafety.I will send something like:
net/l2tp/l2tp_core.c +0
-0100644
net/l2tp/l2tp_core.c int l2tp_tunnel_register(struct l2tp_tunnel · line 1478
Quoted 6 lines
net/l2tp/l2tp_core.c +1
-1@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, } sk = sock->sk; -write_lock (&sk->sk_callback_lock); +write_lock_bh (&sk->sk_callback_lock);Unfortunately this might still not work, because
setup_udp_tunnel_sock->udp_encap_enable()probably could sleep instatic_branch_inc()?
Quoted 5 lines
Quoted 1 line
I will release the syzbot report, and let you folks work on a fix, thanks.
Ah, the problem is with
pppol2tp_connectracing with itself. Thanks for the syzbot report. I will take a look. I live for debugging deadlocks :-)
Hi Jakub, any updates on this issue ? (Other syzbot reports with the same root cause are piling up)
Thanks
Quoted 56 lines · Eric Dumazet
Quoted 55 lines · Jakub Sitnicki
Quoted 52 lines · Eric Dumazet
Quoted 39 lines · Eric Dumazet
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54facI guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lockalways needs_bhsafety.I will send something like:
net/l2tp/l2tp_core.c +0
-0100644
net/l2tp/l2tp_core.c int l2tp_tunnel_register(struct l2tp_tunnel · line 1478
Quoted 6 lines
net/l2tp/l2tp_core.c +1
-1@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, } sk = sock->sk; -write_lock (&sk->sk_callback_lock); +write_lock_bh (&sk->sk_callback_lock);Unfortunately this might still not work, because
setup_udp_tunnel_sock->udp_encap_enable()probably could sleep instatic_branch_inc()?I will release the syzbot report, and let you folks work on a fix, thanks.
Quoted 8 lines
Quoted 3 lines
Ah, the problem is with
pppol2tp_connectracing with itself. Thanks for the syzbot report. I will take a look. I live for debugging deadlocks :-)Hi Jakub, any updates on this issue ? (Other syzbot reports with the same root cause are piling up)
Thanks
Sorry, I don't have anything yet. I have reserved time to work on it this afternoon (I'm in the CET timezone).
Alternatively, I can send a revert right away and come back with fixed patch once I have that, if you prefer.
Quoted 6 lines · Jakub Sitnicki
Sorry, I don't have anything yet. I have reserved time to work on it this afternoon (I'm in the CET timezone).
Alternatively, I can send a revert right away and come back with fixed patch once I have that, if you prefer.
No worries, this can wait for a fix, thanks.
Quoted 10 lines · Eric Dumazet
Quoted 6 lines · Jakub Sitnicki
Sorry, I don't have anything yet. I have reserved time to work on it this afternoon (I'm in the CET timezone).
Alternatively, I can send a revert right away and come back with fixed patch once I have that, if you prefer.
No worries, this can wait for a fix, thanks.
Proposed fix now posted:
Quoted 66 lines · Eric Dumazet
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54facI guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lockalways needs_bhsafety.I will send something like:
net/l2tp/l2tp_core.c +0
-0100644
net/l2tp/l2tp_core.c +1
-1@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, } sk = sock->sk; -write_lock (&sk->sk_callback_lock); +write_lock_bh (&sk->sk_callback_lock); ret =l2tp_validate_socket (sk, net, tunnel->encap);if (ret < 0)net/l2tp/l2tp_core.c +1
-1@@ -1522,7 +1522,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, if (tunnel->fd >= 0)sockfd_put (sock); -write_unlock (&sk->sk_callback_lock); +write_unlock_bh (&sk->sk_callback_lock);return 0 ;
err_sock:net/l2tp/l2tp_core.c +1
-1@@ -1531,7 +1531,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, elsesockfd_put (sock); -write_unlock (&sk->sk_callback_lock); +write_unlock_bh (&sk->sk_callback_lock); err:return ret;
Quoted 1 line
}
Hmm, weird. I double checked - I have PROVE_LOCKING enabled. Didn't see any lockdep reports when running selftests/net/l2tp.sh.
I my defense - I thought _bh was not needed because l2tp_tunnel_register() gets called only in the process context. I mean, it's triggered by Netlink sendmsg, but that gets processed in-line AFAIU:
netlink_sendmsg
netlink_unicast
->netlink_rcv
genl_rcv
genl_rcv_msg
genl_family_rcv_msg
genl_family_rcv_msg_doit
->doit
l2tp_nl_cmd_tunnel_create
l2tp_tunnel_registerQuoted 80 lines · Jakub Sitnicki
Quoted 66 lines · Eric Dumazet
Quoted 17 lines · On Wed, Nov 16, 2022 at 5:30 AM <[email protected]> wrote:
Hello:
This patch was applied to netdev/net.git (master)
by David S. Miller <[email protected]>:Quoted 10 lines · On Mon, 14 Nov 2022 20:16:19 +0100 you wrote:
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.Quoted 7 lines
Quoted 1 line
[...]
Here is the summary with links:
- [net,v4] l2tp: Serialize access tosk_user_datawithsk_callback_lockhttps://git.kernel.org/netdev/net/c/b68777d54facI guess this patch has not been tested with LOCKDEP, right ?
sk_callback_lockalways needs_bhsafety.I will send something like:
net/l2tp/l2tp_core.c +0
-0100644
net/l2tp/l2tp_core.c +1
-1@@ -1474,7 +1474,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, } sk = sock->sk; -write_lock (&sk->sk_callback_lock); +write_lock_bh (&sk->sk_callback_lock); ret =l2tp_validate_socket (sk, net, tunnel->encap);if (ret < 0)net/l2tp/l2tp_core.c +1
-1@@ -1522,7 +1522,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, if (tunnel->fd >= 0)sockfd_put (sock); -write_unlock (&sk->sk_callback_lock); +write_unlock_bh (&sk->sk_callback_lock);return 0 ;
err_sock:net/l2tp/l2tp_core.c +1
-1@@ -1531,7 +1531,7 @@ int l2tp_tunnel_register(struct l2tp_tunnel *tunnel, struct net *net, elsesockfd_put (sock); -write_unlock (&sk->sk_callback_lock); +write_unlock_bh (&sk->sk_callback_lock); err:return ret;Quoted 1 line
}
Hmm, weird. I double checked - I have
PROVE_LOCKINGenabled. Didn't see any lockdep reports when runningselftests/net/l2tp.sh.I my defense - I thought
_bhwas not needed becausel2tp_tunnel_register()gets called only in the process context. I mean, it's triggered by Netlink sendmsg, but that gets processed in-line AFAIU:
netlink_sendmsg
netlink_unicast
Quoted 8 lines
->netlink_rcv
genl_rcv
genl_rcv_msg
genl_family_rcv_msg
genl_family_rcv_msg_doit
->doit
l2tp_nl_cmd_tunnel_create
l2tp_tunnel_registerThree different syzbot reports will help to better understand the issue, sorry it is 2am for me, I am not sure in which time zone you are in ...
Quoted 51 lines · Jakub Sitnicki
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.v4:
- serialize write tosk_user_datain l2tpsk_destructv3:
- switch from sock lock tosk_callback_lock
- document write-protection forsk_user_datav2:
- update Fixes to point to origin of the bug
- use real names in Reported/Tested-by tagsCc Tom Parkin <[email protected]>Fixes 3557baabf280 ("[L2TP]: PPP over L2TP driver core")Reported-by Haowei Yan <[email protected]>Signed-off-by Jakub Sitnicki <[email protected]>---
This took me forever. Sorry about that.
include/net/sock.h | 2 +- net/l2tp/l2tp_core.c | 19 +++++++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-)include/net/sock.h +1
-1@@ -323,7 +323,7 @@ struct sk_filter;* @sk_tskey: counter to disambiguate concurrent tstamp requests * @sk_zckey: counter to order MSG_ZEROCOPY notifications * @sk_socket: Identd and reporting IO signals -* @sk_user_data: RPC layer private data +* @sk_user_data: RPC layer private data. Write-protected by @sk_callback_lock. * @sk_frag: cached page frag * @sk_peek_off: current peek_offset value * @sk_send_head: front of stuff to transmit net/l2tp/l2tp_core.c +0
-0
net/l2tp/l2tp_core.c static void l2tp_tunnel_destruct(struct sock *sk) · line 1159
Quoted 10 lines
net/l2tp/l2tp_core.c +2
-0@@ -1150,8 +1150,10 @@ static void l2tp_tunnel_destruct(struct sock *sk) }/* Remove hooks into tunnel socket */ +write_lock_bh (&sk->sk_callback_lock); sk->sk_destruct = tunnel->old_sk_destruct; sk->sk_user_data =NULL ; +write_unlock_bh (&sk->sk_callback_lock);/* Call the original destructor */ if (sk->sk_destruct)
Hi Jakub,
I have a similar issue with vxlan driver. Similar with commit ad6c9986bcb6 ("vxlan: Fix GRO cells race condition between receive and link delete"). There is still a race condition on vxlan that when receive a packet while deleting a VXLAN device. In vxlan_ecn_decapsulate(), the vxlan_get_sk_family() call panic as sk is NULL.
#0 [ffffa25ec6978a38] machine_kexec at ffffffff8c669757 #1 [ffffa25ec6978a90] __crash_kexec at ffffffff8c7c0a4d #2 [ffffa25ec6978b58] crash_kexec at ffffffff8c7c1c48 #3 [ffffa25ec6978b60] oops_end at ffffffff8c627f2b #4 [ffffa25ec6978b80] page_fault_oops at ffffffff8c678fcb #5 [ffffa25ec6978bd8] exc_page_fault at ffffffff8d109542 #6 [ffffa25ec6978c00] asm_exc_page_fault at ffffffff8d200b62
[exception RIP: vxlan_ecn_decapsulate+0x3b]
RIP: ffffffffc1014e7b RSP: ffffa25ec6978cb0 RFLAGS: 00010246
RAX: 0000000000000008 RBX: ffff8aa000888000 RCX: 0000000000000000
RDX: 000000000000000e RSI: ffff8a9fc7ab803e RDI: ffff8a9fd1168700
RBP: ffff8a9fc7ab803e R8: 0000000000700000 R9: 00000000000010ae
R10: ffff8a9fcb748980 R11: 0000000000000000 R12: ffff8a9fd1168700
R13: ffff8aa000888000 R14: 00000000002a0000 R15: 00000000000010ae
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #7 [ffffa25ec6978ce8] vxlan_rcv at ffffffffc10189cd [vxlan] #8 [ffffa25ec6978d90] udp_queue_rcv_one_skb at ffffffff8cfb6507 #9 [ffffa25ec6978dc0] udp_unicast_rcv_skb at ffffffff8cfb6e45
#10 [ffffa25ec6978dc8] __udp4_lib_rcv at ffffffff8cfb8807 #11 [ffffa25ec6978e20] ip_protocol_deliver_rcu at ffffffff8cf76951 #12 [ffffa25ec6978e48] ip_local_deliver at ffffffff8cf76bde #13 [ffffa25ec6978ea0] __netif_receive_skb_one_core at ffffffff8cecde9b #14 [ffffa25ec6978ec8] process_backlog at ffffffff8cece139 #15 [ffffa25ec6978f00] __napi_poll at ffffffff8ceced1a
#16 [ffffa25ec6978f28] net_rx_action at ffffffff8cecf1f3
#17 [ffffa25ec6978fa0] __softirqentry_text_start at ffffffff8d4000ca #18 [ffffa25ec6978ff0] do_softirq at ffffffff8c6fbdc3
--- <IRQ stack> ---
Quoted 1 line
struct socket
ffff8a9fd1168700
struct socket {
state = SS_FREE,
type = 0,
flags = 0,
file = 0xffff8a9fcb748000,
sk = 0x0,
ops = 0x0,
So I'm wondering if we should also have locks in udp_tunnel_sock_release(). Or should we add a checking in sk state before calling vxlan_get_sk_family()?
Thanks
Hangbin
Quoted 109 lines · Hangbin Liu
Quoted 51 lines · Jakub Sitnicki
sk->sk_user_datahas multiple users, which are not compatible with each other. Writers must synchronize by grabbing thesk->sk_callback_lock.l2tp currently fails to grab the lock when modifying the underlying tunnel socket fields. Fix it by adding appropriate locking.
We err on the side of safety and grab the
sk_callback_lockalso inside thesk_destructcallback overridden by l2tp, even though there should be no refs allowing access to the sock at the time whensk_destructgets called.v4:
- serialize write tosk_user_datain l2tpsk_destructv3:
- switch from sock lock tosk_callback_lock
- document write-protection forsk_user_datav2:
- update Fixes to point to origin of the bug
- use real names in Reported/Tested-by tagsCc Tom Parkin <[email protected]>Fixes 3557baabf280 ("[L2TP]: PPP over L2TP driver core")Reported-by Haowei Yan <[email protected]>Signed-off-by Jakub Sitnicki <[email protected]>---
This took me forever. Sorry about that.
include/net/sock.h | 2 +- net/l2tp/l2tp_core.c | 19 +++++++++++++------ 2 files changed, 14 insertions(+), 7 deletions(-)include/net/sock.h +1
-1@@ -323,7 +323,7 @@ struct sk_filter;* @sk_tskey: counter to disambiguate concurrent tstamp requests * @sk_zckey: counter to order MSG_ZEROCOPY notifications * @sk_socket: Identd and reporting IO signals -* @sk_user_data: RPC layer private data +* @sk_user_data: RPC layer private data. Write-protected by @sk_callback_lock. * @sk_frag: cached page frag * @sk_peek_off: current peek_offset value * @sk_send_head: front of stuff to transmit net/l2tp/l2tp_core.c +0
-0net/l2tp/l2tp_core.c static void l2tp_tunnel_destruct(struct sock *sk) · line 1159
Quoted 10 lines
net/l2tp/l2tp_core.c +2
-0@@ -1150,8 +1150,10 @@ static void l2tp_tunnel_destruct(struct sock *sk) }/* Remove hooks into tunnel socket */ +write_lock_bh (&sk->sk_callback_lock); sk->sk_destruct = tunnel->old_sk_destruct; sk->sk_user_data =NULL ; +write_unlock_bh (&sk->sk_callback_lock);/* Call the original destructor */ if (sk->sk_destruct)Hi Jakub,
I have a similar issue with vxlan driver. Similar with commit
ad6c9986bcb6("vxlan: Fix GRO cells race condition between receive and link delete"). There is still a race condition on vxlan that when receive a packet while deleting a VXLAN device. Invxlan_ecn_decapsulate(), thevxlan_get_sk_family()call panic as sk is NULL.#0 [
ffffa25ec6978a38]machine_kexecatffffffff8c669757#1 [ffffa25ec6978a90]__crash_kexecatffffffff8c7c0a4d#2 [ffffa25ec6978b58]crash_kexecatffffffff8c7c1c48#3 [ffffa25ec6978b60]oops_endatffffffff8c627f2b#4 [ffffa25ec6978b80]page_fault_oopsatffffffff8c678fcb#5 [ffffa25ec6978bd8]exc_page_faultatffffffff8d109542#6 [ffffa25ec6978c00]asm_exc_page_faultatffffffff8d200b62[exception RIP: vxlan_ecn_decapsulate+0x3b] RIP: ffffffffc1014e7b RSP: ffffa25ec6978cb0 RFLAGS: 00010246 RAX: 0000000000000008 RBX: ffff8aa000888000 RCX: 0000000000000000 RDX: 000000000000000e RSI: ffff8a9fc7ab803e RDI: ffff8a9fd1168700 RBP: ffff8a9fc7ab803e R8: 0000000000700000 R9: 00000000000010ae R10: ffff8a9fcb748980 R11: 0000000000000000 R12: ffff8a9fd1168700 R13: ffff8aa000888000 R14: 00000000002a0000 R15: 00000000000010ae ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#7 [
ffffa25ec6978ce8]vxlan_rcvatffffffffc10189cd[vxlan] #8 [ffffa25ec6978d90]udp_queue_rcv_one_skbatffffffff8cfb6507#9 [ffffa25ec6978dc0]udp_unicast_rcv_skbatffffffff8cfb6e45
#10 [ffffa25ec6978dc8]__udp4_lib_rcvatffffffff8cfb8807#11 [ffffa25ec6978e20]ip_protocol_deliver_rcuatffffffff8cf76951#12 [ffffa25ec6978e48]ip_local_deliveratffffffff8cf76bde#13 [ffffa25ec6978ea0]__netif_receive_skb_one_coreatffffffff8cecde9b#14 [ffffa25ec6978ec8]process_backlogatffffffff8cece139#15 [ffffa25ec6978f00]__napi_pollatffffffff8ceced1a
#16 [ffffa25ec6978f28]net_rx_actionatffffffff8cecf1f3
#17 [ffffa25ec6978fa0]__softirqentry_text_startatffffffff8d4000ca#18 [ffffa25ec6978ff0]do_softirqatffffffff8c6fbdc3
--- <IRQ stack> ---Quoted 1 line
struct socket
ffff8a9fd1168700struct socket {
state =SS_FREE,
type = 0,
flags = 0,
file =0xffff8a9fcb748000,
sk =0x0,
ops =0x0,
Quoted 2 lines
So I'm wondering if we should also have locks in
udp_tunnel_sock_release(). Or should we add a checking in sk state before callingvxlan_get_sk_family()?
This is how like to think about it:
To know when it is safe to load vs->sock->sk->sk_family, we have to ask:
1. What ensures that the objects remain alive/valid in our scope?
2. What protects the objects from being mutated?
In case of vxlan_sock object in the context of vxlan_ecn_decapsulate():
1. We are in an RCU read side section (ip_local_deliver_finish).
2. RCU-protected objects are not to be mutated while readers exist.
The classic "What is RCU, Fundamentally?" article explains it much better than I ever could:
https://lwn.net/Articles/262464/
As to where the problem lies. I belive udp_tunnel_sock_release() is not keeping the (2) promise.
After unpublishing the sk_user_data, we should wait for any existing readers accessing the vxlan_sock to finish with synchronize_rcu(), before releaseing the socket.
That is:
net/ipv4/udp_tunnel_core.c +1 -0
@@ -176,6 +176,7 @@ EXPORT_SYMBOL_GPL(udp_tunnel_xmit_skb);void udp_tunnel_sock_release (struct socket *sock) {rcu_assign_sk_user_data (sock->sk,NULL ); +synchronize_rcu ();kernel_sock_shutdown (sock,SHUT_RDWR );sock_release (sock); }
Otherwise accessing vxlan_sock state doesn't look safe to me.
Quoted 49 lines · Jakub Sitnicki
Quoted 10 lines
Hi Jakub,
I have a similar issue with vxlan driver. Similar with commit
ad6c9986bcb6("vxlan: Fix GRO cells race condition between receive and link delete"). There is still a race condition on vxlan that when receive a packet while deleting a VXLAN device. Invxlan_ecn_decapsulate(), thevxlan_get_sk_family()call panic as sk is NULL.So I'm wondering if we should also have locks in
udp_tunnel_sock_release(). Or should we add a checking in sk state before callingvxlan_get_sk_family()?This is how like to think about it:
To know when it is safe to load
vs->sock->sk->sk_family, we have to ask:1. What ensures that the objects remain alive/valid in our scope?
2. What protects the objects from being mutated?In case of
vxlan_sockobject in the context ofvxlan_ecn_decapsulate():1. We are in an RCU read side section (
ip_local_deliver_finish).
2. RCU-protected objects are not to be mutated while readers exist.The classic "What is RCU, Fundamentally?" article explains it much better than I ever could:
https://lwn.net/Articles/262464/
As to where the problem lies. I belive
udp_tunnel_sock_release()is not keeping the (2) promise.After unpublishing the
sk_user_data, we should wait for any existing readers accessing thevxlan_sockto finish withsynchronize_rcu(), before releaseing the socket.That is:
net/ipv4/udp_tunnel_core.c +1
-0@@ -176,6 +176,7 @@ EXPORT_SYMBOL_GPL(udp_tunnel_xmit_skb);void udp_tunnel_sock_release (struct socket *sock) {rcu_assign_sk_user_data (sock->sk,NULL ); +synchronize_rcu ();kernel_sock_shutdown (sock,SHUT_RDWR );sock_release (sock); }
Quoted 1 line
Otherwise accessing
vxlan_sockstate doesn't look safe to me.
Hi Jakub,
Thanks for your explain. As it's a little on my side, I will read your comments and try your suggestion tomorrow. Currently, I use the following draft patch to fix the vxlan issue.
drivers/net/vxlan/vxlan_core.c +1 -0
@@ -4234,6 +4234,7 @@ static void vxlan_dellink(struct net_device *dev, struct list_head *head)struct vxlan_dev *vxlan =netdev_priv (dev);vxlan_flush (vxlan, true); +vxlan_sock_release (vxlan);list_del (&vxlan->next);unregister_netdevice_queue (dev, head);
Cheers
Hangbin