Skip to contentSkip to conversation

bridge:Add rcu read lock when delete br port

PATCH In review [email protected] 3 participants 6 messages Aug 4, 2014 – Aug 5, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bridge:Add rcu read lock when delete br port

From: Chunhe Li <[email protected]>

In the br_hanle_frame function has a bug, when the bridge receive packets which go througth the br_handle_frame, get the net_bridge_port pointer "p", but don't check NULL pointer to use it. If somebody delete the bridge port at the same time, will call a NULL pointer, trigger kernel panic. I see the del_nbp comments, call del_nbp should via RCU, but the caller don't do this.

following steps will make bug happened
1.start vm and add the vm interface to a bridge br0,for example, brctl addbr br0 tap0

2.configuer vm interface and br0 same ip subnet, vm ping br0.

3.add and delete the vm interface port for endless loop.

---
 net/bridge/br_if.c | 4 ++++
 1 file changed, 4 insertions(+)
net/bridge/br_if.c +4 -0
index 3eca3fd..91c611d 100644
@@ -274,9 +274,11 @@ void br_dev_delete(struct net_device *dev, struct list_head *head)
 	struct net_bridge *br = netdev_priv(dev);
 	struct net_bridge_port *p, *n;
 
+	rcu_read_lock();
 	list_for_each_entry_safe(p, n, &br->port_list, list) {
 		del_nbp(p);
 	}
+	rcu_read_unlock();
 
 	br_fdb_delete_by_port(br, NULL, 1);
 
@@ -550,7 +552,9 @@ int br_del_if(struct net_bridge *br, struct net_device *dev)
 	 * there still maybe an alternate path for netconsole to use;
 	 * therefore there is no reason for a NETDEV_RELEASE event.
 	 */
+	rcu_read_lock();
 	del_nbp(p);
+	rcu_read_unlock();
 
 	spin_lock_bh(&br->lock);
 	changed_addr = br_stp_recalculate_bridge_id(br);
Signature · 1 line
1.9.2.0
#1 lore raw
Quoted 7 lines · On Mon, Aug 04, 2014 at 11:37:56AM +0800, [email protected] wrote:

From: Chunhe Li <[email protected]>

In the br_hanle_frame function has a bug, when the bridge receive packets which go througth the br_handle_frame, get the net_bridge_port pointer "p", but don't check NULL pointer to use it. If somebody delete the bridge port at the same time, will call a NULL pointer, trigger kernel panic. I see the del_nbp comments, call del_nbp should via RCU, but the caller don't do this.

I don't see such a comment there.

Are you talking about this line:

        p = br_port_get_rcu(skb->dev);

this is actually rx_handler_data.
The reason it should not be NULL is
explained here:

void netdev_rx_handler_unregister(struct net_device *dev) {

        ASSERT_RTNL();
        RCU_INIT_POINTER(dev->rx_handler, NULL);
        /* a reader seeing a non NULL rx_handler in a rcu_read_lock()
         * section has a guarantee to see a non NULL rx_handler_data
         * as well.
         */
        synchronize_net();
        RCU_INIT_POINTER(dev->rx_handler_data, NULL);
}

EXPORT_SYMBOL_GPL(netdev_rx_handler_unregister);

Quoted 9 lines

following steps will make bug happened
1.start vm and add the vm interface to a bridge br0,for example, brctl addbr br0 tap0

2.configuer vm interface and br0 same ip subnet, vm ping br0.

3.add and delete the vm interface port for endless loop.

OK but apparently something else triggered the bug here. It might be a good idea to enable lockdep and rcu checks see if anything suspicious is reported.

Quoted 21 lines
---
 net/bridge/br_if.c | 4 ++++
 1 file changed, 4 insertions(+)
net/bridge/br_if.c +2 -0
index 3eca3fd..91c611d 100644
@@ -274,9 +274,11 @@ void br_dev_delete(struct net_device *dev, struct list_head *head)
 	struct net_bridge *br = netdev_priv(dev);
 	struct net_bridge_port *p, *n;
 
+	rcu_read_lock();
 	list_for_each_entry_safe(p, n, &br->port_list, list) {
 		del_nbp(p);
 	}
+	rcu_read_unlock();
 
 	br_fdb_delete_by_port(br, NULL, 1);
 

net/bridge/br_if.c int br_del_if(struct net_bridge *br, struct net_device *dev) · line 560

Quoted 9 lines
net/bridge/br_if.c +2 -0
@@ -550,7 +552,9 @@ int br_del_if(struct net_bridge *br, struct net_device *dev)
 	 * there still maybe an alternate path for netconsole to use;
 	 * therefore there is no reason for a NETDEV_RELEASE event.
 	 */
+	rcu_read_lock();
 	del_nbp(p);
+	rcu_read_unlock();
 
 	spin_lock_bh(&br->lock);
 	changed_addr = br_stp_recalculate_bridge_id(br);

Does the problem disappear with this applied?
I don't see how this would help. rcu locks do not synchronize against other readers.

Quoted 3 lines
Signature · 1 line
1.9.2.0
author
#2 lore raw
Quoted 18 lines

-----Original Message-----
From: Michael S. Tsirkin [mailto:[email protected]]
Sent: Monday, August 04, 2014 8:41 PM
To: Lichunhe
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Wuyunfei; Qianhuibin (Huibin QIAN, Euler)
Subject: Re: [PATCH net/next] bridge:Add rcu read lock when delete br port

Quoted 7 lines · On Mon, Aug 04, 2014 at 11:37:56AM +0800, [email protected] wrote:

From: Chunhe Li <[email protected]>

In the br_hanle_frame function has a bug, when the bridge receive packets which go througth the br_handle_frame, get the net_bridge_port pointer "p", but don't check NULL pointer to use it. If somebody delete the bridge port at the same time, will call a NULL pointer, trigger kernel panic. I see the del_nbp comments, call del_nbp should via RCU,

Quoted 7 lines

but the caller don't do this.

I don't see such a comment there.

Are you talking about this line:

       p = br_port_get_rcu(skb->dev);

Yes, this var "p" is NULL when the bug happened.

Quoted 62 lines

this is actually rx_handler_data.
The reason it should not be NULL is
explained here:

void netdev_rx_handler_unregister(struct net_device *dev) {

       ASSERT_RTNL();
       RCU_INIT_POINTER(dev->rx_handler, NULL);
       /* a reader seeing a non NULL rx_handler in a rcu_read_lock()
        * section has a guarantee to see a non NULL rx_handler_data
        * as well.
        */
       synchronize_net();
       RCU_INIT_POINTER(dev->rx_handler_data, NULL); }

EXPORT_SYMBOL_GPL(netdev_rx_handler_unregister);

Quoted 8 lines

following steps will make bug happened 1.start vm and add the vm interface to a bridge br0,for example, brctl addbr br0 tap0

2.configuer vm interface and br0 same ip subnet, vm ping br0.

3.add and delete the vm interface port for endless loop.

OK but apparently something else triggered the bug here.
It might be a good idea to enable lockdep and rcu checks see if anything suspicious is reported.

net/bridge/br_if.c void br_dev_delete(struct net_device *dev, struct

Quoted 9 lines
---
 net/bridge/br_if.c | 4 ++++
 1 file changed, 4 insertions(+)
net/bridge/br_if.c index renamed from net/bridge/br_if.c +0 -0

3eca3fd..91c611d 100644

net/bridge/br_if.c +0 -0
@@ -274,9 +274,11 @@ void br_dev_delete(struct net_device *dev, struct

list_head *head)

net/bridge/br_if.c int br_del_if(struct net_bridge *br, struct net_device

Quoted 12 lines
net/bridge/br_if.c +2 -0
 	struct net_bridge *br = netdev_priv(dev);
 	struct net_bridge_port *p, *n;
 
+	rcu_read_lock();
 	list_for_each_entry_safe(p, n, &br->port_list, list) {
 		del_nbp(p);
 	}
+	rcu_read_unlock();
 
 	br_fdb_delete_by_port(br, NULL, 1);
 
@@ -550,7 +552,9 @@ int br_del_if(struct net_bridge *br, struct net_device

*dev)

Quoted 7 lines
net/bridge/br_if.c +2 -0
 	 * there still maybe an alternate path for netconsole to use;
 	 * therefore there is no reason for a NETDEV_RELEASE event.
 	 */
+	rcu_read_lock();
 	del_nbp(p);
+	rcu_read_unlock();
 
Quoted 9 lines

net/bridge/br_if.c int br_del_if(struct net_bridge *br, struct net_device *dev) · line 560

Quoted 2 lines
net/bridge/br_if.c +0 -0
 	spin_lock_bh(&br->lock);
 	changed_addr = br_stp_recalculate_bridge_id(br);

Does the problem disappear with this applied?
I don't see how this would help. rcu locks do not synchronize against other readers.

Maybe I understand wrong, please ignore this patch, do you have better way to solve this problem?

Quoted 3 lines
Quoted 3 lines

--
1.9.2.0

#3 lore raw
Quoted 100 lines · Lichunhe
Quoted 18 lines

-----Original Message-----
From: Michael S. Tsirkin [mailto:[email protected]]
Sent: Monday, August 04, 2014 8:41 PM
To: Lichunhe
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Wuyunfei; Qianhuibin (Huibin QIAN, Euler)
Subject: Re: [PATCH net/next] bridge:Add rcu read lock when delete br port

Quoted 7 lines · On Mon, Aug 04, 2014 at 11:37:56AM +0800, [email protected] wrote:

From: Chunhe Li <[email protected]>

In the br_hanle_frame function has a bug, when the bridge receive packets which go througth the br_handle_frame, get the net_bridge_port pointer "p", but don't check NULL pointer to use it. If somebody delete the bridge port at the same time, will call a NULL pointer, trigger kernel panic. I see the del_nbp comments, call del_nbp should via RCU,

Quoted 7 lines

but the caller don't do this.

I don't see such a comment there.

Are you talking about this line:

       p = br_port_get_rcu(skb->dev);

Yes, this var "p" is NULL when the bug happened.

Quoted 62 lines

this is actually rx_handler_data.
The reason it should not be NULL is
explained here:

void netdev_rx_handler_unregister(struct net_device *dev) {

       ASSERT_RTNL();
       RCU_INIT_POINTER(dev->rx_handler, NULL);
       /* a reader seeing a non NULL rx_handler in a rcu_read_lock()
        * section has a guarantee to see a non NULL rx_handler_data
        * as well.
        */
       synchronize_net();
       RCU_INIT_POINTER(dev->rx_handler_data, NULL); }

EXPORT_SYMBOL_GPL(netdev_rx_handler_unregister);

Quoted 8 lines

following steps will make bug happened 1.start vm and add the vm interface to a bridge br0,for example, brctl addbr br0 tap0

2.configuer vm interface and br0 same ip subnet, vm ping br0.

3.add and delete the vm interface port for endless loop.

OK but apparently something else triggered the bug here.
It might be a good idea to enable lockdep and rcu checks see if anything suspicious is reported.

net/bridge/br_if.c void br_dev_delete(struct net_device *dev, struct

Quoted 9 lines
---
 net/bridge/br_if.c | 4 ++++
 1 file changed, 4 insertions(+)
net/bridge/br_if.c index renamed from net/bridge/br_if.c +0 -0

3eca3fd..91c611d 100644

net/bridge/br_if.c +0 -0
@@ -274,9 +274,11 @@ void br_dev_delete(struct net_device *dev, struct

list_head *head)

net/bridge/br_if.c int br_del_if(struct net_bridge *br, struct net_device

Quoted 12 lines
net/bridge/br_if.c +2 -0
 	struct net_bridge *br = netdev_priv(dev);
 	struct net_bridge_port *p, *n;
 
+	rcu_read_lock();
 	list_for_each_entry_safe(p, n, &br->port_list, list) {
 		del_nbp(p);
 	}
+	rcu_read_unlock();
 
 	br_fdb_delete_by_port(br, NULL, 1);
 
@@ -550,7 +552,9 @@ int br_del_if(struct net_bridge *br, struct net_device

*dev)

Quoted 7 lines
net/bridge/br_if.c +2 -0
 	 * there still maybe an alternate path for netconsole to use;
 	 * therefore there is no reason for a NETDEV_RELEASE event.
 	 */
+	rcu_read_lock();
 	del_nbp(p);
+	rcu_read_unlock();
 
Quoted 9 lines

net/bridge/br_if.c int br_del_if(struct net_bridge *br, struct net_device *dev) · line 560

Quoted 2 lines
net/bridge/br_if.c +0 -0
 	spin_lock_bh(&br->lock);
 	changed_addr = br_stp_recalculate_bridge_id(br);

Does the problem disappear with this applied?
I don't see how this would help. rcu locks do not synchronize against other readers.

Quoted 1 line

Maybe I understand wrong, please ignore this patch, do you have better way to solve this problem?

As I wrote above, need to understand the problem first, we can't fix it otherwise.

It seems possible that what you are seeing is use after free or an RCU error. Enable as many debugging options in kernel build (under kernel hacking) as you can and see if anything comes up in kernel log.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

--
1.9.2.0

#4 lore raw
In reply to [email protected]
Quoted 12 lines · On Mon, 4 Aug 2014 11:37:56 +0800 <[email protected]> wrote:

From: Chunhe Li <[email protected]>

In the br_hanle_frame function has a bug, when the bridge receive packets which go througth the br_handle_frame, get the net_bridge_port pointer "p", but don't check NULL pointer to use it. If somebody delete the bridge port at the same time, will call a NULL pointer, trigger kernel panic. I see the del_nbp comments, call del_nbp should via RCU, but the caller don't do this.

following steps will make bug happened
1.start vm and add the vm interface to a bridge br0,for example, brctl addbr br0 tap0

Quoted 5 lines

2.configuer vm interface and br0 same ip subnet, vm ping br0.

3.add and delete the vm interface port for endless loop.

This is safe because:
1. br_dev_delete is called with RTNL mutex therefore it is safe with other add/delete RCU does not block anything
2. br_handle_frame is called with RCU lock (in frame rx)
3. port is not freed until after grace period

#5 lore raw
In reply to [email protected]
Quoted 12 lines · On Mon, 4 Aug 2014 11:37:56 +0800 <[email protected]> wrote:

From: Chunhe Li <[email protected]>

In the br_hanle_frame function has a bug, when the bridge receive packets which go througth the br_handle_frame, get the net_bridge_port pointer "p", but don't check NULL pointer to use it. If somebody delete the bridge port at the same time, will call a NULL pointer, trigger kernel panic. I see the del_nbp comments, call del_nbp should via RCU, but the caller don't do this.

following steps will make bug happened
1.start vm and add the vm interface to a bridge br0,for example, brctl addbr br0 tap0

Quoted 5 lines

2.configuer vm interface and br0 same ip subnet, vm ping br0.

3.add and delete the vm interface port for endless loop.

Any possible bridge issues should go [email protected]

Nobody reads LKML anymore.