Skip to contentSkip to conversation

My system stops during startup with curretn git tree of 2.6.25-rc2

DISCUSSION Zdenek Kabelac 6 participants 13 messages Feb 17, 2008 – Feb 19, 2008 Networking

lore
Share & tools
lore
#0 lore raw

Hi

It looks like there is something weird as my systems stops when the swap is mounted.
I've played bisection game and this is the commit which makes the system unusable:

# bad: [45b503548210fe6f23e92b856421c2a3f05fd034] [RTNETLINK]: Send a single notification on device state changes.
git-bisect bad 45b503548210fe6f23e92b856421c2a3f05fd034

I've tried to reverse this commit - and it has compiled & worked.

Zdenek

#1 lore raw
Quoted 7 lines · Zdenek Kabelac

It looks like there is something weird as my systems stops when the swap is mounted. I've played bisection game and this is the commit which makes the system unusable:
# bad: [45b503548210fe6f23e92b856421c2a3f05fd034] [RTNETLINK]: Send a single notification on device state changes.
git-bisect bad 45b503548210fe6f23e92b856421c2a3f05fd034
I've tried to reverse this commit - and it has compiled & worked.

This commit is completely broken (it, for example, breaks locking around dev->link_mode), as has been already mentioned by Rafael at
http://lkml.org/lkml/2008/2/15/542

Dave, do you have a proper fix queued? Otherwise I would propose just to revert it completely from Linus' tree for now.

Signature · 1 line
Jiri Kosina
maintainer
#2 lore raw

From: Jiri Kosina <[email protected]>
Date: Sun, 17 Feb 2008 13:04:59 +0100 (CET)

Quoted 10 lines · Zdenek Kabelac
Quoted 8 lines

It looks like there is something weird as my systems stops when the swap is mounted. I've played bisection game and this is the commit which makes the system unusable:
# bad: [45b503548210fe6f23e92b856421c2a3f05fd034] [RTNETLINK]: Send a single notification on device state changes.
git-bisect bad 45b503548210fe6f23e92b856421c2a3f05fd034
I've tried to reverse this commit - and it has compiled & worked.

Quoted 6 lines

This commit is completely broken (it, for example, breaks locking around dev->link_mode), as has been already mentioned by Rafael at
http://lkml.org/lkml/2008/2/15/542

Dave, do you have a proper fix queued? Otherwise I would propose just to revert it completely from Linus' tree for now.

I just reverted and I'll push that to Linus.

#3 lore raw
Quoted 10 lines · David Miller

From: Jiri Kosina <[email protected]>
Date: Sun, 17 Feb 2008 13:04:59 +0100 (CET)

Quoted 7 lines · Zdenek Kabelac
Quoted 6 lines

It looks like there is something weird as my systems stops when the swap is mounted. I've played bisection game and this is the commit which makes the system unusable:
# bad: [45b503548210fe6f23e92b856421c2a3f05fd034] [RTNETLINK]: Send a
single notification on device state changes.
git-bisect bad 45b503548210fe6f23e92b856421c2a3f05fd034
I've tried to reverse this commit - and it has compiled & worked.

Quoted 6 lines
Quoted 3 lines

This commit is completely broken (it, for example, breaks locking around dev->link_mode), as has been already mentioned by Rafael at http://lkml.org/lkml/2008/2/15/542

Dave, do you have a proper fix queued? Otherwise I would propose just to revert it completely from Linus' tree for now.

I just reverted and I'll push that to Linus.

Okay, but I can't figure out what's the problem with it. I don't have wireless card on my linux box also I can't test it but everything else works. Swap is mounted.

The concurrency cannot be a problem because the write operation is protected by a lock.

My only idea is the notification itself, because the netdev_state_change function is used which calls rtmsg_ifinfo too.

void netdev_state_change(struct net_device *dev)

{
        if (dev->flags & IFF_UP) {
                call_netdevice_notifiers(NETDEV_CHANGE, dev);
                rtmsg_ifinfo(RTM_NEWLINK, dev, 0);
        }
}

Also last lines of my patch could be instead of

        if (modified)
                netdev_state_change(dev);

the following:
if (modified && dev->flags & IFF_UP) {
call_netdevice_notifiers(NETDEV_CHANGE, dev);
}

Regards,
Attila

#4 lore raw
Quoted 4 lines · Laszlo Attila Toth

Okay, but I can't figure out what's the problem with it. I don't have wireless card on my linux box also I can't test it but everything else works. Swap is mounted. The concurrency cannot be a problem because the write operation is protected by a lock.

-               write_lock_bh(&dev_base_lock);
-               dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
-               write_unlock_bh(&dev_base_lock);
+               if (dev->link_mode != nla_get_u8(tb[IFLA_LINKMODE])) {
+                       write_lock_bh(&dev_base_lock);
+                       dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
+                       write_lock_bh(&dev_base_lock);
+                       modified = 1;
+               }
        }

1) you are accessing dev->link_mode and tb[] outside the dev_base_lock
2) there is obvious and immediate deadlock -- you acquire the dev_base_lock twice, without any unlock, just look at the chunk above
3) even with this deadlock fixed, Rafael states that either NM or wpa_supplicant (I don't recall from top of my head) still don't work

Signature · 2 lines
Jiri Kosina
SUSE Labs
#5 lore raw
Quoted 15 lines · Jiri Kosina
Quoted 1 line · Laszlo Attila Toth

Okay, but I can't figure out what's the problem with it. I don't have wireless card on my linux box also I can't test it but everything else works. Swap is mounted. The concurrency cannot be a problem because the write operation is protected by a lock.

(quoted diff) +6 -3
-               write_lock_bh(&dev_base_lock);
-               dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
-               write_unlock_bh(&dev_base_lock);
+               if (dev->link_mode != nla_get_u8(tb[IFLA_LINKMODE])) {
+                       write_lock_bh(&dev_base_lock);
+                       dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
+                       write_lock_bh(&dev_base_lock);
+                       modified = 1;
+               }
        }
Quoted 1 line

1) you are accessing dev->link_mode and tb[] outside the dev_base_lock

yes, because tb[IFLA_LINKMODE] is not used by someone else in this case only dev->link_mode. Although its value is unpredictable in case of a concurrent access in the condition, it does not affect the final value of dev->link_mode but the length of the critical section remains minimal. The if statement may be inside the lock.

Quoted 2 lines

2) there is obvious and immediate deadlock -- you acquire the dev_base_lock twice, without any unlock, just look at the chunk above

Indeed:
"Feb 16 16:51:49 sandman kernel: BUG: rwlock recursion on CPU#0,"

I missed it. I copied the code from another patch which didn't contain the two locking statements and when I copied them back it became a copy-paste bug.

Quoted 2 lines

3) even with this deadlock fixed, Rafael states that either NM or wpa_supplicant (I don't recall from top of my head) still don't work

That's bad. Does my suggestion solve the problem? Again:

- if (modified)
- netdev_state_change(dev);
+ if (modified && dev->flags & IFF_UP)
+ call_netdevice_notifiers(NETDEV_CHANGE, dev)

Regards,
Attila

#6 lore raw
Quoted 43 lines · Laszlo Attila Toth
Quoted 18 lines · Jiri Kosina
Quoted 4 lines · Laszlo Attila Toth

Okay, but I can't figure out what's the problem with it. I don't have wireless card on my linux box also I can't test it but everything else works. Swap is mounted. The concurrency cannot be a problem because the write operation is protected by a lock.

(quoted diff) +6 -3
-               write_lock_bh(&dev_base_lock);
-               dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
-               write_unlock_bh(&dev_base_lock);
+               if (dev->link_mode != nla_get_u8(tb[IFLA_LINKMODE])) {
+                       write_lock_bh(&dev_base_lock);
+                       dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
+                       write_lock_bh(&dev_base_lock);
+                       modified = 1;
+               }
        }
Quoted 1 line

1) you are accessing dev->link_mode and tb[] outside the dev_base_lock

yes, because tb[IFLA_LINKMODE] is not used by someone else in this case only dev->link_mode. Although its value is unpredictable in case of a concurrent access in the condition, it does not affect the final value of dev->link_mode but the length of the critical section remains minimal. The if statement may be inside the lock.

Quoted 2 lines

2) there is obvious and immediate deadlock -- you acquire the dev_base_lock twice, without any unlock, just look at the chunk above

Indeed:
"Feb 16 16:51:49 sandman kernel: BUG: rwlock recursion on CPU#0,"

I missed it. I copied the code from another patch which didn't contain the two locking statements and when I copied them back it became a copy-paste bug.

Quoted 2 lines

3) even with this deadlock fixed, Rafael states that either NM or wpa_supplicant (I don't recall from top of my head) still don't work

That's bad. Does my suggestion solve the problem? Again:

Review

Quoted 4 lines
(quoted diff) +2 -2
-      if (modified)
-                 netdev_state_change(dev);
+      if (modified && dev->flags & IFF_UP)
+                 call_netdevice_notifiers(NETDEV_CHANGE, dev)

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Thanks,
Rafael

---
Fix net/core/rtnetlink.c breakage caused by commit 45b503548210fe6f23e92b856421c2a3f05fd034
"[RTNETLINK]: Send a single notification on device state changes."

---
 net/core/rtnetlink.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
net/core/rtnetlink.c +3 -3
Index: linux-2.6/net/core/rtnetlink.c
===================================================================
@@ -853,7 +853,7 @@ static int do_setlink(struct net_device 
 		if (dev->link_mode != nla_get_u8(tb[IFLA_LINKMODE])) {
 			write_lock_bh(&dev_base_lock);
 			dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
-			write_lock_bh(&dev_base_lock);
+			write_unlock_bh(&dev_base_lock);
 			modified = 1;
 		}
 	}
@@ -870,8 +870,8 @@ errout:
 	if (send_addr_notify)
 		call_netdevice_notifiers(NETDEV_CHANGEADDR, dev);
 
-	if (modified)
-		netdev_state_change(dev);
+	if (modified && dev->flags & IFF_UP)
+		call_netdevice_notifiers(NETDEV_CHANGE, dev);
 
 	return err;
 }
#7 lore raw

Hello,

Quoted 6 lines · Rafael J. Wysocki

On Monday, 18 of February 2008, Laszlo Attila Toth wrote:

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Thanks for the testing. Dave already reverted the patch, also I don't know I should resend it or leave it as is.

Regards,
Attila

Quoted 38 lines

Thanks,
Rafael

---
Fix net/core/rtnetlink.c breakage caused by commit 45b503548210fe6f23e92b856421c2a3f05fd034
"[RTNETLINK]: Send a single notification on device state changes."

---
 net/core/rtnetlink.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Index: linux-2.6/net/core/rtnetlink.c
===================================================================

net/core/rtnetlink.c +3 -3
@@ -853,7 +853,7 @@ static int do_setlink(struct net_device 
 		if (dev->link_mode != nla_get_u8(tb[IFLA_LINKMODE])) {
 			write_lock_bh(&dev_base_lock);
 			dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
-			write_lock_bh(&dev_base_lock);
+			write_unlock_bh(&dev_base_lock);
 			modified = 1;
 		}
 	}
@@ -870,8 +870,8 @@ errout:
 	if (send_addr_notify)
 		call_netdevice_notifiers(NETDEV_CHANGEADDR, dev);
 
-	if (modified)
-		netdev_state_change(dev);
+	if (modified && dev->flags & IFF_UP)
+		call_netdevice_notifiers(NETDEV_CHANGE, dev);
 
 	return err;
 }
#8 lore raw
Quoted 6 lines · Laszlo Attila Toth

Hello,

Quoted 3 lines · Rafael J. Wysocki

On Monday, 18 of February 2008, Laszlo Attila Toth wrote:

Quoted 6 lines
Quoted 3 lines

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Thanks for the testing. Dave already reverted the patch, also I don't know I should resend it or leave it as is.

Since the patch has been reverted, I think you can fold the fix into it and send it again for the next merge window (unless it's _really_ urgent). Of course, if that's fine by Dave. :-)

Thanks,
Rafael

maintainer
#9 lore raw
In reply to Laszlo Attila Toth

From: Laszlo Attila Toth <[email protected]>
Date: Mon, 18 Feb 2008 18:03:47 +0100

Quoted 6 lines

Hello,

Quoted 3 lines · Rafael J. Wysocki

On Monday, 18 of February 2008, Laszlo Attila Toth wrote:

Quoted 6 lines
Quoted 3 lines

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Thanks for the testing. Dave already reverted the patch, also I don't know I should resend it or leave it as is.

After this sits and gets testing and feedback for a few days you'll need to resubmit the entire original patch with these fixes added on top.

For now, it's staying reverted.

#10 lore raw
Quoted 13 lines · David Miller

From: Laszlo Attila Toth <[email protected]>
Date: Mon, 18 Feb 2008 18:03:47 +0100

Quoted 10 lines

Hello,

Quoted 6 lines · Rafael J. Wysocki

On Monday, 18 of February 2008, Laszlo Attila Toth wrote:

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Quoted 8 lines
Quoted 2 lines

Thanks for the testing. Dave already reverted the patch, also I don't know I should resend it or leave it as is.

After this sits and gets testing and feedback for a few days you'll need to resubmit the entire original patch with these fixes added on top.

For now, it's staying reverted.

Can we please get the revert pushed out please? It renders my laptop unusable and I keep forgetting to revert that pesky change when testing new kernels.

Signature · 1 line
Jens Axboe
maintainer
#11 lore raw

From: Jens Axboe <[email protected]>
Date: Tue, 19 Feb 2008 11:54:19 +0100

Quoted 21 lines · David Miller
Quoted 20 lines

From: Laszlo Attila Toth <[email protected]>
Date: Mon, 18 Feb 2008 18:03:47 +0100

Quoted 6 lines

Hello,

Quoted 3 lines · Rafael J. Wysocki

On Monday, 18 of February 2008, Laszlo Attila Toth wrote:

Quoted 6 lines
Quoted 3 lines

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Thanks for the testing. Dave already reverted the patch, also I don't know I should resend it or leave it as is.

After this sits and gets testing and feedback for a few days you'll need to resubmit the entire original patch with these fixes added on top.

Quoted 5 lines
Quoted 1 line

For now, it's staying reverted.

Can we please get the revert pushed out please? It renders my laptop unusable and I keep forgetting to revert that pesky change when testing new kernels.

I sent a push to Linus an hour or so ago, he should pick it up in the morning.

#12 lore raw
Quoted 24 lines · David Miller

From: Jens Axboe <[email protected]>
Date: Tue, 19 Feb 2008 11:54:19 +0100

Quoted 21 lines · David Miller
Quoted 20 lines

From: Laszlo Attila Toth <[email protected]>
Date: Mon, 18 Feb 2008 18:03:47 +0100

Quoted 6 lines

Hello,

Quoted 3 lines · Rafael J. Wysocki

On Monday, 18 of February 2008, Laszlo Attila Toth wrote:

Quoted 6 lines
Quoted 3 lines

All in all, I gather you wanted me to test the patch below. :-)

Yes, that helps.

Thanks for the testing. Dave already reverted the patch, also I don't know I should resend it or leave it as is.

After this sits and gets testing and feedback for a few days you'll need to resubmit the entire original patch with these fixes added on top.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

For now, it's staying reverted.

Can we please get the revert pushed out please? It renders my laptop unusable and I keep forgetting to revert that pesky change when testing new kernels.

I sent a push to Linus an hour or so ago, he should pick it up in the morning.

Great, thanks!

Signature · 1 line
Jens Axboe