Skip to contentSkip to conversation

regression: fireware causes oops during system

DISCUSSION Pavel Machek 4 participants 12 messages Sep 11, 2007 – Oct 5, 2007

lore
Share & tools
lore
#0 lore raw

Hi!

I noticed empty suspend stopped working around 2.6.23-rc4, and it is still present in 2.6.23-rc6. To reproduce

swapoff -a
echo disk > /sys/power/state
echo disk > /sys/power/state

Unsetting

CONFIG_IEEE1394=y

solves the problem.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#1 lore raw
Quoted 9 lines · Pavel Machek

Hi!

I noticed empty suspend stopped working around 2.6.23-rc4, and it is still present in 2.6.23-rc6. To reproduce

swapoff -a
echo disk > /sys/power/state
echo disk > /sys/power/state

Quoted 5 lines

Unsetting

CONFIG_IEEE1394=y

solves the problem.

Since this is easy to reproduce, can you bisect it down?

Signature · 3 lines
Ubuntu   : http://www.ubuntu.com/
Linux1394: http://wiki.linux1394.org/
SwissDisk: http://www.swissdisk.com/
#2 lore raw
Quoted 8 lines · Ben Collins
Quoted 7 lines · Pavel Machek

I noticed empty suspend stopped working around 2.6.23-rc4, and it is still present in 2.6.23-rc6. To reproduce

swapoff -a
echo disk > /sys/power/state
echo disk > /sys/power/state

Quoted 7 lines
Quoted 5 lines

Unsetting

CONFIG_IEEE1394=y

solves the problem.

Since this is easy to reproduce, can you bisect it down?

I will check in a minute what I pushed out after 2.6.23-rc1, may save Pavel some time.

Signature · 3 lines
Stefan Richter
-=====-=-=== =--= -=-==
http://arcgraph.de/sr/
#3 lore raw
Quoted 3 lines
Quoted 3 lines · Pavel Machek
Quoted 2 lines

I noticed empty suspend stopped working around 2.6.23-rc4, and it is still present in 2.6.23-rc6.

...

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Unsetting

CONFIG_IEEE1394=y

solves the problem.

...

Between -rc3 and -rc4:

ieee1394: sbp2: fix sbp2_remove_device for error cases a2ee3f9bbb0ce57102dad8928d54f59acdc4b8f7 should not occur in suspend path

Between -rc1 and -rc2:

	ieee1394: sbp2: more correct Kconfig dependencies
	e4f8cac5e07528f7e0bc21d3682c16c9de993ecb
	unrelated

	ieee1394: revert "sbp2: enforce 32bit DMA mapping"
	a9c2f18800753c82c45fc13b27bdc148849bdbb2
	unrelated

	(not via linux1394-2.6.git)
	raw1394 __user annotation
	5b26e64ea39e45802c5736c8261bf8a8704d212f
	unrelated

So it must be something older which was somehow uncovered. Dmesg with the oops or/and bisection would be good.

Signature · 3 lines
Stefan Richter
-=====-=-=== =--= -=-==
http://arcgraph.de/sr/
#4 lore raw

Hi!

Quoted 32 lines
Quoted 3 lines
Quoted 3 lines · Pavel Machek
Quoted 2 lines

I noticed empty suspend stopped working around 2.6.23-rc4, and it is still present in 2.6.23-rc6.

...

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Unsetting

CONFIG_IEEE1394=y

solves the problem.

...

Between -rc3 and -rc4:

ieee1394: sbp2: fix sbp2_remove_device for error cases a2ee3f9bbb0ce57102dad8928d54f59acdc4b8f7 should not occur in suspend path

Between -rc1 and -rc2:

	ieee1394: sbp2: more correct Kconfig dependencies
	e4f8cac5e07528f7e0bc21d3682c16c9de993ecb
	unrelated

	ieee1394: revert "sbp2: enforce 32bit DMA mapping"
	a9c2f18800753c82c45fc13b27bdc148849bdbb2
	unrelated

	(not via linux1394-2.6.git)
	raw1394 __user annotation
	5b26e64ea39e45802c5736c8261bf8a8704d212f
	unrelated
Quoted 2 lines

So it must be something older which was somehow uncovered. Dmesg with the oops or/and bisection would be good.

Sorry, had to hand-copy. It is oops at virtual adddress 6b6b6b7b -- looks like slab poison to me?

EIP is in task_rq_lock, backtrace is
try_to_wake_up
highlevel_host_reset
ohci_irq_handler
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw

(Adding Cc: Rafael, Ingo)

Quoted 9 lines · Pavel Machek
Quoted 1 line

Dmesg with the oops or/and bisection would be good.

Sorry, had to hand-copy. It is oops at virtual adddress 6b6b6b7b -- looks like slab poison to me?

EIP is in task_rq_lock, backtrace is
try_to_wake_up
highlevel_host_reset
ohci_irq_handler

In reverse order, this trace is most certainly

	drivers/ieee1394/ohci1394.c::ohci_irq_handler
	drivers/ieee1394/highlevel.c::highlevel_host_reset
	drivers/ieee1394/highlevel.c::nodemgr_highlevel.host_reset
						== nodemgr_host_reset
	kernel/sched.c::wake_up_process(hi->thread);

with hi->thread being the kthread which executes drivers/ieee1394/nodemgr.c::nodemgr_host_thread of the ieee1394 core driver.

The ieee1394 core has two (or more) threads: One nodemgr_host_thread alias [knodemgrd_*] for each card, and one hpsbpkt_thread alias [khpsbpkt] for all cards. The knodemgrd should be frozen during suspend or hibernate, while khpsbpkt should not be frozen in order to let transactions to go on in the case of saving the hibernation image to a FireWire disk.

Quoting your other post:

Quoted 9 lines · Stefan Richter
Quoted 5 lines

Between -rc3 and -rc4:

ieee1394: sbp2: fix sbp2_remove_device for error cases a2ee3f9bbb0ce57102dad8928d54f59acdc4b8f7 should not occur in suspend path

Plus I do not have firewire attached disk here, so sbp2 should not be used, right?

Sbp2's host_reset handler would do something if it was loaded even without any SBP-2 devices attached,but it wouldn't under any circumstances call try_to_wake_up. Actually with no devices attached it would simply "iterate" over an empty list.

Quoted 40 lines
Quoted 15 lines

Between -rc1 and -rc2:

	ieee1394: sbp2: more correct Kconfig dependencies
	e4f8cac5e07528f7e0bc21d3682c16c9de993ecb
	unrelated

	ieee1394: revert "sbp2: enforce 32bit DMA mapping"
	a9c2f18800753c82c45fc13b27bdc148849bdbb2
	unrelated

	(not via linux1394-2.6.git)
	raw1394 __user annotation
	5b26e64ea39e45802c5736c8261bf8a8704d212f
	unrelated
Quoted 2 lines

So it must be something older which was somehow uncovered. Dmesg with the oops or/and bisection would be good.

The others look even more innocent. I wonder if this may have been responsible?

commit 831441862956fffa17b9801db37e6ea1650b0f69
tree b0334921341f8f1734bdd3243de76d676329d21c
parent 787d2214c19bcc9b6ac48af0ce098277a801eded
author Rafael J. Wysocki <[email protected]> Tue, 17 Jul 2007 04:03:35 -0700 committer Linus Torvalds <[email protected]> Tue, 17 Jul 2007 10:23:02 -0700

Freezer: make kernel threads nonfreezable by default

Or this one?

commit 20c2df83d25c6a95affe6157a4c9cac4cf5ffaac
tree 415c4453d2b17a50abe7a3e515177e1fa337bd67
parent 64fb98fc40738ae1a98bcea9ca3145b89fb71524
author Paul Mundt <[email protected]> Fri, 20 Jul 2007 10:11:58 +0900
committer Paul Mundt <[email protected]> Fri, 20 Jul 2007 10:11:58 +0900

Quoted 6 lines

mm: Remove slab destructors from kmem_cache_create().

Slab destructors were no longer supported after Christoph's c59def9f222d44bb7e2f0a559f2906191a0862d7 change. They've been BUGs for both slab and slub, and slob never supported them either.

Both of these went in during the merge window between 2.6.22 and 2.6.23-rc1.

The bug could have been introduced by a change outside of the ieee1394 subsystem, like freezer or scheduler. The try_to_wake_up in your backtrace would be directed towards a thread which should be frozen at some point.

Signature · 3 lines
Stefan Richter
-=====-=-=== =--= =----
http://arcgraph.de/sr/
#6 lore raw
Quoted 40 lines · Stefan Richter

(Adding Cc: Rafael, Ingo)

Quoted 9 lines · Pavel Machek
Quoted 1 line

Dmesg with the oops or/and bisection would be good.

Sorry, had to hand-copy. It is oops at virtual adddress 6b6b6b7b -- looks like slab poison to me?

EIP is in task_rq_lock, backtrace is
try_to_wake_up
highlevel_host_reset
ohci_irq_handler

In reverse order, this trace is most certainly

	drivers/ieee1394/ohci1394.c::ohci_irq_handler
	drivers/ieee1394/highlevel.c::highlevel_host_reset
	drivers/ieee1394/highlevel.c::nodemgr_highlevel.host_reset
						== nodemgr_host_reset
	kernel/sched.c::wake_up_process(hi->thread);

with hi->thread being the kthread which executes drivers/ieee1394/nodemgr.c::nodemgr_host_thread of the ieee1394 core driver.

The ieee1394 core has two (or more) threads: One nodemgr_host_thread alias [knodemgrd_*] for each card, and one hpsbpkt_thread alias [khpsbpkt] for all cards. The knodemgrd should be frozen during suspend or hibernate, while khpsbpkt should not be frozen in order to let transactions to go on in the case of saving the hibernation image to a FireWire disk.

Quoting your other post:

Quoted 7 lines · Stefan Richter
Quoted 5 lines

Between -rc3 and -rc4:

ieee1394: sbp2: fix sbp2_remove_device for error cases a2ee3f9bbb0ce57102dad8928d54f59acdc4b8f7 should not occur in suspend path

Quoted 37 lines
Quoted 2 lines

Plus I do not have firewire attached disk here, so sbp2 should not be used, right?

Sbp2's host_reset handler would do something if it was loaded even without any SBP-2 devices attached,but it wouldn't under any circumstances call try_to_wake_up. Actually with no devices attached it would simply "iterate" over an empty list.

Quoted 29 lines
Quoted 15 lines

Between -rc1 and -rc2:

	ieee1394: sbp2: more correct Kconfig dependencies
	e4f8cac5e07528f7e0bc21d3682c16c9de993ecb
	unrelated

	ieee1394: revert "sbp2: enforce 32bit DMA mapping"
	a9c2f18800753c82c45fc13b27bdc148849bdbb2
	unrelated

	(not via linux1394-2.6.git)
	raw1394 __user annotation
	5b26e64ea39e45802c5736c8261bf8a8704d212f
	unrelated
Quoted 2 lines

So it must be something older which was somehow uncovered. Dmesg with the oops or/and bisection would be good.

The others look even more innocent. I wonder if this may have been responsible?

commit 831441862956fffa17b9801db37e6ea1650b0f69
tree b0334921341f8f1734bdd3243de76d676329d21c
parent 787d2214c19bcc9b6ac48af0ce098277a801eded
author Rafael J. Wysocki <[email protected]> Tue, 17 Jul 2007 04:03:35 -0700 committer Linus Torvalds <[email protected]> Tue, 17 Jul 2007 10:23:02 -0700

Freezer: make kernel threads nonfreezable by default

Well, I don't think so.

nodemgr_host_thread() calls set_freezable() as it should.

Greetings,
Rafael

#7 lore raw
Quoted 8 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 7 lines · Pavel Machek

commit 831441862956fffa17b9801db37e6ea1650b0f69
tree b0334921341f8f1734bdd3243de76d676329d21c
parent 787d2214c19bcc9b6ac48af0ce098277a801eded
author Rafael J. Wysocki <[email protected]> Tue, 17 Jul 2007 04:03:35 -0700 committer Linus Torvalds <[email protected]> Tue, 17 Jul 2007 10:23:02 -0700

Quoted 5 lines
Quoted 1 line
Quoted 1 line

Freezer: make kernel threads nonfreezable by default

Well, I don't think so.

nodemgr_host_thread() calls set_freezable() as it should.

This commit is certainly OK, as it should merely preserve status quo. Also note that Pavel wrote in his initial post that the problem became apparent way after -rc1. Full quote:

| I noticed empty suspend stopped working around 2.6.23-rc4, and it is
| still present in 2.6.23-rc6. To reproduce
|
| swapoff -a
| echo disk > /sys/power/state
| echo disk > /sys/power/state
|
| Unsetting
|
| CONFIG_IEEE1394=y
|
| solves the problem.
Signature · 3 lines
Stefan Richter
-=====-=-=== =--= =----
http://arcgraph.de/sr/
#8 lore raw
Quoted 19 lines · Stefan Richter
Quoted 8 lines · Rafael J. Wysocki
Quoted 8 lines
Quoted 7 lines · Pavel Machek

commit 831441862956fffa17b9801db37e6ea1650b0f69
tree b0334921341f8f1734bdd3243de76d676329d21c
parent 787d2214c19bcc9b6ac48af0ce098277a801eded
author Rafael J. Wysocki <[email protected]> Tue, 17 Jul 2007 04:03:35 -0700 committer Linus Torvalds <[email protected]> Tue, 17 Jul 2007 10:23:02 -0700

Quoted 5 lines
Quoted 1 line
Quoted 1 line

Freezer: make kernel threads nonfreezable by default

Well, I don't think so.

nodemgr_host_thread() calls set_freezable() as it should.

This commit is certainly OK, as it should merely preserve status quo. Also note that Pavel wrote in his initial post that the problem became apparent way after -rc1. Full quote:

Quoted 12 lines
| I noticed empty suspend stopped working around 2.6.23-rc4, and it is
| still present in 2.6.23-rc6. To reproduce
|
| swapoff -a
| echo disk > /sys/power/state
| echo disk > /sys/power/state
|
| Unsetting
|
| CONFIG_IEEE1394=y
|
| solves the problem.

Yes.

I thought that there might be a later change that exposed a bug in it.

Hm. Pavel, can you do

# echo test > /sys/power/disk
# echo disk > /sys/power/state

and see what happens?

Greetings,
Rafael

#9 lore raw

Hi!

Quoted 19 lines
Quoted 4 lines

This commit is certainly OK, as it should merely preserve status quo. Also note that Pavel wrote in his initial post that the problem became apparent way after -rc1. Full quote:

Quoted 12 lines
| I noticed empty suspend stopped working around 2.6.23-rc4, and it is
| still present in 2.6.23-rc6. To reproduce
|
| swapoff -a
| echo disk > /sys/power/state
| echo disk > /sys/power/state
|
| Unsetting
|
| CONFIG_IEEE1394=y
|
| solves the problem.

Yes.

Quoted 8 lines

I thought that there might be a later change that exposed a bug in it.

Hm. Pavel, can you do

# echo test > /sys/power/disk
# echo disk > /sys/power/state

and see what happens?

root@amd:~# echo test  > /sys/power/disk
root@amd:~# echo disk > /sys/power/state
root@amd:~# echo disk > /sys/power/state

Produces nothing interesting... I also did few hibernation/resume cycles, and everything seems to work ok. But when I do swapoff then try to hibernate, it fails on second try. Weird.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#10 lore raw
Quoted 30 lines · Pavel Machek

Hi!

Quoted 19 lines
Quoted 4 lines

This commit is certainly OK, as it should merely preserve status quo. Also note that Pavel wrote in his initial post that the problem became apparent way after -rc1. Full quote:

Quoted 12 lines
| I noticed empty suspend stopped working around 2.6.23-rc4, and it is
| still present in 2.6.23-rc6. To reproduce
|
| swapoff -a
| echo disk > /sys/power/state
| echo disk > /sys/power/state
|
| Unsetting
|
| CONFIG_IEEE1394=y
|
| solves the problem.

Yes.

Quoted 8 lines

I thought that there might be a later change that exposed a bug in it.

Hm. Pavel, can you do

# echo test > /sys/power/disk
# echo disk > /sys/power/state

and see what happens?

Quoted 7 lines
root@amd:~# echo test  > /sys/power/disk
root@amd:~# echo disk > /sys/power/state
root@amd:~# echo disk > /sys/power/state

Produces nothing interesting... I also did few hibernation/resume cycles, and everything seems to work ok. But when I do swapoff then try to hibernate, it fails on second try. Weird.

Weird indeed, but it means that there's some history that causes things to break on the second attempt.

To summarize, after running swapoff the suspending of devices during the second attempt to hibernate fails (100% of the time) unless CONFIG_IEEE1394 is unset?

What happens for CONFIG_IEEE1394=m?

Greetings,
Rafael

#11 lore raw
In reply to Stefan Richter
Quoted 8 lines · Stefan Richter
Quoted 3 lines
Quoted 3 lines · Pavel Machek
Quoted 2 lines

I noticed empty suspend stopped working around 2.6.23-rc4, and it is still present in 2.6.23-rc6.

...

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Unsetting

CONFIG_IEEE1394=y

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

solves the problem.

...

Between -rc3 and -rc4:

ieee1394: sbp2: fix sbp2_remove_device for error cases a2ee3f9bbb0ce57102dad8928d54f59acdc4b8f7 should not occur in suspend path

Plus I do not have firewire attached disk here, so sbp2 should not be used, right?

Quoted 15 lines

Between -rc1 and -rc2:

	ieee1394: sbp2: more correct Kconfig dependencies
	e4f8cac5e07528f7e0bc21d3682c16c9de993ecb
	unrelated

	ieee1394: revert "sbp2: enforce 32bit DMA mapping"
	a9c2f18800753c82c45fc13b27bdc148849bdbb2
	unrelated

	(not via linux1394-2.6.git)
	raw1394 __user annotation
	5b26e64ea39e45802c5736c8261bf8a8704d212f
	unrelated
Quoted 2 lines

So it must be something older which was somehow uncovered. Dmesg with the oops or/and bisection would be good.

The others look even more innocent. I wonder if this may have been responsible?

commit 831441862956fffa17b9801db37e6ea1650b0f69
tree b0334921341f8f1734bdd3243de76d676329d21c
parent 787d2214c19bcc9b6ac48af0ce098277a801eded
author Rafael J. Wysocki <[email protected]> Tue, 17 Jul 2007 04:03:35 -0700 committer Linus Torvalds <[email protected]> Tue, 17 Jul 2007 10:23:02 -0700

Freezer: make kernel threads nonfreezable by default

Or this one?

commit 20c2df83d25c6a95affe6157a4c9cac4cf5ffaac
tree 415c4453d2b17a50abe7a3e515177e1fa337bd67
parent 64fb98fc40738ae1a98bcea9ca3145b89fb71524
author Paul Mundt <[email protected]> Fri, 20 Jul 2007 10:11:58 +0900
committer Paul Mundt <[email protected]> Fri, 20 Jul 2007 10:11:58 +0900

mm: Remove slab destructors from kmem_cache_create().

Slab destructors were no longer supported after Christoph's c59def9f222d44bb7e2f0a559f2906191a0862d7 change. They've been BUGs for both slab and slub, and slob never supported them either.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html