Skip to contentSkip to conversation

New: Problems with hibernation, with bisect

Bug 65671 [email protected] 1 participant 20 messages Nov 24, 2013 – Sep 9, 2014

lore
Share & tools
lore

[email protected]

author Acked-by Viresh Kumar
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Bug ID: 65671
Summary: Problems with hibernation, with bisect
Product: Power Management
Version: 2.5
Kernel Version: 3.12
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: cpufreq
Assignee: [email protected]
Reporter: [email protected]
Regression: No

After upgrading the kernel to 3.12, i found that hibernation stop working. Most of the time it gives IO errors writing the swap on hibernation, and when it manage to do that, it crash all apps after the wakeup and remount the filesystem as read-only

As 3.11.8 hibernation worked fine, i have done a bisect and found that the problem show up in this commit:

commit 4a511de96d692f2dfa126c10dda4e41636c0ef27

Author: Mark Brown <[email protected]>
Date:   Tue Aug 13 14:58:24 2013 +0200

cpufreq: cpufreq-cpu0: NULL is a valid regulator

Since NULL could in theory be a valid regulator we ought to check for IS_ERR() rather than for NULL. In practice this is unlikely to be an issue but it's better for neatness.

:040000 040000 3e8d03d2dc170492ba9f0311ef9156344452f6de
fb5d9ad67f332dec478a68cbd9387b15381584ce M drivers

Kernel 3.12.1 still have the problem

i'm using slackware64 14.1 with gcc 4.8.2

Thanks

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#1 lore raw

[Bug 65671] Problems with hibernation, with bisect

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Rafael J. Wysocki <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |NEEDINFO
                 CC|                            |[email protected]

--- Comment #1 from Rafael J. Wysocki <[email protected]> --- If you revert that commit from 3.12.1, does the problem go away?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#3 lore raw

[Bug 65671] Problems with hibernation, with bisect

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #2 from higuita <[email protected]> ---
As asked, i tried to revert the commit on 3.12.1 and test and the crashes after the wakeup are still there. So looks like that commit is not the only one that broke this.

Any tip how to debug/bisect this now?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#4 lore raw

[Bug 65671] Problems with hibernation, with bisect

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #3 from Rafael J. Wysocki <[email protected]> ---

Quoted 8 lines · you

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #2 from higuita <[email protected]> ---
As asked, i tried to revert the commit on 3.12.1 and test and the crashes after the wakeup are still there. So looks like that commit is not the only one that broke this.

Any tip how to debug/bisect this now?

Can you check bug #64841, please?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#5 lore raw

[Bug 65671] Problems with hibernation, with bisect

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #4 from higuita <[email protected]> ---
I don't have a intel card, i have a old ATI HD2600 AGP on a old Asus A8V MB. Here is the lspci:

00:00.0 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge 00:00.1 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge 00:00.2 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge 00:00.3 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge 00:00.4 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge 00:00.7 Host bridge: VIA Technologies, Inc. K8T800Pro Host Bridge 00:01.0 PCI bridge: VIA Technologies, Inc. VT8237/8251 PCI bridge [K8M890/K8T800/K8T890 South]
00:07.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev 80)
00:08.0 RAID bus controller: Promise Technology, Inc. PDC20378 (FastTrak 378/SATA 378) (rev 02)
00:0a.0 Ethernet controller: Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller (rev 13)
00:0e.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
00:0e.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev
11)
00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06) 00:10.0 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.1 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.2 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.3 USB controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
00:10.4 USB controller: VIA Technologies, Inc. USB 2.0 (rev 86) 00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South]
00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
00:11.6 Communication controller: VIA Technologies, Inc. AC'97 Modem Controller (rev 80)
00:18.0 Host bridge: AMD [Advanced Micro Devices, Inc.] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: AMD [Advanced Micro Devices, Inc.] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: AMD [Advanced Micro Devices, Inc.] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: AMD [Advanced Micro Devices, Inc.] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: AMD/ATI [Advanced Micro Devices, Inc.] RV630 XT [Radeon HD 2600 XT AGP]

But that bug asks this:
"I primarily meant outside of gfx domain. It would be likely to see fs corruption for example. Depending on your system, it might be unlikely to see this corruption without using various memory debugging tools. My fear is we've traded silent memory corruption for hangs."

in the past, i had some FS corruption after several hibernations cycles... could it be that i have this kind of silent memory corruption problem and that patch triggers the problem when in the past it was hidden?

when i wakeup the machine, i get this

[ 5137.948087] Uhhuh. NMI received for unknown reason 2c on CPU 0.
[ 5137.950056] Do you have a strange power saving mode enabled?
[ 5137.995447] Dazed and confused, but trying to continue

Could this corrupt some part of the memory?

I recently run the memtest86+ for several hours and didn't got any problem. Also, if i don't hibernate the machine, it works fine for several months, without any problem.

Is any tool to do some sort of memory checksum before and after the hibernation, to try to catch any (bad) change?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#6 lore raw

[Bug 65671] Problems with hibernation, with bisect

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #5 from higuita <[email protected]> ---
ok, more info.

I have spend half of the weekend to try to debug this with kernel 2.12.1 and this is what i found:

in the bios if i disable both the "V-Link 8X Support" and "AGP Fast write", things get more stable and i only have problems if i hibernate with the machine with load (actually, i'm using the glxgears)

If i disable the "ACPI APIC", i have no problem at all, the machine is stable during many hibernations cycles, even on heavy load... saddly, i lost the second CPU without APIC

So i think this points to something related to the AGP bus.

With the APIC turn on and the "Vlink" and the "AGP fast write" turn on, i get the problem even on the console, without the X11 loaded, so maybe the something used by the modeset.

I will try to disable it and do some more test.

Again, i only have problems with hibernations, if i dont hibernate, everything looks stable during several weeks/months

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#7 lore raw

[Bug 65671] Problems with hibernation, with bisect

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #6 from higuita <[email protected]> ---
with modeset disable, it still crashes after hibernating (but only on the second attempt to hibernate)

I also attempt to switch the mode from "platform" to "shutdown", but still crashes

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#8 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Len Brown <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
            Summary|Problems with hibernation,  |hibernation - IO errors
                   |with bisect                 |writing to swap, crashing
                   |                            |apps, 3,11.8 worked

--- Comment #7 from Len Brown <[email protected]> ---
can you disable CONFIG_CPU_FREQ in your build of the latest kernel to see if that helps?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#9 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #8 from higuita <[email protected]> ---
BINGO!

ok, just recompile 3.12.3 without CONFIG_CPU_FREQ i can hibernate without any problem, even under heavy load for several hibernate-resume cycles.

Even re-enable the AGP fast write and the VLINK 8X the hibernation works fine

Using the same kernel, WITH the cpufreq enable it fails to resume usually on the first try under heavy load.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#10 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #9 from higuita <[email protected]> ---
Another recompile and tests and i found that the problem is in the "conservative" governor. Using the "ondemand" governor i found no problem for several hibernate-resume cycles... when i switch to the conservative governor and try to hibernate for the second time, i get IO/swap problems.

I usually use the conservative governor, so that is why i'm getting this problem.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#11 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #10 from higuita <[email protected]> ---
i've been using the ondemand governor and not problem at all. So it's really something that is broken in the conservative governor... maybe not all changes in the ondemand weren't applied to the conservative one?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#12 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #11 from higuita <[email protected]> ---
This problem can be much older than kernel 3.12... i've been having sporadic problems with random corrupted filesystems since maybe about june. During all summer i assumed it was a bad HD (that started to report bad sectors and is mostly unused, it's actually umounted and suspended on boot) was introducing problems in other SATA channels and corrupting other HD partitions.

Since i switch to the performance governor, i had no FS corruption, so looks like the (unused) HD was not the problem, but again some bug in the ondemand governor. If true, this could be a lot older and recent changes made it much more frequent and reproducible.

Sadly, the old FS corruption was hard to trigger, so it's not easy to point when it started to show up

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#13 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #12 from higuita <[email protected]> --- Ok, i have a little more information:

i found that ondemand still causes problems and after the upgrade to 3.13.3, it also lock up, just like the conservative governor. I then change my hibernate script to include this before the hibernation

cpufreq-set -g performance
rmmod powernow_k8

and after the hibernation i revert that:

modprobe powernow_k8
cpufreq-set -g ondemand

All seems to be working fine, i'm getting higher uptimes, many hibernations cycles without problems and zero Filesystem corruption.

i have seen this on the dmesg, after waking from hibernation:

[ 8379.324747] [Firmware Bug]: powernow-k8: No compatible ACPI _PSS objects
found.
[ 8379.324747] [Firmware Bug]: powernow-k8: First, make sure Cool'N'Quiet is
enabled in the BIOS.
[ 8379.324747] [Firmware Bug]: powernow-k8: If that doesn't help, try upgrading
your BIOS.
[ 8379.324858] powernow-k8: Found 1 Dual Core AMD Opteron(tm) Processor 175 (2

cpu cores) (version 2.20.00)

This motherboard (asus A8V deluxe) don't have ACPI cpufreq, but the cool'n'quiet is enabled in the bios.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#14 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked, unless rmmod powernow_k8

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Len Brown <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

            Summary|hibernation - IO errors     |hibernation - IO errors
                   |writing to swap, crashing   |writing to swap, crashing
                   |apps, 3,11.8 worked         |apps, 3,11.8 worked, unless
                   |                            |rmmod powernow_k8

--- Comment #13 from Len Brown <[email protected]> ---

Quoted 1 line

No compatible ACPI _PSS objects found.

Whelp, at least we know it isn't an ACPI bug:-)

This seems to be specific to powernow_k8 -- is it possible to bisect but only changes within that driver?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#16 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked, unless rmmod powernow_k8

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Rafael J. Wysocki <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Regression|Yes |No

--- Comment #15 from Rafael J. Wysocki <[email protected]> --- I'm clearing the regression flag on this anyway, since it is not clear whether or not it is a regression.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#17 lore raw

[Bug 65671] hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked, unless rmmod powernow_k8

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

--- Comment #16 from higuita <[email protected]> ---
i compiled kernel 3.15-rc4, disabled my workarounds and tried to trigger the hibernation problems with more than 15 hibernations cycles (under various loads) and failed.

i also had tried something similar on kernel 4.14.1 and also didn't had any problem.

So or the problem was fixed, or the hidden corruption change it target. I still do have (maybe once a month) some random ext4 corruption, but even that have improved.

maybe because the hibernations tests, after those tests i rebooted and got this ext4 error a few minutes after, on the /home partition:

[  367.200228] EXT4-fs error (device sda8): ext4_mb_generate_buddy:756: group
513, 3097 clusters in bitmap, 3081 in gd; block bitmap corrupt.
[  367.218897] JBD2: Spotted dirty metadata buffer (dev = sda8, blocknr = 0).
There's a risk of filesystem corruption in case of system crash.
[  367.285925] JBD2: Spotted dirty metadata buffer (dev = sda8, blocknr = 0).

There's a risk of filesystem corruption in case of system crash.

I will try to grab the fsck output to check if it helps.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#18 lore raw

[Bug 65671] Asus A8V deluxe(AMD): hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked, unless rmmod powernow_k8

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Lan Tianyu <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

            Summary|hibernation - IO errors     |Asus A8V deluxe(AMD):
                   |writing to swap, crashing   |hibernation - IO errors
                   |apps, 3,11.8 worked, unless |writing to swap, crashing
                   |rmmod powernow_k8           |apps, 3,11.8 worked, unless
                   |                            |rmmod powernow_k8
Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#19 lore raw

[Bug 65671] Asus A8V deluxe(AMD): hibernation - IO errors writing to swap, crashing apps, 3,11.8 worked, unless rmmod powernow_k8

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=65671

Zhang Rui <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
           Hardware|All                         |x86-64
Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.