Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

Andrew Morton

author maintainer

2012-05-11 19:59 UTC #1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 34 lines · On Thu, 10 May 2012 23:29:46 +0000 (UTC) [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=43227
Summary: BUG: Bad page state in process wcg_gfam_6.11_i
Product: Memory Management
Version: 2.5
Kernel Version: 3.3.5
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No
Created an attachment (id=73246)
--> (https://bugzilla.kernel.org/attachment.cgi?id=73246)
Entire dmesg
Hi,
I noticed that a number of bad page state bugs have apparently been fixed in recent stable kernels. Unfortunately it seems there's still some such bug left in 3.3.5.
I just saw this bug for the first time after upgrading from (mainline) 3.3.2 to 3.3.5 (having read the other bug reports I suspect this is the significant change) and changing my root filesystem from ext4 to xfs, hence causing heavier exercise of the xfs code than before (but I've had a backup partition on xfs before with no issues).

Quoted 6 lines

This is the bug I'm seeing:

[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)

AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.

It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?

Quoted 27 lines

...

[67031.755872] Pid: 5229, comm: wcg_gfam_6.11_i Tainted: G           O 3.3.5 #2
[67031.755874] Call Trace:
[67031.755880]  [<ffffffff813f6a33>] bad_page+0xcb/0xe0
[67031.755884]  [<ffffffff810dfc4e>] get_page_from_freelist+0x54e/0x5f0
[67031.755888]  [<ffffffff811e0b4e>] ? string.isra.4+0x3e/0xd0
[67031.755891]  [<ffffffff810dfe05>] __alloc_pages_nodemask+0x115/0x810
[67031.755895]  [<ffffffff81143a06>] ? seq_printf+0x56/0x90
[67031.755898]  [<ffffffff8118570a>] ? do_task_stat+0x6ba/0xbf0
[67031.755901]  [<ffffffff8110561b>] ? anon_vma_prepare+0xfb/0x170
[67031.755904]  [<ffffffff810fb0c6>] handle_pte_fault+0x736/0x970
[67031.755906]  [<ffffffff810fb668>] handle_mm_fault+0x1c8/0x2f0
[67031.755909]  [<ffffffff81400cbe>] do_page_fault+0x16e/0x4f0
[67031.755913]  [<ffffffff813fdbc5>] page_fault+0x25/0x30
[67031.755916]  [<ffffffff811e2bfd>] ? copy_user_generic_string+0x2d/0x40
[67031.755919]  [<ffffffff81143e08>] ? seq_read+0x2c8/0x390
[67031.755922]  [<ffffffff811240b4>] vfs_read+0xa4/0x180
[67031.755924]  [<ffffffff811241d5>] sys_read+0x45/0x90
[67031.755927]  [<ffffffff81406a09>] ia32_do_call+0x13/0x13
[67031.755929] Disabling lock debugging due to kernel taint

Before that there's a warning about "irq 17: nobody cared" (see the attached entire dmesg if you are interested); this is apparently an unrelated, known issue with Asus motherboards which has persisted for a long time without causing bad page state bugs.

Quoted 4 lines

FWIW, I noticed that a number of the patches that may have something to do with this and that have gone in since v3.3.2 are KVM related. I have run a large number of KVM guests on this computer between the boot and this bug occuring, so this might (or might not) be KVM related.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sami Liedes

2012-05-11 20:02 UTC #2 lore raw

Quoted 5 lines · Andrew Morton

Quoted 4 lines

[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)

Quoted 7 lines

AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.
It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?

Would it be useful if I tried to reproduce this with some debugging options turned on, for example CONFIG_DEBUG_VM?

Sami

Andrew Morton

author maintainer

2012-05-11 20:32 UTC #3 lore raw

Quoted 9 lines · On Fri, 11 May 2012 23:02:13 +0300 Sami Liedes <[email protected]> wrote:

Quoted 8 lines · Andrew Morton
Quoted 4 lines
[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)
AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.

Quoted 8 lines

Quoted 4 lines
It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?
Would it be useful if I tried to reproduce this with some debugging options turned on, for example CONFIG_DEBUG_VM?

Sure, thanks, that might turn something up.
Documentation/SubmitChecklist recommends

: 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,

:     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
:     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
:     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.

although that list might be a bit out of date; it certainly should include CONFIG_DEBUG_VM!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Jones

2012-05-11 20:59 UTC #4 lore raw

Quoted 24 lines · Andrew Morton

Quoted 9 lines · On Fri, 11 May 2012 23:02:13 +0300 Sami Liedes <[email protected]> wrote:
Quoted 8 lines · Andrew Morton
Quoted 4 lines
[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)
AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.
Quoted 8 lines
Quoted 4 lines
It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?
Would it be useful if I tried to reproduce this with some debugging options turned on, for example CONFIG_DEBUG_VM?
Sure, thanks, that might turn something up.
Documentation/SubmitChecklist recommends

Quoted 7 lines

: 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
:     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
:     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
:     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.
although that list might be a bit out of date; it certainly should include CONFIG_DEBUG_VM!

FWIW, that Fedora report of this had DEBUG_VM enabled, and it doesn't seem that there was any earlier oops/warn judging by the lack of tainting.

Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sami Liedes

2012-05-11 22:23 UTC #5 lore raw

In reply to Andrew Morton

Quoted 10 lines · Andrew Morton

Sure, thanks, that might turn something up.
Documentation/SubmitChecklist recommends
: 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
:     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
:     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
:     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.
although that list might be a bit out of date; it certainly should include CONFIG_DEBUG_VM!

I wonder if there's somewhere a recommended list of generally most useful debug options that only have a moderate performance impact? I'd be happy to use a set of useful debug flags that generally impacts performance by, say, <10%, on the computers I use for my everyday work to help catch bugs. But it's sometimes quite hard to assess the impact of different Kernel hacking options from just the description...

Sami

Dave Jones

2012-05-14 00:26 UTC #6 lore raw

Quoted 12 lines · Sami Liedes

Quoted 10 lines · Andrew Morton
Sure, thanks, that might turn something up.
Documentation/SubmitChecklist recommends
: 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
:     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
:     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
:     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.
although that list might be a bit out of date; it certainly should include CONFIG_DEBUG_VM!

Quoted 6 lines

I wonder if there's somewhere a recommended list of generally most useful debug options that only have a moderate performance impact? I'd be happy to use a set of useful debug flags that generally impacts performance by, say, <10%, on the computers I use for my everyday work to help catch bugs. But it's sometimes quite hard to assess the impact of different Kernel hacking options from just the description...

One problem here is that acceptable performance is entirely subjective, based on your individual workload. For example, I nearly always run the fedora 'debug' builds, which enable near every debug option, and don't find it particularly painful. Other people find those same builds completely intolerable. Especially for example users of shiny new desktops that rely heavily on the drm. Seems that code is particularly miserable to use when spinlock debugging/lockdep are enabled.

A few releases back, something changed which made debug builds even more 'heavyweight' for some users. Still haven't figured out what that might be.

Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Jones

2012-05-11 20:06 UTC #7 lore raw

In reply to Andrew Morton

Quoted 7 lines · Andrew Morton

Quoted 6 lines

https://bugzilla.kernel.org/show_bug.cgi?id=43227

[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)

Quoted 7 lines

AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.
It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?

We got a similar bug reported in Fedora against 3.3.4 yesterday.

https://bugzilla.redhat.com/show_bug.cgi?id=820603

:[162276.196730] BUG: Bad page state in process transmission-gt pfn:1bddff :[162276.196735] page:ffffea0006f77fc0 count:0 mapcount:0 mapping: (null) index:0x7f2fdb962 :[162276.196737] page flags: 0x40000000000014(referenced|dirty)

Different trace, but same result.

Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

author maintainer

2012-05-11 20:10 UTC #8 lore raw

Quoted 19 lines · On Fri, 11 May 2012 16:06:15 -0400 Dave Jones <[email protected]> wrote:

Quoted 7 lines · Andrew Morton
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=43227
[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)
Quoted 7 lines
AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.
It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?
We got a similar bug reported in Fedora against 3.3.4 yesterday.

Quoted 8 lines

https://bugzilla.redhat.com/show_bug.cgi?id=820603
:[162276.196730] BUG: Bad page state in process transmission-gt pfn:1bddff :[162276.196735] page:ffffea0006f77fc0 count:0 mapcount:0 mapping: (null) index:0x7f2fdb962 :[162276.196737] page flags: 0x40000000000014(referenced|dirty)
Different trace, but same result.

That narrows it down to a 3.3.2 -> 3.3.4 regression, perhaps.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sami Liedes

2012-05-11 22:14 UTC #9 lore raw

Quoted 1 line · Andrew Morton

That narrows it down to a 3.3.2 -> 3.3.4 regression, perhaps.

I believe 3.3.2 -> 3.3.3 in that case; I also hit this on 3.3.3 yesterday after downgrading 3.3.5 -> 3.3.3, although in that case I got a large number (total of 36) bad page state warnings within a single second. I have been using 3.3.2 succesfully for quite a while before I first saw this when I upgraded to 3.3.5.

Sami

Hugh Dickins

2012-05-11 21:31 UTC #10 lore raw

In reply to Andrew Morton

Quoted 6 lines · Andrew Morton

Quoted 5 lines

[67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
[67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
(null) index:0x7f1eb293b
[67031.755792] page flags: 0x4000000000000014(referenced|dirty)

Quoted 7 lines

AFAICT we got this warning because the page allocator found a free page with PG_referenced and PG_dirty set.
It would be a heck of a lot more useful if we'd been told about this when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?

Because those flags may validly be set when a page is freed (I do have an old patch to change anon dirty handling to stop that, but it's not really needed). They are then immediately cleared, along with all other page flags. So if page allocation finds any page flags set, it happened while the page was supposedly free.

The only thought I have on this report: what binutils was used to build this kernel? We had "Bad page" and isolate_lru_pages BUG reports at the start of the month, and they were traced to buggy binutils 2.22.52.0.2

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sami Liedes

2012-05-11 22:57 UTC #11 lore raw

Quoted 3 lines · Hugh Dickins

The only thought I have on this report: what binutils was used to build this kernel? We had "Bad page" and isolate_lru_pages BUG reports at the start of the month, and they were traced to buggy binutils 2.22.52.0.2

Debian unstable's binutils 2.22-6.

Sami

Dave Jones

2012-05-14 00:30 UTC #12 lore raw

In reply to Hugh Dickins

Quoted 6 lines · Hugh Dickins

Quoted 3 lines
when the page was freed, not when it was reused! Can anyone think of a reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at least)?
Because those flags may validly be set when a page is freed (I do have an old patch to change anon dirty handling to stop that, but it's not really needed).

It would be nice to be able to distinguish from things like random memory scribbles setting those flags.

Quoted 3 lines

The only thought I have on this report: what binutils was used to build this kernel? We had "Bad page" and isolate_lru_pages BUG reports at the start of the month, and they were traced to buggy binutils 2.22.52.0.2

The fedora reports we have are from binutils prior to that (2.21.53.0.1 is the newest we have in F16 for eg).

Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>