hunting an IO hang

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

(lots of cc's added)

Quoted 32 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

Quoted 8 lines

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:

Quoted 45 lines

(lots of cc's added)

Quoted 32 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

Quoted 8 lines

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

Quoted 2 lines

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

We'll see. I gave THP this same run of tests back in November, it passed without any problems (after fixing the related btrfs migration bug). All of the crashes I've seen this weekend had this in the .config:

# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y

Quoted 54 lines · Chris Mason

Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:

Quoted 45 lines

(lots of cc's added)

Quoted 32 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

Quoted 8 lines

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

Quoted 2 lines

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

We'll see. I gave THP this same run of tests back in November, it passed without any problems (after fixing the related btrfs migration bug). All of the crashes I've seen this weekend had this in the .config:

Quoted 3 lines

# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y

I think it's unrelated but reading commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab if page_count(page) == 1 we leave the page in the lru but we return 0 (so the caller of migrate_pages won't call putback_lru_pages to actually free the page,

however compaction would free it because it checks if the list is empty and it ignores the migrate_pages retval). And in

mm/memory-failure.c:1419, nobody is calling putback_lru_pages (it seems a missing bit from that older patch). They seem just two memleak unrelated to the above though.

NOTE: with the last changes compaction is used for all order > 0 and even from kswapd, so you will now be able to trigger bugs in compaction or migration even with THP off. However I'm surprised that you have issues with compaction...

I'm posting this for Minchan to review (not meant for merging, untested).

======
Subject: when migrate_pages returns 0, all pages must have been released

From: Andrea Arcangeli <[email protected]>

In some cases migrate_pages could return zero while still leaving a few pages in the pagelist (and some caller wouldn't notice it has to call putback_lru_pages).

Add one missing putback_lru_pages not added by commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab.

Signed-off-by Andrea Arcangeli <[email protected]>

---

mm/memory-failure.c +1 -0

index 548fbd7..75398b0 100644
@@ -1419,6 +1419,7 @@ int soft_offline_page(struct page *page, int flags)
 		ret = migrate_pages(&pagelist, new_page, MPOL_MF_MOVE_ALL,
 								0, true);
 		if (ret) {
+			putback_lru_pages(&pagelist);
 			pr_info("soft offline: %#lx: migration failed %d, type %lx\n",
 				pfn, ret, page->flags);
 			if (ret > 0)

mm/migrate.c +13 -5

index 46fe8cc..bea2a34 100644
@@ -611,6 +611,14 @@ static int move_to_new_page(struct page *newpage, struct page *page,
 	return rc;
 }
 
+static void unmap_and_move_release_page(struct page *page)
+{
+	list_del(&page->lru);
+	dec_zone_page_state(page, NR_ISOLATED_ANON +
+			    page_is_file_cache(page));
+	putback_lru_page(page);
+}
+
 /*
  * Obtain the lock on page, remove all ptes and migrate the page
  * to the newly allocated page in newpage.

@@ -631,11 +639,14 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
 
 	if (page_count(page) == 1) {
 		/* page was freed from under us. So we are done. */
+		unmap_and_move_release_page(page);
 		goto move_newpage;
 	}
 	if (unlikely(PageTransHuge(page)))
-		if (unlikely(split_huge_page(page)))
+		if (unlikely(split_huge_page(page))) {
+			unmap_and_move_release_page(page);
 			goto move_newpage;
+		}
 
 	/* prepare cgroup just returns 0 or -ENOMEM */
 	rc = -EAGAIN;

@@ -779,10 +790,7 @@ unlock:
  		 * migrated will have kepts its references and be
  		 * restored.
  		 */
- 		list_del(&page->lru);
-		dec_zone_page_state(page, NR_ISOLATED_ANON +
-				page_is_file_cache(page));
-		putback_lru_page(page);
+		unmap_and_move_release_page(page);
 	}
 
 move_newpage:

NOTE: with the last changes compaction is used for all order > 0 and even from kswapd, so you will now be able to trigger bugs in compaction or migration even with THP off. However I'm surprised that you have issues with compaction...

Excerpts from Andrea Arcangeli's message of 2011-01-17 00:11:35 -0500:

[ crashes under load ]

Quoted 1 line

Quoted 4 lines

NOTE: with the last changes compaction is used for all order > 0 and even from kswapd, so you will now be able to trigger bugs in compaction or migration even with THP off. However I'm surprised that you have issues with compaction...

I know I mentioned this in another email, but it is kind of buried in other context. I reproduced my crash with CONFIG_COMPACTION and CONFIG_MIGRATION off.

Quoted 5 lines · Chris Mason

Excerpts from Andrea Arcangeli's message of 2011-01-17 00:11:35 -0500:

[ crashes under load ]

Quoted 1 line

Quoted 8 lines

Quoted 4 lines

NOTE: with the last changes compaction is used for all order > 0 and even from kswapd, so you will now be able to trigger bugs in compaction or migration even with THP off. However I'm surprised that you have issues with compaction...

I know I mentioned this in another email, but it is kind of buried in other context. I reproduced my crash with CONFIG_COMPACTION and CONFIG_MIGRATION off.

Ok, then it was an accident the page->lru got corrupted during migration and it has nothing to do with migration/compaction/thp. This makes sense because we should have noticed long ago if something wasn't stable there.

I reworked the fix for the two memleaks I found while reviewing migration code for this bug (unrelated) introduced by the commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab. It was enough to move the goto to fix this without having to add a new function (it's functionally identical to the one I sent before). It also wouldn't leak memory if it was compaction invoking migrate_pages (only other callers checking the retval of migrate_pages instead of list_empty, could leak memory). As said before, this couldn't explain your problem, and this is only a code review fix, I never triggered this.

This is still only for review for Minchan, not meant for inclusion yet.

===
Subject: when migrate_pages returns 0, all pages must have been released

From: Andrea Arcangeli <[email protected]>

In some cases migrate_pages could return zero while still leaving a few pages in the pagelist (and some caller wouldn't notice it has to call putback_lru_pages after commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab).

Add one missing putback_lru_pages not added by commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab.

Signed-off-by Andrea Arcangeli <[email protected]>

Quoted 43 lines · Andrea Arcangeli

Quoted 5 lines · Chris Mason

Excerpts from Andrea Arcangeli's message of 2011-01-17 00:11:35 -0500:

[ crashes under load ]

Quoted 1 line

Quoted 8 lines

Quoted 4 lines

NOTE: with the last changes compaction is used for all order > 0 and even from kswapd, so you will now be able to trigger bugs in compaction or migration even with THP off. However I'm surprised that you have issues with compaction...

I know I mentioned this in another email, but it is kind of buried in other context. I reproduced my crash with CONFIG_COMPACTION and CONFIG_MIGRATION off.

Ok, then it was an accident the page->lru got corrupted during migration and it has nothing to do with migration/compaction/thp. This makes sense because we should have noticed long ago if something wasn't stable there.

I reworked the fix for the two memleaks I found while reviewing migration code for this bug (unrelated) introduced by the commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab. It was enough to move the goto to fix this without having to add a new function (it's functionally identical to the one I sent before). It also wouldn't leak memory if it was compaction invoking migrate_pages (only other callers checking the retval of migrate_pages instead of list_empty, could leak memory). As said before, this couldn't explain your problem, and this is only a code review fix, I never triggered this.

This is still only for review for Minchan, not meant for inclusion yet.

===
Subject: when migrate_pages returns 0, all pages must have been released

From: Andrea Arcangeli <[email protected]>

In some cases migrate_pages could return zero while still leaving a few pages in the pagelist (and some caller wouldn't notice it has to call putback_lru_pages after commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab).

Quoted 2 lines

Add one missing putback_lru_pages not added by commit cf608ac19c95804dc2df43b1f4f9e068aa9034ab.

It would be better to have another patch.

Quoted 2 lines

Signed-off-by Andrea Arcangeli <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

And don't we need this patch, either?

Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:

Quoted 45 lines

(lots of cc's added)

Quoted 40 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1
 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

Quoted 2 lines

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

We'll see. I gave THP this same run of tests back in November, it passed without any problems (after fixing the related btrfs migration bug). All of the crashes I've seen this weekend had this in the .config:

Quoted 49 lines

Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:

Quoted 47 lines

(lots of cc's added)

Quoted 32 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

Quoted 8 lines

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

Quoted 5 lines

We'll see. I gave THP this same run of tests back in November, it passed without any problems (after fixing the related btrfs migration bug). All of the crashes I've seen this weekend had this in the .config:

I can't find the reset of the thread on any mailing list and am trying to reproduce the problem locally. What workload were you running?

Excerpts from Mel Gorman's message of 2011-01-17 05:27:44 -0500:

Quoted 56 lines · Chris Mason

Quoted 54 lines

Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:

Quoted 45 lines

(lots of cc's added)

Quoted 32 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

Quoted 8 lines

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

Quoted 2 lines

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

We'll see. I gave THP this same run of tests back in November, it passed without any problems (after fixing the related btrfs migration bug). All of the crashes I've seen this weekend had this in the .config:

Quoted 2 lines

I can't find the reset of the thread on any mailing list and am trying to reproduce the problem locally. What workload were you running?

I'm running a very basic IO stress test:

http://oss.oracle.com/~mason/stress.sh

The command line is stress.sh -n 50 -c /mnt/linux-2.6 /mnt

Which starts 50 processes that do cp -a /mnt/linux-2.6 /mnt/stress/$$. Then they verify the result was correct and then they delete it, forever in a loop. In this case my linux-2.6 directory is a full git tree with sources checked out. No obj files though.

This was my crash from an overnight run with CONFIG_COMPACTION off:

# CONFIG_COMPACTION is not set
CONFIG_MIGRATION=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PAGEALLOC=y

I do have an NFS mount active during the run, but it isn't part of the test at all.

I've also managed to get all the procs in the system stuck waiting for IO requests. It is possible these are two different bugs. This list_del oops hits faster if I run the test with a good deal of memory pressure via an external memory hog.

------------[ cut here ]------------
WARNING: at lib/list_debug.c:54 list_del+0x97/0xed()
Hardware name: Bochs
list_del corruption. prev->next should be ffffea000116d478, but was ffffea00014ba2c8
Modules linked in: btrfs lzo_compress

Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #182

Call Trace:
 [<ffffffff8106edc1>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ee7c>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81262e27>] ? list_del+0x97/0xed
 [<ffffffff810dbc59>] ? putback_lru_pages+0x7c/0x1eb
 [<ffffffff810dc070>] ? shrink_inactive_list+0x2a8/0x342
 [<ffffffff810dc676>] ? shrink_zone+0x327/0x3d6
 [<ffffffff8119a17a>] ? nfs_access_cache_shrinker+0x179/0x1a0
 [<ffffffff815c302e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810d2055>] ? zone_watermark_ok_safe+0xa9/0xb8
 [<ffffffff810dd26c>] ? kswapd+0x509/0x876
 [<ffffffff810dcd63>] ? kswapd+0x0/0x876
 [<ffffffff81089e40>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089dbe>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10

-chris

Quoted 62 lines

Excerpts from Mel Gorman's message of 2011-01-17 05:27:44 -0500:

Quoted 50 lines · Chris Mason

Quoted 49 lines

Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:

Quoted 47 lines

(lots of cc's added)

Quoted 32 lines · On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <[email protected]> wrote:

Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:

Quoted 6 lines

.. except I actually didn't add Andrew to the cc after all.

NOW I did.

Oh, and if you can repeat this and bisect it, it would obviously be great. But that sounds rather painful.

Ok, so I've got 3 different problems in 3 totally different areas. I'm running w/kvm, but this VM is very stable with 2.6.37. Running Linus' current git it goes boom in exotic ways, this time it was only on ext3, btrfs code never loaded.

Linus, if you're planning on rc1 tonight I'll send my pull request out the door. Otherwise I'd prefer to fix this and send my pull after actually getting a long btrfs run on the current code.

Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)

WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
Hardware name: Bochs
list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
Modules linked in:
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
Call Trace:
 [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81263d6c>] ? list_del+0xc0/0xed
 [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
 [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
 [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
 [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1

Quoted 8 lines

 [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
 [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
 [<ffffffff81089ce8>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089c66>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
---[ end trace 5c6b7933d16b301f ]---

uh-oh. Does disabling CONFIG_COMPACTION make this go away (requires disabling CONFIG_TRANSPARENT_HUGEPAGE first).

Quoted 8 lines

Quoted 5 lines

We'll see. I gave THP this same run of tests back in November, it passed without any problems (after fixing the related btrfs migration bug). All of the crashes I've seen this weekend had this in the .config:

I can't find the reset of the thread on any mailing list and am trying to reproduce the problem locally. What workload were you running?

I'm running a very basic IO stress test:

Quoted 4 lines

http://oss.oracle.com/~mason/stress.sh

The command line is stress.sh -n 50 -c /mnt/linux-2.6 /mnt

Good to have for future reference. I also successfully reproduced it by having a lot of dd instances running with fsmark running at the same time - basically anything that pounds a filesystem when memory is low.

I'm checking
through parts of the tree to see can I pin down where it goes wrong.

A bisect in this case is problematic. Until commit c5a73c3d55be1faadba35b41a862e036a3b12ddb, compaction was not used very heavily but is used more frequently after that. Hence, "Good" results before that can simply because compaction is not being used. Fortunately, commit 1ce82b69e96c838d007f316b8347b911fdfa9842 looks good so I don't think it's new breakage introduced to migration or compaction.

I did have CONFIG_COMPACTION off for my latest reproduce. The last two have been corruption on the page->lru lists, maybe that'll help narrow our bisect pool down.

Excerpts from Chris Mason's message of 2011-01-17 09:07:40 -0500:

[ various crashes under load with current git ]

Quoted 4 lines

I did have CONFIG_COMPACTION off for my latest reproduce. The last two have been corruption on the page->lru lists, maybe that'll help narrow our bisect pool down.

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

I'll give this a few hours but they seem the most related to my various crashes so far.

Quoted 9 lines · Chris Mason

Excerpts from Chris Mason's message of 2011-01-17 09:07:40 -0500:

[ various crashes under load with current git ]

Quoted 4 lines

I did have CONFIG_COMPACTION off for my latest reproduce. The last two have been corruption on the page->lru lists, maybe that'll help narrow our bisect pool down.

Quoted 6 lines

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

I'll give this a few hours but they seem the most related to my various crashes so far.

I went through the new batched activation code. Shaohua, can you explain to me why the following sequence is not possible?

1. CPU A and B schedule activation of a page (PG_lru && !PG_active)
2. CPU A flushes the page to the active list (PG_lru && PG_active)
3. CPU A isolates the page for scanning/migration and
puts it on private list (!PG_lru && PG_active)
4. CPU B flushes the page to the active list (!PG_lru && PG_active), the deferred activation code now assumes putback mode and adds the page to the active list, thus corrupting the link to the private list of CPU A
5. CPU A does list_del() from the private list (like unmap_and_move() does) and trips up on the corruption

Excerpts from Chris Mason's message of 2011-01-17 09:07:40 -0500:

[ various crashes under load with current git ]

Quoted 1 line

Quoted 3 lines

I did have CONFIG_COMPACTION off for my latest reproduce. The last two have been corruption on the page->lru lists, maybe that'll help narrow our bisect pool down.

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

I'll give this a few hours but they seem the most related to my various crashes so far.

-chris

Quoted 9 lines

Excerpts from Chris Mason's message of 2011-01-17 09:07:40 -0500:

[ various crashes under load with current git ]

Quoted 4 lines

I did have CONFIG_COMPACTION off for my latest reproduce. The last two have been corruption on the page->lru lists, maybe that'll help narrow our bisect pool down.

Quoted 4 lines

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

Confirmed that reverting these patches makes the problem unreproducible for the many_dd's + fsmark for at least an hour here.

Quoted 6 lines

Quoted 5 lines

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

Quoted 2 lines

Confirmed that reverting these patches makes the problem unreproducible for the many_dd's + fsmark for at least an hour here.

After 2+ hours I'm still running with those two commits gone.  I'm confident they are the cause of the crashes.  I also haven't triggered the cfq stalls without them.

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

Quoted 6 lines

Quoted 2 lines

Confirmed that reverting these patches makes the problem unreproducible for the many_dd's + fsmark for at least an hour here.

After 2+ hours I'm still running with those two commits gone. A I'm confident they are the cause of the crashes. A I also haven't triggered the cfq stalls without them.

Ok, so the question is how to proceed from here.

I can easily revert them, and since I was planning on doing -rc1 tonight, I probably will. But I promised Chris to delay until tomorrow if he needed time to chase this down, and while it's now apparently chased down, I'll certainly also be open to delaying until tomorrow if somebody has a patch to fix it.

So right now my plan is:
- I will revert those two later today and then release -rc1 in the evening
UNLESS
- somebody posts a patch for the problem in the next few hours and
Chris/others are willing to give it a good test overnight (or whatever people feel is "sufficient" based on how easily they can trigger the issue), in which case I'd do -rc1 tomorrow (either with the reverts or the patch, depending on how testing works out)

Excerpts from Linus Torvalds's message of 2011-01-17 13:24:55 -0500:

Quoted 22 lines · Chris Mason

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

Quoted 6 lines

Quoted 2 lines

Confirmed that reverting these patches makes the problem unreproducible for the many_dd's + fsmark for at least an hour here.

After 2+ hours I'm still running with those two commits gone. I'm confident they are the cause of the crashes. I also haven't triggered the cfq stalls without them.

Ok, so the question is how to proceed from here.

I can easily revert them, and since I was planning on doing -rc1 tonight, I probably will. But I promised Chris to delay until tomorrow if he needed time to chase this down, and while it's now apparently chased down, I'll certainly also be open to delaying until tomorrow if somebody has a patch to fix it.

Quoted 8 lines

So right now my plan is:
- I will revert those two later today and then release -rc1 in the evening
UNLESS
- somebody posts a patch for the problem in the next few hours and
Chris/others are willing to give it a good test overnight (or whatever people feel is "sufficient" based on how easily they can trigger the issue), in which case I'd do -rc1 tomorrow (either with the reverts or the patch, depending on how testing works out)

If a patch does come in, I'm happy to test it. Mel had a test that triggered within 1-2 minutes, mine took 30 or so, which means I'd want a 2 hour run to convince myself it was really fixed. But, I'll give Mel's fs_mark + dd workload a try on the buggy kernel.

Quoted 32 lines · Chris Mason

Excerpts from Linus Torvalds's message of 2011-01-17 13:24:55 -0500:

Quoted 22 lines · Chris Mason

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

I've reverted 744ed1442757767ffede5008bb13e0805085902e, and d8505dee1a87b8d41b9c4ee1325cd72258226fbc and the run has lasted longer than any runs in the past.

Quoted 6 lines

Quoted 2 lines

Confirmed that reverting these patches makes the problem unreproducible for the many_dd's + fsmark for at least an hour here.

After 2+ hours I'm still running with those two commits gone. I'm confident they are the cause of the crashes. I also haven't triggered the cfq stalls without them.

Ok, so the question is how to proceed from here.

I can easily revert them, and since I was planning on doing -rc1 tonight, I probably will. But I promised Chris to delay until tomorrow if he needed time to chase this down, and while it's now apparently chased down, I'll certainly also be open to delaying until tomorrow if somebody has a patch to fix it.

Quoted 8 lines

So right now my plan is:
- I will revert those two later today and then release -rc1 in the evening
UNLESS
- somebody posts a patch for the problem in the next few hours and
Chris/others are willing to give it a good test overnight (or whatever people feel is "sufficient" based on how easily they can trigger the issue), in which case I'd do -rc1 tomorrow (either with the reverts or the patch, depending on how testing works out)

Quoted 5 lines

If a patch does come in, I'm happy to test it. Mel had a test that triggered within 1-2 minutes, mine took 30 or so, which means I'd want a 2 hour run to convince myself it was really fixed. But, I'll give Mel's fs_mark + dd workload a try on the buggy kernel.

I spent a while seeing if there was a simple patch but it's not trivially fixable. __activate_page() is getting called in too many different situations to be fully sure the function is doing the right thing in all cases. I also couldn't convince myself that the accounting was correct in all cases. I think the idea of batching updates from mark_page_accessed() in particular is a good idea but the patch needs a do-over.

So right now my plan is:
 - I will revert those two later today and then release -rc1 in the evening UNLESS
 - somebody posts a patch for the problem in the next few hours [..]

Quoted 5 lines · Linus Torvalds

So right now my plan is:
 - I will revert those two later today and then release -rc1 in the evening UNLESS
 - somebody posts a patch for the problem in the next few hours [..]

Ok, so nothing obvious popped up, and I reverted the two patches.

I've also seen two other patches floating around here in this thread (one by Andrea, one by Minchan), but didn't apply them as it wasn't entirely clear what the status of those patches were. My current plan is to do -rc1 tonight, and hopefully with the two reverts it will be reasonably stable. We obviously will have several weeks for polishing.

                         Linus