New: Large write to USB stick freezes unrelated tasks for a long time

https://bugzilla.kernel.org/show_bug.cgi?id=31142

Summary: Large write to USB stick freezes unrelated tasks for a
long time
Product: IO/Storage
Version: 2.5
Kernel Version: 2.6.38-rc8
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Block Layer
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Created an attachment (id=50902)
--> (https://bugzilla.kernel.org/attachment.cgi?id=50902)
kernel backtraces from hung Eclipse task while writing to usb stick

System is Fedora 14 x86_64 with 4 GB RAM, running vanilla kernel 2.6.38-rc8.

I have a USB 2.0 high-speed memory stick with around 7.5 GB of space. Whenenver I write a large amount of data (several GBs of files) through any means (cp, nautilus GUI, etc), I notice some large applications that I consider unrelated to the I/O operation (Firefox web browser, Thunderbird email viewer, Eclipse IDE) may randomly freeze whenever I try to interact with them. I use Compiz, and I notice the apps getting grayed out, but I have also seen the freeze happening with Metacity and Gnome-shell, so I believe the window manager is irrelevant. Sometimes other smaller tasks (gnome-terminal, gedit) also freeze. For Eclipse, the hang also cause a series of kernel backtraces, attached to this report. The hang usually lasts for several tens of seconds, and may freeze and unfreeze several times while the file copying to USB takes place. All of the hung applications unfreeze themselves after write activity (as seen from the LED in the memory stick) ceases.

Reproducibility: always (with sufficiently large bulk write)
To reproduce:
1) have an usb stick with several GB of free space, with any filesystem (tried vfat and udf)
2) prepare several gb of files to copy from hard disk to usb stick
3) start large application (firefox, eclipse, or thunderbird)
4) check that application is responsive before file copy starts
5) insert usb stick and (auto)mount it. Previously started app is still responsive.
6) start file copy to usb stick with any command
7) attempt to interact with chosen application during the entirety of the file write
Expected result: I/O to usb stick takes place in background, unrelated apps continue to be responsive in foreground.
Actual result: some large tasks freeze for tens of seconds while write takes place.

Feel free to reassign this bug to a different category. It involves I/O, block, USB, and mmap.

rofl, will we ever fix this.

Please enable sysrq and do a sysrq-w when the tasks are blocked so we can find where things are getting stuck. Please avoid email client wordwrapping when sending us the sysrq output.

El 15/03/11 15:53, Andrew Morton escribi__:

Quoted 2 lines

rofl, will we ever fix this.

Does this mean there is already a duplicate of this issue? If so, which one?

Quoted 4 lines

Please enable sysrq and do a sysrq-w when the tasks are blocked so we can find where things are getting stuck. Please avoid email client wordwrapping when sending us the sysrq output.

Quoted 3 lines · On Tue, 15 Mar 2011 17:53:16 -0500 Alex Villac____s…

El 15/03/11 15:53, Andrew Morton escribi__:

Quoted 1 line

rofl, will we ever fix this.

Does this mean there is already a duplicate of this issue? If so, which one?

Nothing specific. Nonsense like this has been happening for at least a decade and it never seems to get a lot better.

Quoted 4 lines

Quoted 4 lines

Please enable sysrq and do a sysrq-w when the tasks are blocked so we can find where things are getting stuck. Please avoid email client wordwrapping when sending us the sysrq output.

El 15/03/11 18:19, Andrew Morton escribi__:

Quoted 6 lines

Quoted 3 lines · On Tue, 15 Mar 2011 17:53:16 -0500 Alex Villac____s…

El 15/03/11 15:53, Andrew Morton escribi__:

Quoted 1 line

rofl, will we ever fix this.

Does this mean there is already a duplicate of this issue? If so, which one?

Quoted 7 lines

Nothing specific. Nonsense like this has been happening for at least a decade and it never seems to get a lot better.

Quoted 4 lines

Quoted 4 lines

Please enable sysrq and do a sysrq-w when the tasks are blocked so we can find where things are getting stuck. Please avoid email client wordwrapping when sending us the sysrq output.

Posted sysrq-w report into original bug report to avoid email word-wrap.

Quoted 7 lines · On Wed, 16 Mar 2011 10:25:16 -0500 Alex Villac____s…

El 15/03/11 18:19, Andrew Morton escribi__:

Quoted 6 lines

Quoted 3 lines · On Tue, 15 Mar 2011 17:53:16 -0500 Alex Villac____s…

El 15/03/11 15:53, Andrew Morton escribi__:

Quoted 1 line

rofl, will we ever fix this.

Does this mean there is already a duplicate of this issue? If so, which one?

Quoted 8 lines

Quoted 7 lines

Nothing specific. Nonsense like this has been happening for at least a decade and it never seems to get a lot better.

Quoted 4 lines

Quoted 4 lines

Please enable sysrq and do a sysrq-w when the tasks are blocked so we can find where things are getting stuck. Please avoid email client wordwrapping when sending us the sysrq output.

Posted sysrq-w report into original bug report to avoid email word-wrap.

https://bugzilla.kernel.org/attachment.cgi?id=50952

Interesting bits:

[70874.969550] thunderbird-bin D 000000010434e04e     0 32283  32279 0x00000080
[70874.969553]  ffff88011ba91838 0000000000000086 ffff880100000000 0000000000013880
[70874.969557]  0000000000013880 ffff88010a231780 ffff88011ba91fd8 0000000000013880
[70874.969560]  0000000000013880 0000000000013880 0000000000013880 ffff88011ba91fd8
[70874.969564] Call Trace:
[70874.969567]  [<ffffffff810d7ed3>] ? sync_page+0x0/0x4d
[70874.969569]  [<ffffffff810d7ed3>] ? sync_page+0x0/0x4d
[70874.969572]  [<ffffffff8147cedb>] io_schedule+0x47/0x62
[70874.969575]  [<ffffffff810d7f1c>] sync_page+0x49/0x4d
[70874.969577]  [<ffffffff8147d36a>] __wait_on_bit+0x48/0x7b
[70874.969580]  [<ffffffff810d8100>] wait_on_page_bit+0x72/0x79
[70874.969583]  [<ffffffff8106cdb4>] ? wake_bit_function+0x0/0x31
[70874.969586]  [<ffffffff8111833c>] migrate_pages+0x1ac/0x38d
[70874.969589]  [<ffffffff8110d6d7>] ? compaction_alloc+0x0/0x2a4
[70874.969592]  [<ffffffff8110ddfd>] compact_zone+0x3f4/0x60e
[70874.969595]  [<ffffffff8110e1dc>] compact_zone_order+0xc2/0xd1
[70874.969599]  [<ffffffff8110e27f>] try_to_compact_pages+0x94/0xea
[70874.969602]  [<ffffffff810de916>] __alloc_pages_direct_compact+0xa9/0x1a5
[70874.969605]  [<ffffffff810ddbb8>] ? drain_local_pages+0x0/0x17
[70874.969607]  [<ffffffff810df0b0>] __alloc_pages_nodemask+0x69e/0x766
[70874.969610]  [<ffffffff81113701>] ? __slab_free+0x6d/0xf6
[70874.969614]  [<ffffffff8110c0de>] alloc_pages_vma+0xec/0xf1
[70874.969617]  [<ffffffff8111be1c>] do_huge_pmd_anonymous_page+0xbf/0x267
[70874.969620]  [<ffffffff810f2497>] ? pmd_offset+0x19/0x40
[70874.969623]  [<ffffffff810f5c70>] handle_mm_fault+0x15d/0x20f
[70874.969626]  [<ffffffff8100f26a>] ? arch_get_unmapped_area_topdown+0x195/0x28f
[70874.969629]  [<ffffffff8148178c>] do_page_fault+0x33b/0x35d
[70874.969632]  [<ffffffff810fb07d>] ? do_mmap_pgoff+0x29a/0x2f4
[70874.969635]  [<ffffffff8112dcee>] ? path_put+0x22/0x27
[70874.969638]  [<ffffffff8147f145>] page_fault+0x25/0x30

[70874.969731] gedit           D 000000010434dfb0     0 32356      1 0x00000080
[70874.969734]  ffff8800982ab558 0000000000000082 ffff880102400001 0000000000013880
[70874.969737]  0000000000013880 ffff880117408000 ffff8800982abfd8 0000000000013880
[70874.969741]  0000000000013880 0000000000013880 0000000000013880 ffff8800982abfd8
[70874.969744] Call Trace:
[70874.969747]  [<ffffffff8147cedb>] io_schedule+0x47/0x62
[70874.969750]  [<ffffffff8121c403>] get_request_wait+0x10a/0x197
[70874.969753]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
[70874.969756]  [<ffffffff8121ccc4>] __make_request+0x2c8/0x3e0
[70874.969759]  [<ffffffff8111487d>] ? kmem_cache_alloc+0x73/0xeb
[70874.969762]  [<ffffffff8121bb67>] generic_make_request+0x2bc/0x336
[70874.969765]  [<ffffffff811213a5>] ? lookup_page_cgroup+0x36/0x4c
[70874.969768]  [<ffffffff8121bcc1>] submit_bio+0xe0/0xff
[70874.969770]  [<ffffffff8114d72d>] ? bio_alloc_bioset+0x4d/0xc4
[70874.969773]  [<ffffffff810edf1f>] ? inc_zone_page_state+0x2d/0x2f
[70874.969776]  [<ffffffff81149274>] submit_bh+0xe8/0x10e
[70874.969779]  [<ffffffff8114b9fa>] __block_write_full_page+0x1ea/0x2da
[70874.969782]  [<ffffffffa0689202>] ? udf_get_block+0x0/0x115 [udf]
[70874.969785]  [<ffffffff8114a640>] ? end_buffer_async_write+0x0/0x12d
[70874.969788]  [<ffffffff8114a640>] ? end_buffer_async_write+0x0/0x12d
[70874.969791]  [<ffffffffa0689202>] ? udf_get_block+0x0/0x115 [udf]
[70874.969794]  [<ffffffff8114bb76>] block_write_full_page_endio+0x8c/0x98
[70874.969796]  [<ffffffff8114bb97>] block_write_full_page+0x15/0x17
[70874.969800]  [<ffffffffa0686027>] udf_writepage+0x18/0x1a [udf]
[70874.969803]  [<ffffffff81117fed>] move_to_new_page+0x106/0x195
[70874.969806]  [<ffffffff811183de>] migrate_pages+0x24e/0x38d
[70874.969809]  [<ffffffff8110d6d7>] ? compaction_alloc+0x0/0x2a4
[70874.969812]  [<ffffffff8110ddfd>] compact_zone+0x3f4/0x60e
[70874.969815]  [<ffffffff81049c78>] ? load_balance+0xcb/0x6b0
[70874.969818]  [<ffffffff8110e1dc>] compact_zone_order+0xc2/0xd1
[70874.969821]  [<ffffffff8110e27f>] try_to_compact_pages+0x94/0xea
[70874.969824]  [<ffffffff810de916>] __alloc_pages_direct_compact+0xa9/0x1a5
[70874.969827]  [<ffffffff810dee79>] __alloc_pages_nodemask+0x467/0x766
[70874.969830]  [<ffffffff810fcfd3>] ? anon_vma_alloc+0x1a/0x1c
[70874.969833]  [<ffffffff81048a30>] ? get_parent_ip+0x11/0x41
[70874.969833]  [<ffffffff8110c0de>] alloc_pages_vma+0xec/0xf1
[70874.969833]  [<ffffffff8123430e>] ? rb_insert_color+0x66/0xe1
[70874.969833]  [<ffffffff8111be1c>] do_huge_pmd_anonymous_page+0xbf/0x267
[70874.969833]  [<ffffffff810f2497>] ? pmd_offset+0x19/0x40
[70874.969833]  [<ffffffff810f5c70>] handle_mm_fault+0x15d/0x20f
[70874.969833]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
[70874.969833]  [<ffffffff8148178c>] do_page_fault+0x33b/0x35d
[70874.969833]  [<ffffffff810fb07d>] ? do_mmap_pgoff+0x29a/0x2f4
[70874.969833]  [<ffffffff8112dcee>] ? path_put+0x22/0x27
[70874.969833]  [<ffffffff8147f145>] page_fault+0x25/0x30

So it appears that the system is full of dirty pages against a slow device and your foreground processes have got stuck in direct reclaim
-> compaction -> migration. That's Mel ;)

What happened to the plans to eliminate direct reclaim?

Quoted 7 lines

So it appears that the system is full of dirty pages against a slow device and your foreground processes have got stuck in direct reclaim
-> compaction -> migration. That's Mel ;)

What happened to the plans to eliminate direct reclaim?

Browsing around bugzilla, I believe that bug 12309 looks very similar to the issue I am experiencing, especially from comment #525 onwards. Am I correct in this?

ah, the epic 12309. https://bugzilla.kernel.org/show_bug.cgi?id=12309.
If you're ever wondering how much we suck, go read that one.

I think what we're seeing in 31142 is a large amount of dirty data buffered against a slow device. Innocent processes enter page reclaim and end up getting stuck trying to write to that heavily-queued and slow device.

If so, that's probably what some of the 12309 participants are seeing. But there are lots of other things in that report too.

Now, the problem you're seeing in 31142 isn't really supposed to happen. In the direct-reclaim case the code will try to avoid initiation of blocking I/O against a congested device, via the bdi_write_congested() test in may_write_to_queue(). Although that code now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case, whodidthat.

However in the case of the new(ish) compaction/migration code I don't think we're performing that test. migrate_pages()->unmap_and_move() will get stuck behind that large&slow IO queue if page reclaim decided to pass it down sync==true, as it apparently has done.

IOW, Mel broke it ;)

El 17/03/11 16:47, Andrew Morton escribi__:

Quoted 25 lines

ah, the epic 12309. https://bugzilla.kernel.org/show_bug.cgi?id=12309.
If you're ever wondering how much we suck, go read that one.

I think what we're seeing in 31142 is a large amount of dirty data buffered against a slow device. Innocent processes enter page reclaim and end up getting stuck trying to write to that heavily-queued and slow device.

If so, that's probably what some of the 12309 participants are seeing. But there are lots of other things in that report too.

Now, the problem you're seeing in 31142 isn't really supposed to happen. In the direct-reclaim case the code will try to avoid initiation of blocking I/O against a congested device, via the bdi_write_congested() test in may_write_to_queue(). Although that code now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case, whodidthat.

However in the case of the new(ish) compaction/migration code I don't think we're performing that test. migrate_pages()->unmap_and_move() will get stuck behind that large&slow IO queue if page reclaim decided to pass it down sync==true, as it apparently has done.

Quoted 2 lines

IOW, Mel broke it ;)

I don't quite follow. In my case, the congested device is the USB stick, but the affected processes should be reading/writing on the hard disk. What kind of queue(s) implementation results in pending writes to the USB stick interfering with I/O to the hard disk? Or am I misunderstanding? I had the (possibly incorrect) impression that each block device had its own I/O queue.

Quoted 4 lines · On Thu, 17 Mar 2011 16:27:29 -0500 Alex Villac____s Lasso…

Quoted 4 lines

So it appears that the system is full of dirty pages against a slow device and your foreground processes have got stuck in direct reclaim
-> compaction -> migration. That's Mel ;)

Quoted 4 lines

Quoted 3 lines

What happened to the plans to eliminate direct reclaim?

Browsing around bugzilla, I believe that bug 12309 looks very similar to the issue I am experiencing, especially from comment #525 onwards. Am I correct in this?

ah, the epic 12309. https://bugzilla.kernel.org/show_bug.cgi?id=12309.
If you're ever wondering how much we suck, go read that one.

I think what we're seeing in 31142 is a large amount of dirty data buffered against a slow device. Innocent processes enter page reclaim and end up getting stuck trying to write to that heavily-queued and slow device.

If so, that's probably what some of the 12309 participants are seeing. But there are lots of other things in that report too.

Now, the problem you're seeing in 31142 isn't really supposed to happen. In the direct-reclaim case the code will try to avoid initiation of blocking I/O against a congested device, via the bdi_write_congested() test in may_write_to_queue(). Although that code now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case, whodidthat.

However in the case of the new(ish) compaction/migration code I don't think we're performing that test. migrate_pages()->unmap_and_move() will get stuck behind that large&slow IO queue if page reclaim decided to pass it down sync==true, as it apparently has done.

IOW, Mel broke it ;)

To confirm if this is the case, I'd be very interested in hearing if this problem persists in the following cases

1. 2.6.38-rc8 with defrag disabled by
echo never >/sys/kernel/mm/transparent_hugepage/defrag (this will stop THP allocations calling into compaction)
2. 2.6.38-rc8 with THP disabled by
echo never > /sys/kernel/mm/transparent_hugepage/enabled (if the problem still persists, then page reclaim is still a problem
but we should still stop THP doing sync writes)
3. 2.6.37 vanilla
(in case this is a new regression introduced since then)

Migration can do sync writes on dirty pages which is why it looks so similar to page reclaim but this can be controlled by the value of sync_migration passed into try_to_compact_pages(). If we find that option 1 above makes the regression go away or at least helps a lot, then a reasonable fix may be to never set sync_migration if __GFP_NO_KSWAPD which is always set for THP allocations. I've added Andrea to the cc to see what he thinks.

\o/ ... no wait, it's the other one - :(

If you look at the stack traces though, all of them had called do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger is new because it's THP triggering compaction that is causing the stalls rather than page reclaim doing direct writeback which was the culprit in the past.

To confirm if this is the case, I'd be very interested in hearing if this problem persists in the following cases

1. 2.6.38-rc8 with defrag disabled by
echo never>/sys/kernel/mm/transparent_hugepage/defrag (this will stop THP allocations calling into compaction)
2. 2.6.38-rc8 with THP disabled by
echo never>
/sys/kernel/mm/transparent_hugepage/enabled
(if the problem still persists, then page reclaim is still a problem
but we should still stop THP doing sync writes)
3. 2.6.37 vanilla
(in case this is a new regression introduced since then)

Migration can do sync writes on dirty pages which is why it looks so similar to page reclaim but this can be controlled by the value of sync_migration passed into try_to_compact_pages(). If we find that option 1 above makes the regression go away or at least helps a lot, then a reasonable fix may be to never set sync_migration if __GFP_NO_KSWAPD which is always set for THP allocations. I've added Andrea to the cc to see what he thinks.

Thanks for the report.

El 18/03/11 06:13, Mel Gorman escribio:

Quoted 23 lines

\o/ ... no wait, it's the other one - :(

If you look at the stack traces though, all of them had called do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger is new because it's THP triggering compaction that is causing the stalls rather than page reclaim doing direct writeback which was the culprit in the past.

To confirm if this is the case, I'd be very interested in hearing if this problem persists in the following cases

1. 2.6.38-rc8 with defrag disabled by
echo never>/sys/kernel/mm/transparent_hugepage/defrag (this will stop THP allocations calling into compaction)
2. 2.6.38-rc8 with THP disabled by
echo never>
/sys/kernel/mm/transparent_hugepage/enabled
(if the problem still persists, then page reclaim is still a problem
but we should still stop THP doing sync writes)
3. 2.6.37 vanilla
(in case this is a new regression introduced since then)

Quoted 9 lines

Migration can do sync writes on dirty pages which is why it looks so similar to page reclaim but this can be controlled by the value of sync_migration passed into try_to_compact_pages(). If we find that option 1 above makes the regression go away or at least helps a lot, then a reasonable fix may be to never set sync_migration if __GFP_NO_KSWAPD which is always set for THP allocations. I've added Andrea to the cc to see what he thinks.

Thanks for the report.

I have just done tests 1 and 2 on 2.6.38 (final, not -rc8), and I have verified that echoing "never" on either /sys/kernel/mm/transparent_hugepage/defrag or /sys/kernel/mm/transparent_hugepage/enabled does allow the file copy to USB to proceed smoothly (copying 4GB of data). Just to verify, I later wrote "always" to both files, and sure enough, some applications stalled when I repeated the same file copy. So I have at least a workaround for the issue. Given this evidence, will the patch at comment #14 fix the issue for good?

Quoted 32 lines

El 18/03/11 06:13, Mel Gorman escribio:

Quoted 22 lines

\o/ ... no wait, it's the other one - :(

If you look at the stack traces though, all of them had called do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger is new because it's THP triggering compaction that is causing the stalls rather than page reclaim doing direct writeback which was the culprit in the past.

To confirm if this is the case, I'd be very interested in hearing if this problem persists in the following cases

1. 2.6.38-rc8 with defrag disabled by
echo never>/sys/kernel/mm/transparent_hugepage/defrag (this will stop THP allocations calling into compaction)
2. 2.6.38-rc8 with THP disabled by
echo never>
/sys/kernel/mm/transparent_hugepage/enabled
(if the problem still persists, then page reclaim is still a problem
but we should still stop THP doing sync writes)
3. 2.6.37 vanilla
(in case this is a new regression introduced since then)

Quoted 9 lines

Migration can do sync writes on dirty pages which is why it looks so similar to page reclaim but this can be controlled by the value of sync_migration passed into try_to_compact_pages(). If we find that option 1 above makes the regression go away or at least helps a lot, then a reasonable fix may be to never set sync_migration if __GFP_NO_KSWAPD which is always set for THP allocations. I've added Andrea to the cc to see what he thinks.

Thanks for the report.

Quoted 10 lines

I have just done tests 1 and 2 on 2.6.38 (final, not -rc8), and I have verified that echoing "never" on either /sys/kernel/mm/transparent_hugepage/defrag or /sys/kernel/mm/transparent_hugepage/enabled does allow the file copy to USB to proceed smoothly (copying 4GB of data). Just to verify, I later wrote "always" to both files, and sure enough, some applications stalled when I repeated the same file copy. So I have at least a workaround for the issue. Given this evidence, will the patch at comment #14 fix the issue for good?

Thanks for testing and reporting, it's very helpful. Based on that that report the patch should help. Can you test it to be absolutly sure please?

The patch did not help. I have attached a sysrq-w trace with the patch applied in the bug report.

Quoted 1 line · Alex Villaci-s Lasso

The patch did not help. I have attached a sysrq-w trace with the patch applied in the bug report.

Most processes are stuck in udf_writepage. That's because migrate is calling ->writepage on dirty pages even when sync=0.

This may do better, can you test it in replacement of the previous patch?

===
Subject: compaction: use async migrate for __GFP_NO_KSWAPD

From: Andrea Arcangeli <[email protected]>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory to succeed (they have graceful fallback). Waiting for I/O in those, tends to be overkill in terms of latencies, so we can reduce their latency by disabling sync migrate.

Stop calling ->writepage on dirty cache when migrate sync mode is not set.

Signed-off-by Andrea Arcangeli <[email protected]>

---
 mm/migrate.c    |   35 ++++++++++++++++++++++++++---------
 mm/page_alloc.c |    2 +-
 2 files changed, 27 insertions(+), 10 deletions(-)

mm/page_alloc.c +1 -1

@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

mm/migrate.c +2 -2

@@ -536,10 +536,15 @@ static int writeout(struct address_space
  * Default handling if a filesystem does not provide a migration function.
  */
 static int fallback_migrate_page(struct address_space *mapping,
-	struct page *newpage, struct page *page)
+				 struct page *newpage, struct page *page,
+				 int sync)
 {
-	if (PageDirty(page))

mm/migrate.c +1 -1

 {
 	struct address_space *mapping;
 	int rc;

@@ -597,7 +602,7 @@ static int move_to_new_page(struct page 
 		rc = mapping->a_ops->migratepage(mapping,
 						newpage, page);
 	else
-		rc = fallback_migrate_page(mapping, newpage, page);
+		rc = fallback_migrate_page(mapping, newpage, page, sync);
 
 	if (rc) {
 		newpage->mapping = NULL;

mm/migrate.c +0 -0

 		if (!force)
 			goto move_newpage;
 

mm/migrate.c +1 -1

 		wait_on_page_writeback(page);
 	}

@@ -757,7 +770,7 @@ static int unmap_and_move(new_page_t get
 
 skip_unmap:
 	if (!page_mapped(page))
-		rc = move_to_new_page(newpage, page, remap_swapcache);
+		rc = move_to_new_page(newpage, page, remap_swapcache, sync);
 
 	if (rc && remap_swapcache)
 		remove_migration_ptes(page, page);

@@ -834,7 +847,11 @@ static int unmap_and_move_huge_page(new_
 	rc = -EAGAIN;
 

The check is at the wrong level I believe because it misses NFS pages which will still get queued for IO which can block waiting on a request to complete.

sync should be bool.

It's overkill to return EBUSY just because we failed to get a lock which could be released very quickly. If we left rc as -EAGAIN it would retry again. The worst case scenario is that the current process is the holder of the lock and the loop is pointless but this is a relatively rare situation (other than Hugh's loopback test aside which seems to be particularly good at triggering that situation).

Quoted 5 lines

(quoted diff) +0 -0

 			goto move_newpage;

@@ -686,7 +695,11 @@ static int unmap_and_move(new_page_t get
 	BUG_ON(charge);
 

Review static int unmap_and_move(new_page_t get · line 703

Quoted 8 lines

(quoted diff) +5 -1

 	if (PageWriteback(page)) {
-		if (!force || !sync)
+		if (!sync) {
+			rc = -EBUSY;
+			goto uncharge;
+		}
+		if (!force)
 			goto uncharge;

Where as this is ok because if the page is being written back, it's fairly unlikely it'll get cleared quickly enough for the retry loop to make sense.

Because of the NFS pages and being a bit aggressive about using -EBUSY, how about the following instead? (build tested only unfortunately)

(quoted diff) +0 -6

@@ -586,18 +586,23 @@ static int move_to_new_page(struct page *newpage, struct page *page,
 	mapping = page_mapping(page);
 	if (!mapping)
 		rc = migrate_page(mapping, newpage, page);
-	else if (mapping->a_ops->migratepage)
-		/*
-		 * Most pages have a mapping and most filesystems
-		 * should provide a migration function. Anonymous
-		 * pages are part of swap space which also has its
-		 * own migration function. This is the most common

===
Subject: mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback

From: Andrea Arcangeli<[email protected]>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory to succeed as they have graceful fallback. Waiting for I/O in those, tends to be overkill in terms of latencies, so we can reduce their latency by disabling sync migrate.

Unfortunately, even with async migration it's still possible for the process to be blocked waiting for a request slot (e.g. get_request_wait in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD blocking, this patch prevents ->writepage being called on dirty page cache for asynchronous migration.

[[email protected]: Avoid writebacks for NFS, retry locked pages, use bool]

Signed-off-by Andrea Arcangeli<[email protected]>

Signed-off-by Mel Gorman<[email protected]>

---
  mm/migrate.c    |   48 +++++++++++++++++++++++++++++++++---------------
  mm/page_alloc.c |    2 +-
  2 files changed, 34 insertions(+), 16 deletions(-)

El 21/03/11 08:48, Andrea Arcangeli escribio:

Quoted 17 lines

===
Subject: mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback

From: Andrea Arcangeli<[email protected]>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory to succeed as they have graceful fallback. Waiting for I/O in those, tends to be overkill in terms of latencies, so we can reduce their latency by disabling sync migrate.

Unfortunately, even with async migration it's still possible for the process to be blocked waiting for a request slot (e.g. get_request_wait in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD blocking, this patch prevents ->writepage being called on dirty page cache for asynchronous migration.

Quoted 7 lines

[[email protected]: Avoid writebacks for NFS, retry locked pages, use bool]

Signed-off-by Andrea Arcangeli<[email protected]>

Signed-off-by Mel Gorman<[email protected]>

---
  mm/migrate.c    |   48 +++++++++++++++++++++++++++++++++---------------
  mm/page_alloc.c |    2 +-
  2 files changed, 34 insertions(+), 16 deletions(-)

The latest patch fails to apply in vanilla 2.6.38:

[alex@srv64 linux-2.6.38]$ patch -p1 --dry-run < ../\[Bug\ 31142\]\ Large\ write\ to\ USB\ stick\ freezes\ unrelated\ tasks\ for\ a\ long\ time.eml (Stripping trailing CRs from patch.)
patching file mm/migrate.c
Hunk #1 FAILED at 564.
Hunk #2 FAILED at 586.
Hunk #3 FAILED at 641.
Hunk #4 FAILED at 686.
Hunk #5 FAILED at 757.
Hunk #6 FAILED at 850.
6 out of 6 hunks FAILED -- saving rejects to file mm/migrate.c.rej
(Stripping trailing CRs from patch.)
patching file mm/page_alloc.c
Hunk #1 FAILED at 2085.
1 out of 1 hunk FAILED -- saving rejects to file mm/page_alloc.c.rej

I will try to apply the patch manually.

I will try to apply the patch manually.

Quoted 2 lines · Mel Gorman

The check is at the wrong level I believe because it misses NFS pages which will still get queued for IO which can block waiting on a request to complete.

But for example ->migratepage won't block at all for swapcache... it's just a pointer for migrate_page... so I didnt' want to skip what could be nonblocking, it just makes migrate less reliable for no good in some case. The fallback case is very likely blocking instead so I only returned -EBUSY there.

Best would be to pass a sync/nonblock param to migratepage(nonblock) so that nfs_migrate_page can pass "nonblock" instead of "false" to nfs_find_and_lock_request.

Quoted 1 line

sync should be bool.

That's better thanks.

Quoted 6 lines

It's overkill to return EBUSY just because we failed to get a lock which could be released very quickly. If we left rc as -EAGAIN it would retry again. The worst case scenario is that the current process is the holder of the lock and the loop is pointless but this is a relatively rare situation (other than Hugh's loopback test aside which seems to be particularly good at triggering that situation).

This change was only meant to possibly avoid some cpu waste in the tight loop, not really "blocking" related so I'm sure ok to drop it for now. The page lock holder better to be quick because with sync=0 the tight loop will retry real fast. If the holder blocks we're not so smart at retrying in a tight loop but for now it's ok.

Quoted 5 lines

Quoted 5 lines

mm/migrate.c +0 -0

 			goto move_newpage;

@@ -686,7 +695,11 @@ static int unmap_and_move(new_page_t get
 	BUG_ON(charge);
 

Quoted 11 lines

mm/migrate.c static int unmap_and_move(new_page_t get · line 708

Quoted 8 lines

mm/migrate.c +5 -1

 	if (PageWriteback(page)) {
-		if (!force || !sync)
+		if (!sync) {
+			rc = -EBUSY;
+			goto uncharge;
+		}
+		if (!force)
 			goto uncharge;

Where as this is ok because if the page is being written back, it's fairly unlikely it'll get cleared quickly enough for the retry loop to make sense.

Agreed.

Quoted 2 lines

Because of the NFS pages and being a bit aggressive about using -EBUSY, how about the following instead? (build tested only unfortunately)

I tested my version below but I think one needs udf with lots of dirty pages plus the usb to trigger this which I don't have setup immediately.

Quoted 10 lines

mm/migrate.c +0 -6

@@ -586,18 +586,23 @@ static int move_to_new_page(struct page *newpage, struct page *page,
 	mapping = page_mapping(page);
 	if (!mapping)
 		rc = migrate_page(mapping, newpage, page);
-	else if (mapping->a_ops->migratepage)
-		/*
-		 * Most pages have a mapping and most filesystems
-		 * should provide a migration function. Anonymous
-		 * pages are part of swap space which also has its
-		 * own migration function. This is the most common

mm/migrate.c static int move_to_new_page(struct page *newpage, struct page *page, · line 592

Quoted 10 lines

mm/migrate.c +4 -6

-		 * path for page migration.
-		 */
-		rc = mapping->a_ops->migratepage(mapping,
-						newpage, page);
-	else
-		rc = fallback_migrate_page(mapping, newpage, page);
+	else {
+		/* Do not writeback pages if !sync */
+		if (PageDirty(page) && !sync)
+			rc = -EBUSY;

I think it's better to at least change it to:

if (PageDirty(page) && !sync && mapping->a_ops->migratepage != migrate_page))

I wasn't sure how to handle noblocking ->migratepage for swapcache and tmpfs but probably the above check is a good enough approximation.

Before sending my patch I thought of adding a "sync" parameter to ->migratepage(..., sync/nonblock) but then the patch become bigger... and I just wanted to know if this was the problem or not so I deferred it.

If we're sure that all migratepage blocks except for things like swapcache/tmpfs or other not-filebacked things that defines it to migrate_page, we're pretty well covered by adding a check like above migratepage == migrate_page and maybe we don't need to add a "sync/nonblock" parameter to ->migratepage(). For example the buffer_migrate_page can block too in lock_buffer.

This is the patch I'm trying with the addition of the above check and some comment space/tab issue cleanup.

Nothing bad jumped out at me. Lets see how it gets on with testing.

Thanks

El 21/03/11 11:37, Mel Gorman escribio:

Quoted 6 lines · Andrea Arcangeli

Nothing bad jumped out at me. Lets see how it gets on with testing.

Thanks

As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).

I have attached the new sysrq-w trace to the bug report.

Quoted 10 lines

El 21/03/11 11:37, Mel Gorman escribio:

Quoted 6 lines · Andrea Arcangeli

Nothing bad jumped out at me. Lets see how it gets on with testing.

Thanks

As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).

I have attached the new sysrq-w trace to the bug report.

migrate and compaction disappeared from the traces as we hoped for. The THP allocations left throttles on writeback during reclaim like any 4k allocation would do:

[ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
[ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
[ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
[ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
[ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
[ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
[ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
[ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1
[ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
[ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
[ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
[ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
[ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
[ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
[ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
[ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30

They throttle on writeback I/O completion like kswapd too:

[ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
[ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
[ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
[ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
[ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
[ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
[ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
[ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
[ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
[ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
[ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
[ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
[ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
[ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
[ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
[ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
[ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
[ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
[ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
[ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f

I'm unsure if there's any other problem left that can be attributed to compaction/migrate (especially considering the THP allocations have no __GFP_REPEAT set and should_continue_reclaim should break the loop if nr_reclaim is zero, plus compaction_suitable requires not much more memory to be reclaimed if compared to no-compaction).

I'd suggest to try a few more times with "echo never> /sys/kernel/mm/transparent_hugepage/enabled" and see if it still makes a difference.

I doubt udf is as optimal as other fs in optimizing locks and writebacks but that may not be relevant to this issue (or at least it most certainly wasn't until now and the same we had so far probably applied to all fs, from this point things aren't as clear as before if responsiveness is still worse than without THP). Anther thing that I find confusing is the udf vs USB stick, I'd expect udf on a usb dvd not a usbstick flash drive, I'd use vfat or ext4 on a flash drive.

Quoted 10 lines · Alex Villaci-s Lasso

El 21/03/11 11:37, Mel Gorman escribio:

Quoted 6 lines · Andrea Arcangeli

Nothing bad jumped out at me. Lets see how it gets on with testing.

Thanks

As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).

I have attached the new sysrq-w trace to the bug report.

migrate and compaction disappeared from the traces as we hoped for. The THP allocations left throttles on writeback during reclaim like any 4k allocation would do:

[ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
[ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
[ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
[ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
[ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
[ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
[ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
[ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1

[ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
[ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
[ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
[ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
[ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
[ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
[ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
[ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30

They throttle on writeback I/O completion like kswapd too:

[ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
[ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
[ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
[ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
[ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
[ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
[ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
[ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
[ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
[ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
[ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
[ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
[ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
[ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
[ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
[ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
[ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
[ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
[ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
[ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f

I'm unsure if there's any other problem left that can be attributed to compaction/migrate (especially considering the THP allocations have no __GFP_REPEAT set and should_continue_reclaim should break the loop if nr_reclaim is zero, plus compaction_suitable requires not much more memory to be reclaimed if compared to no-compaction).

Quoted 26 lines · Andrea Arcangeli

Quoted 10 lines · Alex Villaci-s Lasso

El 21/03/11 11:37, Mel Gorman escribio:

Quoted 6 lines · Andrea Arcangeli

Nothing bad jumped out at me. Lets see how it gets on with testing.

Thanks

As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).

I have attached the new sysrq-w trace to the bug report.

migrate and compaction disappeared from the traces as we hoped for. The THP allocations left throttles on writeback during reclaim like any 4k allocation would do:

[ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
[ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
[ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
[ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
[ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
[ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
[ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
[ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1

Quoted 9 lines

[ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
[ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
[ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
[ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
[ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
[ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
[ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
[ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30

There is an important difference between THP and generic order-0 reclaim though. Once defrag is enabled in THP, it can enter direct reclaim for reclaim/compaction where more pages may be claimed than for a base page fault thereby encountering more dirty pages and stalling.

Quoted 29 lines

They throttle on writeback I/O completion like kswapd too:

[ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
[ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
[ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
[ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
[ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
[ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
[ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
[ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
[ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
[ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
[ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
[ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
[ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
[ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
[ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
[ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
[ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
[ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
[ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
[ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f

Quoted 6 lines

I'm unsure if there's any other problem left that can be attributed to compaction/migrate (especially considering the THP allocations have no __GFP_REPEAT set and should_continue_reclaim should break the loop if nr_reclaim is zero, plus compaction_suitable requires not much more memory to be reclaimed if compared to no-compaction).

I think we are breaking out because the report says the stalls aren't as bad but not before we have waited on writeback of a few dirty pages. This could be addressed in a number of ways but all of them impact THP in some way.

1. We could disable defrag by default. This will avoid the stalling at the cost of fewer pages being promoted even when plenty of clean pages were available.

2. We could redefine __GFP_NO_KSWAPD as __GFP_ASYNC to mean a) do not wake up kswapd that generates IO possibly causing syncs later b) does not queue any pages for IO itself and c) never waits on page writeback. This would also avoid stalls but it would disrupt LRU ordering by reclaiming younger pages than would otherwise have been reclaimed.

3. Again redefine __GFP_NO_KSWAPD but abort allocation if any dirty or writeback page is encountered during reclaim. This makes the assumption that dirty pages at the end of the LRU implies memory is under enough pressure to not care about promotion. This will also result in THP promoting fewer pages but has less impact on LRU ordering.

Which would you prefer? Other suggestions?

I asked yesterday by PM to Alex if the mouse pointer moves or not during the stalls (if it doesn't that may be a scheduler issue with the compaction irq disabled and lack of cond_resched) and to try aa.git. Upstream still misses several compaction improvements that we did over the last weeks and that I've in my queue and that are in -mm as well. So before making more changes, considering the stack traces looks very healthy now, I'd wait to be sure the hangs aren't already solved by any of the other scheduling/irq latency fixes. I guess they aren't going to help but it worth a try. Verifying if this happens with a more optimal filesystem like ext4 I think is also interesting, it may be something in udf internal locking that gets in the way of compaction.

If we still have a problem with current aa.git and ext4, then I'd hope we can find some other more genuine bit to improve like the bits we've improved so far, but if there's nothing wrong and it gets unfixable, then my preference would be to either create a defrag mode that is in between "yes/no", or alternatively to be simpler and make the default between defrag yes|no configurable at build time and through a command line in grub, and hope that SLUB doesn't clashes on it too. The current "default" is optimal for several server environments where we know most of the allocations are long lived. So we want to still have an option to be as reliable as we are toady for those.

I have just tested aa.git as of today, with the USB stick formatted as FAT32. I could no longer reproduce the stalls. There was no need to format as ext4. No /proc workarounds required.

Hi Alex,

could you also try to reverse this below bit (not the whole previous
patch: only the bit below quoted below) with "patch -p1 -R< thismail" on top of your current aa.git tree, and see if you notice any regression compared to the previous aa.git build that worked well?

This is part of the fix, but I'd need to be sure this really makes a difference before sticking to it for long. I'm not concerned by keeping it, but it adds dirt, and the closer THP allocations are to any other high order allocation the better. So the less __GFP_NO_KSWAPD affects the better. The hint about not telling kswapd to insist in the background for order 9 allocations with fallback (like THP) is the maximum I consider clean because there's khugepaged with its alloc_sleep_millisecs that replaces the kswapd task for THP allocations. So that is clean enough, but when __GFP_NO_KSWAPD starts to make compaction behave slightly different from a SLUB order 2 allocation I don't like it (especially because if you later enable SLUB or some driver you may run into the same compaction issue again if the below change is making a difference).

If things works fine even after you reverse the below, we can safely undo this change and also feel safer for all other high order allocations, so it'll make life easier. (plus we don't want unnecessary special changes, we need to be sure this makes a difference to keep it for long)

mm/page_alloc.c +1 -1

@@ -2085,7 +2085,7 @@ rebalance:
  					sync_migration);
  	if (page)
  		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
 
  	/* Try direct reclaim and then allocating */
  	page = __alloc_pages_direct_reclaim(gfp_mask, order,

Quoted 2 lines · Alex Villaci-s Lasso

Quoted 2 lines

I have just tested aa.git as of today, with the USB stick formatted as FAT32. I could no longer reproduce the stall

Probably udf is not optimized enough but I wonder if maybe the udf->vfat change helped more than the other patches. We need the other patches anyway to provide responsive behavior including the one you tested before aa.git so it's not very important if udf was the problem, but it might have been.

Latest update: with 2.6.39-rc1 the stalls only last for a few seconds, but they are still there. So, as I said before, they are reduced but not eliminated.

Hi Alex,

Quoted 3 lines · Alex Villaci-s Lasso

Latest update: with 2.6.39-rc1 the stalls only last for a few seconds, but they are still there. So, as I said before, they are reduced but not eliminated.

Ok from a complete stall for the whole duration of the I/O to a few second stall we're clearly going into the right direction.

The few second stalls happen with udf? And vfat doesn't stall?

Minchan rightly pointed out that the (panik) change we made in page_alloc.c changes the semantics of the __GFP_NO_KSWAPD bit. I also talked with Mel about it. We think it's nicer if we can keep THP allocations as close as any other high order allocation as possible. There are already plenty of __GFP bits with complex semantics. __GFP_NO_KSWAPD is simple and it's nice to stay simple: it means the allocation relies on a different kernel daemon for the background work (which is khugepaged instead of kswapd in the THP case, where khugepaged uses a non intrusive alloc_sleep_millisec throttling in case of MM congestion, unlike kswapd would do).

So this is no solution to your problem (if vfat already works I think that might be a better solution), but we'd like to know if you get any _worse_ stall compared to current 2.6.39-rc, by applying the below patch on top of 2.6.39-rc. If this doesn't make any difference, we can safely apply it to remove unnecessary complications.

Thanks,
Andrea

===
Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD

From: Andrea Arcangeli<[email protected]>

It's uncertain this has been beneficial, so it's safer to undo it. All other compaction users would still go in synchronous mode if a first attempt of async compaction failed. Hopefully we don't need to force special behavior for THP (which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise and to be noticeable). This also make __GFP_NO_KSWAPD return to its original strict semantics specific to bypass kswapd, as THP allocations have khugepaged for the async THP allocations/compactions.

Signed-off-by Andrea Arcangeli<[email protected]>

---
  mm/page_alloc.c |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

El 08/04/11 14:09, Andrea Arcangeli escribio:

Quoted 58 lines

Hi Alex,

Quoted 3 lines · Alex Villaci-s Lasso

Latest update: with 2.6.39-rc1 the stalls only last for a few seconds, but they are still there. So, as I said before, they are reduced but not eliminated.

Ok from a complete stall for the whole duration of the I/O to a few second stall we're clearly going into the right direction.

The few second stalls happen with udf? And vfat doesn't stall?

Minchan rightly pointed out that the (panik) change we made in page_alloc.c changes the semantics of the __GFP_NO_KSWAPD bit. I also talked with Mel about it. We think it's nicer if we can keep THP allocations as close as any other high order allocation as possible. There are already plenty of __GFP bits with complex semantics. __GFP_NO_KSWAPD is simple and it's nice to stay simple: it means the allocation relies on a different kernel daemon for the background work (which is khugepaged instead of kswapd in the THP case, where khugepaged uses a non intrusive alloc_sleep_millisec throttling in case of MM congestion, unlike kswapd would do).

So this is no solution to your problem (if vfat already works I think that might be a better solution), but we'd like to know if you get any _worse_ stall compared to current 2.6.39-rc, by applying the below patch on top of 2.6.39-rc. If this doesn't make any difference, we can safely apply it to remove unnecessary complications.

Thanks,
Andrea

===
Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD

From: Andrea Arcangeli<[email protected]>

It's uncertain this has been beneficial, so it's safer to undo it. All other compaction users would still go in synchronous mode if a first attempt of async compaction failed. Hopefully we don't need to force special behavior for THP (which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise and to be noticeable). This also make __GFP_NO_KSWAPD return to its original strict semantics specific to bypass kswapd, as THP allocations have khugepaged for the async THP allocations/compactions.

Signed-off-by Andrea Arcangeli<[email protected]>

---
  mm/page_alloc.c |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

mm/page_alloc.c +1 -1

@@ -2105,7 +2105,7 @@ rebalance:
                      sync_migration);
      if (page)
          goto got_pg;
-    sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
+    sync_migration = true;
 

mm/page_alloc.c rebalance: · line 2111

Quoted 3 lines

mm/page_alloc.c +0 -0

      /* Try direct reclaim and then allocating */
      page = __alloc_pages_direct_reclaim(gfp_mask, order,

The stalls occur even with vfat. I am no longer using udf, since (right now) it is not necessary. I will test this patch now.

Quoted 27 lines

Quoted 27 lines

===
Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD

From: Andrea Arcangeli<[email protected]>

It's uncertain this has been beneficial, so it's safer to undo it. All other compaction users would still go in synchronous mode if a first attempt of async compaction failed. Hopefully we don't need to force special behavior for THP (which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise and to be noticeable). This also make __GFP_NO_KSWAPD return to its original strict semantics specific to bypass kswapd, as THP allocations have khugepaged for the async THP allocations/compactions.

Signed-off-by Andrea Arcangeli<[email protected]>

---
  mm/page_alloc.c |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

mm/page_alloc.c +1 -1

@@ -2105,7 +2105,7 @@ rebalance:
                      sync_migration);
      if (page)
          goto got_pg;
-    sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
+    sync_migration = true;
 

Quoted 5 lines

mm/page_alloc.c rebalance: · line 2111

Quoted 3 lines

mm/page_alloc.c +0 -0

      /* Try direct reclaim and then allocating */
      page = __alloc_pages_direct_reclaim(gfp_mask, order,

The stalls occur even with vfat. I am no longer using udf, since (right now) it is not necessary. I will test this patch now.

From preliminary tests, I feel that the patch actually eliminates the stalls. I have just copied nearly 6 GB of data into my USB stick and noticed no application freezes.

I retract that. I have tested 2.6.39-rc3 after a day of having several heavy applications loaded in memory, and the stalls do get worse when reversing the patch.