Skip to contentSkip to conversation

xfs cluster rewrites is broken?

PATCH Alexander Y. Fomichev 2 participants 3 messages Mar 17, 2006 – Mar 21, 2006 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hello,

Two days ago i've try 2.6.16-rc5 on 2-way dual-core Opteron server and faced with a strange system behaviour.
Bulky database updates (host is intended to be a database mysql server), at some point of time leads to the state when system begins continuously write to disk with a speed about of 100-250 Mb/s, really, near to limit of raid controller [lsi320-2x]).
On particular drive only innodb logfiles ( rollback segmets ) have some relationship to mysql. It seems strange because write speed to innodb datafile itself within limits of 20 Mb/s )
( both on the xfs partitions in this case ).

vmstat 1 shows something like this:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0      0 2746688   4232 465784    0    0   132 156046 2853   762 12  4 73 

11
1 0 0 2777440 4232 465784 0 0 0 242742 4050 432 6 4 74
17
1 1 0 2746696 4232 465784 0 0 0 134551 2201 556 13 5 74
8
1 1 0 2760712 4232 465920 0 0 128 296360 4892 1083 5 5 70
19
0 1 0 2746596 4232 465920 0 0 0 209254 3560 9072 10 5 70
15
0 1 0 2745736 4232 466192 0 0 256 142445 2477 721 12 4 75
9
1 0 0 2757396 4232 466328 0 0 128 190102 3375 829 8 4 74
14
0 1 0 2746360 4232 466328 0 0 0 192885 3122 256 9 4 75

and iostat:

nuclear ~ # iostat 1
Linux 2.6.16-rc6-g232a347a (nuclear.srv.ehouse.ru) 03/17/06
[skip]
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 1196.76 222.90 152462.91 194620 133118417
sdb 42.69 11.82 5024.36 10322 4386869

avg-cpu:  %user   %nice    %sys %iowait   %idle
          12.72    0.00    3.99    8.73   74.56

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 2079.00 0.00 265946.00 0 265946
sdb 0.00 0.00 0.00

( it is sda contains ib_logfile[0-3], sdb -- ib_data itself )

while normaly it looks like:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0      0 1772460  98660 1242776    0    0     0  2045  329   697 24  2 74  

0
1 0 0 1801452 98660 1242776 0 0 0 20022 671 1164 23 2 74
1
2 0 0 1782372 98660 1242776 0 0 880 3845 383 1070 20 2 74
4
1 0 0 1771876 98660 1242776 0 0 0 7897 428 718 24 2 74
0
1 0 0 1781572 98660 1242776 0 0 0 8200 446 1314 24 2 74
0
1 0 0 1770020 98660 1242776 0 0 0 11402 478 742 23 2 74
0

I don't seen smothing similar with previous 2.6.15. so assume this is a kernel issue.
clone of last git tree seems affected too, so i've try to 'bisect' a little. One day crowling with git bisect reveal commit related to this.

$git bisect bad 6c4fe19f66a839bce68fcb7b99cdcb0f31c7a59e is first bad commit diff-tree 6c4fe19f66a839bce68fcb7b99cdcb0f31c7a59e (from 7336cea8c2737bbaf0296d67782f760828301d56)

Author: Christoph Hellwig <[email protected]>
Date:   Wed Jan 11 20:49:28 2006 +1100

[XFS] cluster rewrites We can cluster mapped pages aswell, this
improves
performances on rewrites since we can reduce the number of allocator calls.

SGI-PV: 947118
SGI-Modid: xfs-linux-melb:xfs-kern:203829a

http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6c4fe19f66a839bce68fcb7b99cdcb0f31c7a59e;hp=7336cea8c2737bbaf0296d67782f760828301d56

Reverting of this on 2.6.16-rc5 eliminate symptoms completely.

half-intuitive:

fs/xfs/linux-2.6/xfs_aops.c +0 -2
diff -urN a/fs/xfs/linux-2.6/xfs_aops.c b/fs/xfs/linux-2.6/xfs_aops.c
@@ -616,8 +616,6 @@
 				acceptable = (type == IOMAP_UNWRITTEN);
 			else if (buffer_delay(bh))
 				acceptable = (type == IOMAP_DELAY);
-			else if (buffer_mapped(bh))
-				acceptable = (type == 0);
 			else
 				break;
 		} while ((bh = bh->b_this_page) != head);

works too, as i can see, but this is just illustration.

Signature · 3 lines
Best regards.
        Alexander Y. Fomichev <[email protected]>
        Public PGP key: http://sysadminday.org.ru/gluk.asc
#1 lore raw

Hi Alexander,

Quoted 4 lines · Alexander Y. Fomichev

Hello,

Two days ago i've try 2.6.16-rc5 on 2-way dual-core Opteron server and faced with a strange system behaviour.

.....

Quoted 4 lines

[XFS] cluster rewrites We can cluster mapped pages aswell, this
improves
performances on rewrites since we can reduce the number of allocator calls.

FYI, prior to this mod, XFS never clustered rewrites, so it's not surprising that this particular issue has only recently come to light.

fs/xfs/linux-2.6/xfs_aops.c line 621

Quoted 12 lines

diff -urN a/fs/xfs/linux-2.6/xfs_aops.c b/fs/xfs/linux-2.6/xfs_aops.c

fs/xfs/linux-2.6/xfs_aops.c +0 -2
@@ -616,8 +616,6 @@
 				acceptable = (type == IOMAP_UNWRITTEN);
 			else if (buffer_delay(bh))
 				acceptable = (type == IOMAP_DELAY);
-			else if (buffer_mapped(bh))
-				acceptable = (type == 0);
 			else
 				break;
 		} while ((bh = bh->b_this_page) != head);

Well, that switches off rewrite clustering altogether, so it's not surprising that it fixed your problem. It also points out the problem as well - we don't every check if the buffer is dirty before declaring that the page is acceptible for write clustering.

The other cases checked here (buffer_unwritten() and buffer_delay()) are, by defintition, dirty buffers and so they only ever cluster dirty pages. buffer_mapped(), OTOH, could be clean or dirty.....

Can you try the patch below, and see if that fixes the problem you are seeing?

Cheers,

Dave.

Signature · 18 lines
Dave Chinner
R&D Software Enginner
SGI Australian Software Group


Index: 2.6.x-xfs-new/fs/xfs/linux-2.6/xfs_aops.c
===================================================================
--- 2.6.x-xfs-new.orig/fs/xfs/linux-2.6/xfs_aops.c	2006-03-17 13:16:13.000000000 +1100
+++ 2.6.x-xfs-new/fs/xfs/linux-2.6/xfs_aops.c	2006-03-20 10:51:36.906723758 +1100
@@ -647,7 +647,7 @@ xfs_is_delayed_page(
 				acceptable = (type == IOMAP_UNWRITTEN);
 			else if (buffer_delay(bh))
 				acceptable = (type == IOMAP_DELAY);
-			else if (buffer_mapped(bh))
+			else if (buffer_mapped(bh) && buffer_dirty(bh))
 				acceptable = (type == 0);
 			else
 				break;
#2 lore raw
Quoted 40 lines · David Chinner

Hi Alexander,

Quoted 4 lines · Alexander Y. Fomichev

Hello,

Two days ago i've try 2.6.16-rc5 on 2-way dual-core Opteron server and faced with a strange system behaviour.

.....

Quoted 4 lines

[XFS] cluster rewrites We can cluster mapped pages aswell, this
improves
performances on rewrites since we can reduce the number of allocator calls.

FYI, prior to this mod, XFS never clustered rewrites, so it's not surprising that this particular issue has only recently come to light.

fs/xfs/linux-2.6/xfs_aops.c line 621

Quoted 12 lines

diff -urN a/fs/xfs/linux-2.6/xfs_aops.c b/fs/xfs/linux-2.6/xfs_aops.c

fs/xfs/linux-2.6/xfs_aops.c +0 -2
@@ -616,8 +616,6 @@
 				acceptable = (type == IOMAP_UNWRITTEN);
 			else if (buffer_delay(bh))
 				acceptable = (type == IOMAP_DELAY);
-			else if (buffer_mapped(bh))
-				acceptable = (type == 0);
 			else
 				break;
 		} while ((bh = bh->b_this_page) != head);

Well, that switches off rewrite clustering altogether, so it's not surprising that it fixed your problem. It also points out the problem as well - we don't every check if the buffer is dirty before declaring that the page is acceptible for write clustering.

The other cases checked here (buffer_unwritten() and buffer_delay()) are, by defintition, dirty buffers and so they only ever cluster dirty pages. buffer_mapped(), OTOH, could be clean or dirty.....

Quoted 6 lines

Can you try the patch below, and see if that fixes the problem you are seeing?

Cheers,

Dave.

Yes, it works as expected. Thank you very mach for quick answer and clarification.

Signature · 3 lines
Best regards.
        Alexander Y. Fomichev <[email protected]>
        Public PGP key: http://sysadminday.org.ru/gluk.asc