fix corruption case for block size < page size
PATCH 1/1 fix corruption case for block size < page size
On a 4k page system and 512-byte blocksize, this:
xfs_io \
-c "pwrite -S 0x11 -b 4096 0 4096" \
-c "mmap -r 0 512" -c "mread 0 512" -c "munmap" \
-c "truncate 256" \
-c "truncate 513" \
-c "pwrite -S 0x22 -b 512 2048 512" \
-t -d -f testfile
leads to this in the resulting file:
# 00000000 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 |................| # *
# 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| # *
# 00000400 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 |................| <- BAD # *
# 00000800 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 |""""""""""""""""| # *
# 00000a00
laid out like this:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..4]: 63..67 0 (63..67) 5
which is wrong. :) The 2nd series of 1's should be a hole; it is stale data left over from the original write which gets re- mapped in xfs_page_state convert; this is largely because on the previous truncate down, discard_buffer() leaves the discarded bh uptodate - due to page vs. bh uptodate rules, I think.
This all got turned up by xfsqa 091 on ppc with 64k pages & 4k blocks; see also http://oss.sgi.com/bugzilla/show_bug.cgi?id=801.
This would hit 64k page ia64 as well.
This could probably use a bit more investigation; why for example is the mmap read needed above; but the below fixes the problem for me, by recognizing an uptodate but not dirty or mapped buffer at this stage as a hole. At any rate it certainly doesn't need to be written (not dirty)...
I previously submitted an xfsqa testcase for this too.
I've run it through qa but more soak time would probably be good.
---
fs/xfs/linux-2.6/xfs_aops.c +7 -0
@@ -1042,6 +1042,13 @@ xfs_page_state_convert(continue ; } +/* This means its a hole (discard_buffer leaves uptodate set) */ +if (!buffer_dirty (bh) && !buffer_mapped (bh) && +buffer_uptodate (bh)) { + iomap_valid =0 ; +continue ; + } +if (iomap_valid) iomap_valid =xfs_iomap_valid (&iomap, offset);
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
fs/xfs/linux-2.6/xfs_aops.c xfs_page_state_convert( · line 1046
Quoted 10 lines · Eric Sandeen
Index:
xfs/fs/xfs/linux-2.6/xfs_aops.c
===================================================================fs/xfs/linux-2.6/xfs_aops.c +1
-0@@ -1042,6 +1042,13 @@ xfs_page_state_convert(continue ; } +/* This means its a hole (discard_buffer leaves uptodate set) */
I've offended cw's sense of grammar, and I did cringe when I removed the apostrophe in the name of an 80char line, so how about this :)
fs/xfs/linux-2.6/xfs_aops.c +7 -0
@@ -1042,6 +1042,13 @@ xfs_page_state_convert(continue ; } +/* It's a hole. (discard_buffer leaves uptodate set) */ +if (!buffer_dirty (bh) && !buffer_mapped (bh) && +buffer_uptodate (bh)) { + iomap_valid =0 ; +continue ; + } +if (iomap_valid) iomap_valid =xfs_iomap_valid (&iomap, offset);
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Review
Quoted 9 lines · Eric Sandeen
On a 4k page system and 512-byte blocksize, this:
xfs_io\(quoted diff) +0
-6-c "pwrite -S 0x11 -b 4096 0 4096" \ -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" \ -c "truncate 256" \ -c "truncate 513" \ -c "pwrite -S 0x22 -b 512 2048 512" \ -t -d -f testfile
Not to keep belaboring the point, but if anyone reviews this here's a bit more info.
If I blktrace the testcase it looks like this:
8,16 0 1 0.000000000 4222 C W 166979666 + 8 [0] 4k wr 8,16 0 2 0.000367043 4222 C R 166979666 + 8 [0] 4k map rd 8,16 0 3 0.002923548 4222 C N (35 00 ..) [0] 8,16 0 4 0.003108924 4222 C W 200708307 + 9 [0] Log?(trunc) 8,16 0 5 0.020357902 4222 C N (35 00 ..) [0] 8,16 0 6 0.020361434 4222 C W 200708307 + 9 [0] Log?(trunc) 8,16 0 7 0.020745509 4222 C W 166979666 + 1 [0] 512 wr @0 8,16 0 8 0.020940005 4222 C W 166979667 + 1 [0] 512 wr @1 8,16 0 9 0.021172749 4222 C W 166979670 + 1 [0] 512 wr @4
and a detailed look at the data on disk is this:
00000000 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 Block 0(OK) *
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Block 0... *
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Block 1(OK) *
00000400 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 Block 2(BAD) *
00000600 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 Block 3(BAD) *
00000800 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 Block 4(OK) *00000a00
And the bmap information is this:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..4]: 56..60 0 (56..60) 5
So the bad data in blocks 2 and 3 were never rewritten; the buffer heads probably were fine (containing 0's, but I should check) and we simply re-mapped blocks 2 and 3 back into existence, along with their stale data, it seems.
So I think this was just a bad mapping decision, and not a buffer head state/zeroing problem...?
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 47 lines · Eric Sandeen
Review
Quoted 9 lines · Eric Sandeen
On a 4k page system and 512-byte blocksize, this:
xfs_io\(quoted diff) +0
-6-c "pwrite -S 0x11 -b 4096 0 4096" \ -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" \ -c "truncate 256" \ -c "truncate 513" \ -c "pwrite -S 0x22 -b 512 2048 512" \ -t -d -f testfileNot to keep belaboring the point, but if anyone reviews this here's a bit more info.
If I blktrace the testcase it looks like this:
8,16 0 1 0.000000000 4222 C W 166979666 + 8 [0] 4k wr 8,16 0 2 0.000367043 4222 C R 166979666 + 8 [0] 4k map rd 8,16 0 3 0.002923548 4222 C N (35 00 ..) [0] 8,16 0 4 0.003108924 4222 C W 200708307 + 9 [0] Log?(trunc) 8,16 0 5 0.020357902 4222 C N (35 00 ..) [0] 8,16 0 6 0.020361434 4222 C W 200708307 + 9 [0] Log?(trunc) 8,16 0 7 0.020745509 4222 C W 166979666 + 1 [0] 512 wr @0 8,16 0 8 0.020940005 4222 C W 166979667 + 1 [0] 512 wr @1 8,16 0 9 0.021172749 4222 C W 166979670 + 1 [0] 512 wr @4and a detailed look at the data on disk is this:
00000000 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 Block 0(OK) *
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Block 0... *
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Block 1(OK) *
00000400 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 Block 2(BAD) *
00000600 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 Block 3(BAD) *
00000800 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 Block 4(OK) *00000a00And the bmap information is this:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..4]: 56..60 0 (56..60) 5
Quoted 7 lines
So the bad data in blocks 2 and 3 were never rewritten; the buffer heads probably were fine (containing 0's, but I should check) and we simply re-mapped blocks 2 and 3 back into existence, along with their stale data, it seems.
So I think this was just a bad mapping decision, and not a buffer head state/zeroing problem...?
I'm still working through this Eric so I don't fully understand what's going on.
It looks to me like the region was never zeroed at all. In xfs_zero_last_block() we only zero up to the end of the last block (hence the name) but if the last page extends beyond that last block we wont zero that extra space in the page. If that remaining space in the page sits over a hole then xfs_zero_eof() wont zero it either.
In your example above the last write extends the file size from 513 bytes to 2048 bytes. In xfs_zero_last_block() we'll only zero from 513 up to 1024 bytes (ie up to the end of the last block) but leave the rest of the page untouched. Because of the truncate to 256 bytes only the first block is allocated and everything beyond 512 bytes is a hole. More specifically there is a hole under the remainder of the page so xfs_zero_eof() will skip that region and not zero anything.
Quoted 7 lines
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Lachlan McIlroy
I'm still working through this Eric so I don't fully understand what's going on.
It looks to me like the region was never zeroed at all. In
xfs_zero_last_block()we only zero up to the end of the last block (hence the name) but if the last page extends beyond that last block we wont zero that extra space in the page. If that remaining space in the page sits over a hole thenxfs_zero_eof()wont zero it either.
So the testcase steps are (8 blocks per page here).
1: |1111|1111|1111|1111|1111|1111|1111|1111| write 1's
2: |RRRR|1111|1111|1111|1111|1111|1111|1111| mapread first block
|<--------trunc-----------------------
3: |1100| trunc down to 1/2 block
trunc-->|
4: |1100| trunc up to block+1byte
5: | | | | |2222| write 2's (extending)
And we get:
# |1100|0000|1111|1111|2222|----|----|----|
But we should get:
# |1100|HHHH|HHHH|HHHH|2222|----|----|----|
(where "H" means hole)
So no, the data for blocks 1,2,3 is never zeroed, nor should it be, I think - we never partially write those blocks. And the stale 1's coming through in blocks 2 & 3 are not the result of non-zero buffer heads getting written to disk in the last stage; they are simply stale disk blocks from the first step mapped back into the file. (They were originally written as 1's in the first step.)
Quoted 4 lines
In your example above the last write extends the file size from 513 bytes to 2048 bytes. In
xfs_zero_last_block()we'll only zero from 513 up to 1024 bytes (ie up to the end of the last block) but leave the rest of the page untouched.
Actually; after the truncate down step (3) we should have:
|<--------trunc-----------------------
3: |11??| trunc down to 1/2 block
^
|
EOFHm, but does the end of this block get zeroed now or only when we subsequently extend the size? The latter I think...?
So I think in the next step:
trunc-->|
4: |1100| trunc up to block+1byte
^^
now || this part of the block gets zeroed, right, by xfs_zero_eof?
Quoted 3 lines
Because of the truncate to 256 bytes
only the first block is allocated and everything beyond 512 bytes is a hole.
Yep, up until the last write anyway.
Quoted 2 lines
More specifically there is a hole under the remainder of the page so
xfs_zero_eof()will skip that region and not zero anything.
Well, the last write (step 5) is still completely within the page...
Right, that's what it *should* be doing; but in page_state_convert (and I'll admit to not having this 100% nailed down) we write block 1 and map blocks 2 & 3 back into the file, and get:
# |1100|0000|1111|1111|2222|----|----|----|
^^^^ ^^^^
where these |||| |||| blocks are stale data, and block 1 is written (but at least zeroed). How block 1 got zeroed I guess I'm not quite certain yet. But it does not appear that blocks 2 and 3 get *written* any time other than step 1; blktrace seems to confirm this. block 1 does get written, and 0s are written. (But I don't think this block ever should get written either; EOF landed there but only via truncate, not a write).
Crap, now you've got me slightly confused again, and I'll need to look a bit more to be sure I'm 100% clear on what's getting zeroed and when vs. what's getting mapped and why. :)
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 33 lines · Eric Sandeen
Quoted 9 lines · Lachlan McIlroy
I'm still working through this Eric so I don't fully understand what's going on.
It looks to me like the region was never zeroed at all. In
xfs_zero_last_block()we only zero up to the end of the last block (hence the name) but if the last page extends beyond that last block we wont zero that extra space in the page. If that remaining space in the page sits over a hole thenxfs_zero_eof()wont zero it either.So the testcase steps are (8 blocks per page here).
1: |1111|1111|1111|1111|1111|1111|1111|1111| write 1's
2: |RRRR|1111|1111|1111|1111|1111|1111|1111| mapread first block
|<--------trunc-----------------------3: |1100| trunc down to 1/2 block
trunc-->|
4: |1100| trunc up to block+1byte5: | | | | |2222| write 2's (extending)
And we get:
# |1100|0000|1111|1111|2222|----|----|----|
But we should get:
# |1100|HHHH|HHHH|HHHH|2222|----|----|----|
Quoted 8 lines
(where "H" means hole)
So no, the data for blocks 1,2,3 is never zeroed, nor should it be, I think - we never partially write those blocks. And the stale 1's coming through in blocks 2 & 3 are not the result of non-zero buffer heads getting written to disk in the last stage; they are simply stale disk blocks from the first step mapped back into the file. (They were originally written as 1's in the first step.)
Okay, it's making sense now. I've actually fixed this same problem on IRIX and our CXFS clients on solaris, osx and windows but in all those cases when a page is dirty the whole page is dirty so needs to be zeroed. With the individual states of the buffer heads we can remap parts of pages that now cover holes.
Quoted 8 lines
Quoted 5 lines
In your example above the last write extends the file size from 513 bytes to 2048 bytes. In
xfs_zero_last_block()we'll only zero from 513 up to 1024 bytes (ie up to the end of the last block) but leave the rest of the page untouched.Actually; after the truncate down step (3) we should have:
Quoted 8 lines
|<--------trunc----------------------- 3: |11??| trunc down to 1/2 block ^ | EOFHm, but does the end of this block get zeroed now or only when we subsequently extend the size? The latter I think...?
Only when extending the file size.
Quoted 7 lines
So I think in the next step:
trunc-->|
4: |1100| trunc up to block+1byte^^now || this part of the block gets zeroed, right, by
xfs_zero_eof?
Yes (by xfs_zero_last_block()).
Quoted 12 lines
Quoted 4 lines
Because of the truncate to 256 bytes
only the first block is allocated and everything beyond 512 bytes is a hole.Yep, up until the last write anyway.
Quoted 2 lines
More specifically there is a hole under the remainder of the page so
xfs_zero_eof()will skip that region and not zero anything.Well, the last write (step 5) is still completely within the page...
Quoted 8 lines
Right, that's what it *should* be doing; but in
page_state_convert(and I'll admit to not having this 100% nailed down) we write block 1 and map blocks 2 & 3 back into the file, and get:# |1100|0000|1111|1111|2222|----|----|----|
^^^^ ^^^^where these |||| |||| blocks are stale data, and block 1 is written (but at least zeroed). How block 1 got zeroed I guess I'm not quite
I think block 1 got zeroed during the last write because the file size was extended from 513 to 2048. Byte 513 is just inside block 1. But that block should have been a hole and xfs_zero_last_block() should have skipped it.
Quoted 5 lines
certain yet. But it does not appear that blocks 2 and 3 get *written* any time other than step 1; blktrace seems to confirm this. block 1 does get written, and 0s are written. (But I don't think this block ever should get written either; EOF landed there but only via truncate, not a write).
Agree.
Quoted 4 lines
Crap, now you've got me slightly confused again, and I'll need to look a bit more to be sure I'm 100% clear on what's getting zeroed and when vs. what's getting mapped and why. :)
That makes two.
Something else to consider is that there may be allocated blocks entirely beyond eof due to speculative allocation. This means that just because a block within a page is beyond eof does not mean it covers a hole. This is why xfs_zero_eof() looks for blocks to zero between the old eof and the new eof.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Lachlan McIlroy
Quoted 8 lines · Eric Sandeen
Actually; after the truncate down step (3) we should have:
|<--------trunc----------------------- 3: |11??| trunc down to 1/2 block ^ | EOF
Quoted 3 lines
Quoted 2 lines
Hm, but does the end of this block get zeroed now or only when we subsequently extend the size? The latter I think...?
Only when extending the file size.
Right.
Quoted 7 lines
Quoted 6 lines
So I think in the next step:
trunc-->|
4: |1100| trunc up to block+1byte^^now || this part of the block gets zeroed, right, by
xfs_zero_eof?Yes (by
xfs_zero_last_block()).
Right. :) But I *think* that after this step we are actually zeroing into block 1 (2nd block) and causing it to get zeroed/mapped. Off by one maybe?
Quoted 13 lines
Quoted 13 lines
Quoted 3 lines
Because of the truncate to 256 bytes
only the first block is allocated and everything beyond 512 bytes is a hole.Yep, up until the last write anyway.
Quoted 2 lines
More specifically there is a hole under the remainder of the page so
xfs_zero_eof()will skip that region and not zero anything.Well, the last write (step 5) is still completely within the page...
Right, that's what it *should* be doing; but in
page_state_convert(and I'll admit to not having this 100% nailed down) we write block 1 and map blocks 2 & 3 back into the file, and get:
Quoted 8 lines
Quoted 4 lines
# |1100|0000|1111|1111|2222|----|----|----|
^^^^ ^^^^where these |||| |||| blocks are stale data, and block 1 is written (but at least zeroed). How block 1 got zeroed I guess I'm not quite
I think block 1 got zeroed during the last write because the file size was extended from 513 to 2048. Byte 513 is just inside block 1. But that block should have been a hole and
xfs_zero_last_block()should have skipped it.
I think the 2nd extending write does skip it but from a bit more looking the first extending truncate might step into it by one... still looking into that.
Quoted 11 lines
Quoted 5 lines
certain yet. But it does not appear that blocks 2 and 3 get *written* any time other than step 1; blktrace seems to confirm this. block 1 does get written, and 0s are written. (But I don't think this block ever should get written either; EOF landed there but only via truncate, not a write).
Agree.
Quoted 3 lines
Crap, now you've got me slightly confused again, and I'll need to look a bit more to be sure I'm 100% clear on what's getting zeroed and when vs. what's getting mapped and why. :)
That makes two.
:)
Quoted 5 lines
Something else to consider is that there may be allocated blocks entirely beyond eof due to speculative allocation. This means that just because a block within a page is beyond eof does not mean it covers a hole. This is why
xfs_zero_eof()looks for blocks to zero between the old eof and the new eof.
true... yeah, my test may yet be a bit naiive.
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 45 lines · Eric Sandeen
Quoted 9 lines · Lachlan McIlroy
Quoted 8 lines · Eric Sandeen
Actually; after the truncate down step (3) we should have:
|<--------trunc----------------------- 3: |11??| trunc down to 1/2 block ^ | EOFQuoted 3 lines
Quoted 2 lines
Hm, but does the end of this block get zeroed now or only when we subsequently extend the size? The latter I think...?
Only when extending the file size.
Right.
Quoted 7 lines
Quoted 6 lines
So I think in the next step:
trunc-->|
4: |1100| trunc up to block+1byte^^now || this part of the block gets zeroed, right, by
xfs_zero_eof?Yes (by
xfs_zero_last_block()).Right. :) But I *think* that after this step we are actually zeroing into block 1 (2nd block) and causing it to get zeroed/mapped. Off by one maybe?
Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
Because of the truncate to 256 bytes
only the first block is allocated and everything beyond 512 bytes is a hole.Yep, up until the last write anyway.
Quoted 2 lines
More specifically there is a hole under the remainder of the page so
xfs_zero_eof()will skip that region and not zero anything.Well, the last write (step 5) is still completely within the page...
Right, that's what it *should* be doing; but in
page_state_convert(and I'll admit to not having this 100% nailed down) we write block 1 and map blocks 2 & 3 back into the file, and get:# |1100|0000|1111|1111|2222|----|----|----|
^^^^ ^^^^where these |||| |||| blocks are stale data, and block 1 is written (but at least zeroed). How block 1 got zeroed I guess I'm not quite
Quoted 8 lines
Quoted 4 lines
I think block 1 got zeroed during the last write because the file size was extended from 513 to 2048. Byte 513 is just inside block 1. But that block should have been a hole and
xfs_zero_last_block()should have skipped it.I think the 2nd extending write does skip it but from a bit more looking the first extending truncate might step into it by one... still looking into that.
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile
# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1
It looks like what I expect, at this point. But then:
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2
Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
-Eric (heading to bed now...)
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 15 lines · Eric Sandeen
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1It looks like what I expect, at this point. But then:
Quoted 8 lines
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
Replying to myself twice? I really need to go to bed.
So this all does seem to come back to page_state_convert.
Both the extending write in the original case and the sync above find their way there; but esp. in the sync test above, why do we have *any* work to do?
With a little instrumentation I see that for the truncate out; sync test above we get to xfs_vm_writepage() for a page which is *not* dirty, and yet we call page_state_convert on it and map in that 2nd block... Is that right!? I guess it is; ->write_cache_pages() clears dirty before calling writepage. Still why would this page be found dirty on this path. Bah. Bedtime.
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 22 lines · Eric Sandeen
Quoted 9 lines · Lachlan McIlroy
Quoted 8 lines · Eric Sandeen
Actually; after the truncate down step (3) we should have:
|<--------trunc----------------------- 3: |11??| trunc down to 1/2 block ^ | EOFQuoted 3 lines
Quoted 2 lines
Hm, but does the end of this block get zeroed now or only when we subsequently extend the size? The latter I think...?
Only when extending the file size.
Right.
Quoted 6 lines
Quoted 6 lines
So I think in the next step:
trunc-->|
4: |1100| trunc up to block+1byte^^now || this part of the block gets zeroed, right, by
xfs_zero_eof?
Quoted 5 lines
Quoted 1 line
Yes (by
xfs_zero_last_block()).Right. :) But I *think* that after this step we are actually zeroing into block 1 (2nd block) and causing it to get zeroed/mapped. Off by one maybe?
I assumed that the zeroing would stop at the new file size. This bit of code in xfs_zero_eof() should ensure that:
if ((zero_off + zero_len) > offset) zero_len = offset - zero_off;
The truncate down eventually calls truncate_inode_pages_range() which will zero the remainder of a page beyond eof. This could be how block 1 is becoming dirty. If I remember correctly there are checks in fsx to ensure that when mmaping a file with a file size that is not page aligned that the space in the last page beyond eof is all zeroes (a POSIX requirement I think but only holds true on initial mapping since the area can be modified by a user). This has to be done somewhere and may well be in the truncate down.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Because of the truncate to 256 bytes
only the first block is allocated and everything beyond 512 bytes is a hole.Yep, up until the last write anyway.
Quoted 2 lines
More specifically there is a hole under the remainder of the page so
xfs_zero_eof()will skip that region and not zero anything.Well, the last write (step 5) is still completely within the page...
Right, that's what it *should* be doing; but in
page_state_convert(and I'll admit to not having this 100% nailed down) we write block 1 and map blocks 2 & 3 back into the file, and get:# |1100|0000|1111|1111|2222|----|----|----|
^^^^ ^^^^where these |||| |||| blocks are stale data, and block 1 is written (but at least zeroed). How block 1 got zeroed I guess I'm not quite
Quoted 8 lines
Quoted 4 lines
I think block 1 got zeroed during the last write because the file size was extended from 513 to 2048. Byte 513 is just inside block 1. But that block should have been a hole and
xfs_zero_last_block()should have skipped it.I think the 2nd extending write does skip it but from a bit more looking the first extending truncate might step into it by one... still looking into that.
Yes, true since the new file size (513 bytes) is not fsb aligned then the first byte in block 1 needs to be zeroed. But this will only happen if block 1 is allocated. If it's a hole or an unwritten extent then it will be skipped.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
certain yet. But it does not appear that blocks 2 and 3 get *written* any time other than step 1; blktrace seems to confirm this. block 1 does get written, and 0s are written. (But I don't think this block ever should get written either; EOF landed there but only via truncate, not a write).
Quoted 6 lines
Agree.
Quoted 3 lines
Crap, now you've got me slightly confused again, and I'll need to look a bit more to be sure I'm 100% clear on what's getting zeroed and when vs. what's getting mapped and why. :)
That makes two.
:)
Quoted 7 lines
Quoted 5 lines
Something else to consider is that there may be allocated blocks entirely beyond eof due to speculative allocation. This means that just because a block within a page is beyond eof does not mean it covers a hole. This is why
xfs_zero_eof()looks for blocks to zero between the old eof and the new eof.true... yeah, my test may yet be a bit naiive.
I used this command to trigger the problem:
fsx -r 512 -w 512 -W -Z -p 1000 file
It mixes direct I/O with a mmap'ed reads.
1000 mapread 0x3800 thru 0x609c (0x289d bytes) 2000 mapread 0x7600 thru 0x16d0e (0xf70f bytes) READ BAD DATA: offset = 0x9a00, size = 0xa600, fname = file OFFSET GOOD BAD RANGE 0x aa00 0x0000 0x43fb 0x 0
operation# (mod 256) for the bad data may be 67
...
2126(78 mod 256): SKIPPED (no operation)
2127(79 mod 256): TRUNCATE UP from 0xc580 to 0x26629
2128(80 mod 256): WRITE 0x2bc00 thru 0x35fff (0xa400 bytes) HOLE
2129(81 mod 256): TRUNCATE DOWN from 0x36000 to 0x231be
2130(82 mod 256): MAPREAD 0x9e00 thru 0xd6f6 (0x38f7 bytes)
***RRRR***
2131(83 mod 256): READ 0x9a00 thru 0x13fff (0xa600 bytes) ***RRRR***
Correct content saved for comparison
(maybe hexdump "file" vs "file.fsxgood")
At operation 2130 the data at offset 0xaa00 is okay but on the next operation the data at that offset is now bad.
Your patch stops this failure but frankly I don't know why. Zeroing to the end of the page in xfs_zero_last_block() also prevents the failure but now I'm not so sure that makes sense.
The '-f' option to fsx (if you're using the fsx from the XFSQA suite) is supposed to flush and invalidate pages after write operations and force them to be read in from disk. This is very handy to find bugs in the read path that would otherwise be disguised by the page cache. But it looks like msync(MS_INVALIDATE) doesn't work anymore - I'm not seeing any reads, weird.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 26 lines · Eric Sandeen
Quoted 15 lines · Eric Sandeen
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1It looks like what I expect, at this point. But then:
Quoted 8 lines
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
Quoted 7 lines
Replying to myself twice? I really need to go to bed.
So this all does seem to come back to
page_state_convert.Both the extending write in the original case and the sync above find their way there; but esp. in the sync test above, why do we have *any* work to do?
Eric, did you find out why sync was allocating that second block?
Quoted 9 lines
With a little instrumentation I see that for the truncate out; sync test above we get to
xfs_vm_writepage()for a page which is *not* dirty, and yet we callpage_state_converton it and map in that 2nd block... Is that right!? I guess it is; ->write_cache_pages() clears dirty before calling writepage. Still why would this page be found dirty on this path. Bah. Bedtime.-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 26 lines · Lachlan McIlroy
Quoted 25 lines · Eric Sandeen
Quoted 23 lines · Eric Sandeen
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1It looks like what I expect, at this point. But then:
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
Quoted 8 lines
Quoted 7 lines
Replying to myself twice? I really need to go to bed.
So this all does seem to come back to
page_state_convert.Both the extending write in the original case and the sync above find their way there; but esp. in the sync test above, why do we have *any* work to do?
Eric, did you find out why sync was allocating that second block?
I'm afraid this has been on the back burner (or maybe further back) for a while... so... either "no" or "I don't remember" :)
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 31 lines · Eric Sandeen
Quoted 30 lines · Lachlan McIlroy
Quoted 29 lines · Eric Sandeen
Quoted 15 lines · Eric Sandeen
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1It looks like what I expect, at this point. But then:
Quoted 8 lines
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
Replying to myself twice? I really need to go to bed.
So this all does seem to come back to
page_state_convert.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Both the extending write in the original case and the sync above find their way there; but esp. in the sync test above, why do we have *any* work to do?
Eric, did you find out why sync was allocating that second block?
I'm afraid this has been on the back burner (or maybe further back) for a while... so... either "no" or "I don't remember" :)
Just trying your test case. It's not related to direct I/O or mmap I/O since I can reproduce it without those.
# xfs_io -f -c "pwrite -S 0x11 -b 513 0 513" -c "truncate 1" -c "truncate 513" file
wrote 513/513 bytes at offset 0
513.000000 bytes, 1 ops; 0.0000 sec (8.895 MiB/sec and 18181.8182 ops/sec)
# xfs_bmap -vvp file; sync; xfs_bmap -vvp file
file:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..0]: 48..48 0 (48..48) 1 00000
1: [1..1]: hole 1
FLAG Values:
010000 Unwritten preallocated extent
001000 Doesn't begin on stripe unit
000100 Doesn't end on stripe unit
000010 Doesn't begin on stripe width
000001 Doesn't end on stripe width
file:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..1]: 48..49 0 (48..49) 2 00000
FLAG Values:
010000 Unwritten preallocated extent
001000 Doesn't begin on stripe unit
000100 Doesn't end on stripe unit
000010 Doesn't begin on stripe width
000001 Doesn't end on stripe widthxfs_bmap will cause the file to be flushed so there should be no dirty data to be flushed during the sync. Strange.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 51 lines · Lachlan McIlroy
Quoted 31 lines · Eric Sandeen
Quoted 30 lines · Lachlan McIlroy
Quoted 29 lines · Eric Sandeen
Quoted 15 lines · Eric Sandeen
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1It looks like what I expect, at this point. But then:
Quoted 8 lines
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
Replying to myself twice? I really need to go to bed.
So this all does seem to come back to
page_state_convert.Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Both the extending write in the original case and the sync above find their way there; but esp. in the sync test above, why do we have *any* work to do?
Eric, did you find out why sync was allocating that second block?
I'm afraid this has been on the back burner (or maybe further back) for a while... so... either "no" or "I don't remember" :)
Just trying your test case. It's not related to direct I/O or mmap I/O since I can reproduce it without those.
# xfs_io -f -c "pwrite -S 0x11 -b 513 0 513" -c "truncate 1" -c "truncate 513" file wrote 513/513 bytes at offset 0 513.000000 bytes, 1 ops; 0.0000 sec (8.895 MiB/sec and 18181.8182 ops/sec) # xfs_bmap -vvp file; sync; xfs_bmap -vvp file file: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..0]: 48..48 0 (48..48) 1 00000 1: [1..1]: hole 1
....
Quoted 3 lines
file:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS0: [0..1]: 48..49 0 (48..49) 2 00000
....
Quoted 2 lines
xfs_bmapwill cause the file to be flushed so there should be no dirty data to be flushed during the sync. Strange.
Yeah, I noticed that as well. AFAICT it is because the state on on the second buffer has been trashed by the discard but it has been left uptodate. The result is that xfs_flushpages() doesn't think that there is anything to write out and the bmap code finds no extent on that range so іt is considered a hole.
However, when the VFS comes along and writes the dirty inode it thinks that the page is dirty and writes it. xfs_page_state_convert() then sees the second buffer as uptodate but unmapped and within EOF so it allocates the block and writes it out.
IIRC, the code in xfs_page_state_convert() did this allocation to catch mmap() writes into holes. We have ->page_mkwrite to catch this now and turn them into delalloc writes, so perhaps this code path is no longer needed anymore?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 69 lines · Dave Chinner
Quoted 49 lines · Lachlan McIlroy
Quoted 29 lines · Eric Sandeen
Quoted 28 lines · Lachlan McIlroy
Quoted 27 lines · Eric Sandeen
Quoted 15 lines · Eric Sandeen
Gah; or not. what is going on here... Doing just steps 1, 2, 3, 4 (ending on the extending truncate):
# xfs_io -c "pwrite -S 0x11 -b 4096 0 4096" -c "mmap -r 0 512" -c "mread 0 512" -c "munmap" -c "truncate 256" -c "truncate 514" -t -d -f /mnt/scratch/testfile# xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..0]: 63..63 0 (63..63) 1 1: [1..1]: hole 1It looks like what I expect, at this point. But then:
Quoted 8 lines
# sync # xfs_bmap -v /mnt/scratch/testfile /mnt/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..1]: 63..64 0 (63..64) 2Um, why'd that last block get mapped in? mmap vs. direct IO I'm guessing... w/o the mmap read this does not happen.
Replying to myself twice? I really need to go to bed.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
So this all does seem to come back to
page_state_convert.Both the extending write in the original case and the sync above find their way there; but esp. in the sync test above, why do we have *any* work to do?
Eric, did you find out why sync was allocating that second block?
I'm afraid this has been on the back burner (or maybe further back) for a while... so... either "no" or "I don't remember" :)
Just trying your test case. It's not related to direct I/O or mmap I/O since I can reproduce it without those.
# xfs_io -f -c "pwrite -S 0x11 -b 513 0 513" -c "truncate 1" -c "truncate 513" file wrote 513/513 bytes at offset 0 513.000000 bytes, 1 ops; 0.0000 sec (8.895 MiB/sec and 18181.8182 ops/sec) # xfs_bmap -vvp file; sync; xfs_bmap -vvp file file: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..0]: 48..48 0 (48..48) 1 00000 1: [1..1]: hole 1....
Quoted 3 lines
file:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS0: [0..1]: 48..49 0 (48..49) 2 00000....
Quoted 2 lines
xfs_bmapwill cause the file to be flushed so there should be no dirty data to be flushed during the sync. Strange.Yeah, I noticed that as well. AFAICT it is because the state on on the second buffer has been trashed by the discard but it has been left uptodate. The result is that
xfs_flushpages()doesn't think that there is anything to write out and the bmap code finds no extent on that range so іt is considered a hole.However, when the VFS comes along and writes the dirty inode it thinks that the page is dirty and writes it.
xfs_page_state_convert()then sees the second buffer as uptodate but unmapped and within EOF so it allocates the block and writes it out.
Quoted 4 lines
IIRC, the code in
xfs_page_state_convert()did this allocation to catchmmap()writes into holes. We have ->page_mkwrite to catch this now and turn them into delalloc writes, so perhaps this code path is no longer needed anymore?
Thanks Dave.
Eric, I tried your patch again and it fixed this problem but from the sound of Dave's description it may be a case of removing code rather than adding another check.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs