ext4: Fix mmap write protection for data=journal mode
Jan Kara
reviewerPATCH 1/1 ext4: Fix mmap write protection for data=journal mode
Commit afb585a97f81 "ext4: data=journal: write-protect pages on j_submit_inode_data_buffers()") added calls ext4_jbd2_inode_add_write() to track inode ranges whose mappings need to get write-protected during transaction commits. However the added calls use wrong start of a range (0 instead of page offset) and so write protection is not necessarily effective. Use correct range start to fix the problem.
--- fs/ext4/inode.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Mauricio, I think this could be the reason for occasional test failures you were still seeing. Can you try whether this patch fixes those for you? Thanks!
fs/ext4/inode.c +3 -2
@@ -1918,7 +1918,7 @@ static int __ext4_journalled_writepage(struct page *page, }if (ret ==0 ) ret = err; - err =ext4_jbd2_inode_add_write (handle, inode,0 , len); + err =ext4_jbd2_inode_add_write (handle, inode,page_offset (page), len);if (ret ==0 ) ret = err;EXT4_I (inode)->i_datasync_tid = handle->h_transaction->t_tid;
@@ -6157,7 +6157,8 @@ vm_fault_t ext4_page_mkwrite(struct vm_fault *vmf)if (ext4_walk_page_buffers (handle,page_buffers (page),0 , len,NULL , write_end_fn)) goto out_error; -if (ext4_jbd2_inode_add_write (handle, inode,0 , len)) +if (ext4_jbd2_inode_add_write (handle, inode, +page_offset (page), len)) goto out_error;ext4_set_inode_state (inode,EXT4_STATE_JDATA ); }else {
Signature · 1 line
2.16.4
Hey Jan,
Quoted 8 lines · Jan Kara
Commit
afb585a97f81"ext4: data=journal: write-protect pages onj_submit_inode_data_buffers()") added callsext4_jbd2_inode_add_write()to track inode ranges whose mappings need to get write-protected during transaction commits. However the added calls use wrong start of a range (0 instead of page offset) and so write protection is not necessarily effective. Use correct range start to fix the problem.
Quoted 9 lines
Fixes afb585a97f81 ("ext4: data=journal: write-protect pages on j_submit_inode_data_buffers()")Signed-off-by Jan Kara <[email protected]>--- fs/ext4/inode.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Mauricio, I think this could be the reason for occasional test failures you were still seeing. Can you try whether this patch fixes those for you? Thanks!
Thanks! Nice catch. Sure, I'll give it a try and follow up.
Just as FYI, I've been working on debugging and instrumentation to identify the corner cases
where the page has not been write protected; but got a slower pace due to other work items.
cheers,
Mauricio
Quoted 26 lines
fs/ext4/inode.c +3
-3@@ -1918,7 +1918,7 @@ static int __ext4_journalled_writepage(struct page *page, }if (ret ==0 ) ret = err; - err =ext4_jbd2_inode_add_write (handle, inode,0 , len); + err =ext4_jbd2_inode_add_write (handle, inode,page_offset (page), len);if (ret ==0 ) ret = err;EXT4_I (inode)->i_datasync_tid = handle->h_transaction->t_tid;@@ -6157,7 +6157,8 @@ vm_fault_t ext4_page_mkwrite(struct vm_fault *vmf)if (ext4_walk_page_buffers (handle,page_buffers (page),0 , len,NULL , write_end_fn)) goto out_error; -if (ext4_jbd2_inode_add_write (handle, inode,0 , len)) +if (ext4_jbd2_inode_add_write (handle, inode, +page_offset (page), len)) goto out_error;ext4_set_inode_state (inode,EXT4_STATE_JDATA ); }else { --2.16.4
Signature · 1 line
Mauricio Faria de Oliveira
Hey Jan,
Quoted 15 lines · Mauricio Faria de Oliveira
Quoted 14 lines · Jan Kara
Commit
afb585a97f81"ext4: data=journal: write-protect pages onj_submit_inode_data_buffers()") added callsext4_jbd2_inode_add_write()to track inode ranges whose mappings need to get write-protected during transaction commits. However the added calls use wrong start of a range (0 instead of page offset) and so write protection is not necessarily effective. Use correct range start to fix the problem.Fixes afb585a97f81 ("ext4: data=journal: write-protect pages on j_submit_inode_data_buffers()")Signed-off-by Jan Kara <[email protected]>--- fs/ext4/inode.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Quoted 5 lines
Quoted 3 lines
Mauricio, I think this could be the reason for occasional test failures you were still seeing. Can you try whether this patch fixes those for you? Thanks!
Thanks! Nice catch. Sure, I'll give it a try and follow up.
TL;DR:
1) Thanks! The patch fixed almost 100% of the checksum failures.
2) I can send a debug patch to verify buffer checksums before write-out.
3) The remaining, rare checksum failures seem to be due to
a race between commit/write-protect and page writeback related to PageChecked(), clearing pagecache dirty tag used to
write-protect.
4) Test results statistics confirm that the occurrence of checksum failures is really low.
...
Sorry for the delay in following up on this.
Thanks for the patch! The results with v5.10-rc4 are almost 100%:
There are now _very rare_ occasions of journal checksum failures detected; with _zero_ recovery failures in stress-ng/crash/reboot/mount in 1187 loops overnight.
(Previously I'd get 3-5 out of 10.)
I plan to send the debug patch used to verify the buffer checksum in the tag before write-out (catches the checksum failures that fail recovery in advance), if you think it might be useful. I thought of it under CONFIG_JBD2_DEBUG.
...
The remaining checksum changes due to write-protect failures _seem_ to be a race between our write-protect with write_cache_pages() and writeback/sync. But I'm not exactly sure as it's been hard to find the timing/steps for both threads.
The idea is,
Our write_cache_pages() during commit / ext4_journalled_submit_inode_data_buffers()
depends on PAGECACHE_TAG_DIRTY being set, for pagevec_lookup_range_tag() to put such pages in the array to be write-protected with clear_page_dirty_for_io().
With a debug patch to dump_stack() callers of __test_set_page_writeback(), that canxas_clear_mark() it, _while_ the page is still going in our call to write_cache_pages(),
we see this: wb_workfn() -> ext4_writepage() -> ext4_ext4_bio_write_page(),
i.e., _not_ going through ext4_journalled_writepage(), which knows/waits on journal.
The other leg of ext4_writepage() _doesn't_, and thus can clear the tag and prevent
write-protect while the journal commit / our write_cache_pages() is running.
Since the switch to either leg is PageChecked(), I guess this might be due to clearing it right away in ext4_journalled_writepage(), before write-protection.
If that is the issue, perhaps that should be done in our writepage callback or finish callback,
but I haven't thought much about it, until confirming if that's actually the root cause.
(but it seems there's a problem if that bit is set again while we process and clear it.)
...
And just details in the testing methodology, for documentation purposes.
The reproducer for the checksum errors is 'stress-ng --mmap $((4 * $(nproc))) --mmap-file' on data=journal.
There's a test/wait/crash/reboot script that runs it and crashes in $RANDOM <= 120 seconds.
This used to _often_ hit a journal recovery failure in mount on boot, that prevents mount/boot.
Now that the checksum errors occur much less often, there's been _zero_ recovery failures.
With the debug patch to check/report for checksum changes, the kernel logs now show
such occurrences along with the (re)boot markers; e.g.,
[ 0.000000] Linux version 5.10.0-rc4.rechecksum ... [ 80.534910] TESTCASE: checksum changed (tag3: 0x2d57bbdf, calc: 0xe0b4ddcb) [ 0.000000] Linux version 5.10.0-rc4.rechecksum ...
So it's relatively easy to convert those into numbers/statistics with shell glue.
There's a total of 1187 reboots (i.e., stress-ng runs); 1131 had no checksum errors; only 56 had.
The average number of reboots/runs between checksum errors being detected is 20 (min 1, max 140, mean 16)
The average number of checksum errors per reboot/run _that has errors_ is 1 (95%), and max is 2 (5%).
So this confirms it's really rare now, to the point that it's even harder to happen
in the moment that crash happened, and thus jbd2 recovery can finish correctly:
[ 1.492521] EXT4-fs (vda1): INFO: recovery required on readonly filesystem [ 1.493906] EXT4-fs (vda1): write access will be enabled during recovery [ 1.826665] EXT4-fs (vda1): orphan cleanup on readonly fs [ 1.832006] EXT4-fs (vda1): 16 orphan inodes deleted [ 1.833050] EXT4-fs (vda1): recovery complete [ 1.837041] EXT4-fs (vda1): mounted filesystem with journalled data
mode. Opts: (null)
cheers,
--
Mauricio Faria de Oliveira
Theodore Y. Ts'o
maintainerMauricio,
Thanks for your work in finding the corner cases for data=journal. If you don't mind me asking, however --- what are the use cases for you where data=journal is the preferred mode for ext4? There are a lot of advanced features for data=journal mode which don't work. This includes things like dioread_nolock (now the default), delayed allocation, and other optimizations.
It used to be that data=journal pretty nicely fell out of how ext4 worked in "normally". These days, data=journal is becoming more and more an exception case that requires special handling. And to be honest, every so often there has been discussion about whether the maintenance overhead of data=journal has been worth keeping it. So far, we just don't bother making data=journal work with things like delayed allocation, and one of ther reasons why we've kept it around is because it's a unique mode that none of the Linux file systems have.
It would be useful, though, to understand what are the use cases where you (or your customers) are finding data=journal useful, so we can better optimize for their use case. And if there are enough people who care about it --- and clearly, you've invested so much effort that you definitely fall into that category :-) --- then maybe there's a business case for investing more into data=journal and trying to make it something which is easier to maintain and can work with things like delayed allocation.
Thanks,
- Ted
Hey Ted,
Quoted 4 lines · Theodore Y. Ts'o
Mauricio,
Thanks for your work in finding the corner cases for data=journal. If
Thank you and Jan for your suggestions and guidance to fix them.
Quoted 16 lines
you don't mind me asking, however --- what are the use cases for you where data=journal is the preferred mode for ext4? There are a lot of advanced features for data=journal mode which don't work. This includes things like
dioread_nolock(now the default), delayed allocation, and other optimizations.It used to be that data=journal pretty nicely fell out of how ext4 worked in "normally". These days, data=journal is becoming more and more an exception case that requires special handling. And to be honest, every so often there has been discussion about whether the maintenance overhead of data=journal has been worth keeping it. So far, we just don't bother making data=journal work with things like delayed allocation, and one of ther reasons why we've kept it around is because it's a unique mode that none of the Linux file systems have.
Quoted 9 lines
It would be useful, though, to understand what are the use cases where you (or your customers) are finding data=journal useful, so we can better optimize for their use case. And if there are enough people who care about it --- and clearly, you've invested so much effort that you definitely fall into that category :-) --- then maybe there's a business case for investing more into data=journal and trying to make it something which is easier to maintain and can work with things like delayed allocation.
The main reason a customer used data=journal in some cases is it provided them more consistency w/ data on power loss.
Specifically, it prevented some consistency check errors on applications in which it's unfortunately not always possible to request/guarantee that sync() is called (not ext4's fault.)
We've previously exposed the disadvantages with data=journal, including the discussion about its future/maintenance upstream and associated risks with less exposure/testing (eg, this bug! :) in order to assist them with decision-making and maybe switching.
It seems that data=ordered with nodelalloc and a smaller commit= interval could help them, but that's going on experiment stages.
So, it seems this use case is more about data integrity than features, and data=journal does seem to help with that despite the applications.
Glad that ext4 still provides such mode; but it's understandable that maintenance and features and special handling leads to the discussion about no longer keeping it.
Thanks for reaching out to understand the use cases from users, and being willing to consider that into development/maintenance.
There's also a few bits about it from Andreas in the v1 series [1].
Hope this helps!
Mauricio
[1] https://lore.kernel.org/linux-ext4/[email protected]/
Quoted 3 lines
Thanks,
- Ted
Signature · 1 line
Mauricio Faria de Oliveira
Jan Kara
reviewerHi Mauricio!
Quoted 25 lines · Mauricio Faria de Oliveira
Quoted 15 lines · Mauricio Faria de Oliveira
Quoted 14 lines · Jan Kara
Commit
afb585a97f81"ext4: data=journal: write-protect pages onj_submit_inode_data_buffers()") added callsext4_jbd2_inode_add_write()to track inode ranges whose mappings need to get write-protected during transaction commits. However the added calls use wrong start of a range (0 instead of page offset) and so write protection is not necessarily effective. Use correct range start to fix the problem.Fixes afb585a97f81 ("ext4: data=journal: write-protect pages on j_submit_inode_data_buffers()")Signed-off-by Jan Kara <[email protected]>--- fs/ext4/inode.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Quoted 5 lines
Quoted 3 lines
Mauricio, I think this could be the reason for occasional test failures you were still seeing. Can you try whether this patch fixes those for you? Thanks!
Thanks! Nice catch. Sure, I'll give it a try and follow up.
TL;DR:
Quoted 8 lines
1) Thanks! The patch fixed almost 100% of the checksum failures.
2) I can send a debug patch to verify buffer checksums before write-out.
3) The remaining, rare checksum failures seem to be due to
a race between commit/write-protect and page writeback related toPageChecked(), clearing pagecache dirty tag used to
write-protect.
4) Test results statistics confirm that the occurrence of checksum failures is really low.
Glad to hear that!
Quoted 25 lines
Thanks for the patch! The results with v5.10-rc4 are almost 100%:
There are now
_veryrare_occasions of journal checksum failures detected; with _zero_ recovery failures in stress-ng/crash/reboot/mount in 1187 loops overnight.
(Previously I'd get 3-5 out of 10.)I plan to send the debug patch used to verify the buffer checksum in the tag before write-out (catches the checksum failures that fail recovery in advance), if you think it might be useful. I thought of it under
CONFIG_JBD2_DEBUG....
The remaining checksum changes due to write-protect failures _seem_ to be a race between our write-protect with
write_cache_pages()and writeback/sync. But I'm not exactly sure as it's been hard to find the timing/steps for both threads.
The idea is,Our
write_cache_pages()during commit /ext4_journalled_submit_inode_data_buffers()
depends onPAGECACHE_TAG_DIRTYbeing set, forpagevec_lookup_range_tag()to put such pages in the array to be write-protected withclear_page_dirty_for_io().
Quoted 5 lines
With a debug patch to
dump_stack()callers of__test_set_page_writeback(), that canxas_clear_mark()it, _while_ the page is still going in our call towrite_cache_pages(),
we see this:wb_workfn()->ext4_writepage()->ext4_ext4_bio_write_page(),
I guess there was something between wb_workfn() and ext4_writepage(), wasn't there? There should be write_cache_pages()...
Quoted 5 lines
i.e., _not_ going through
ext4_journalled_writepage(), which knows/waits on journal.
The other leg ofext4_writepage()_doesn't_, and thus can clear the tag and prevent
write-protect while the journal commit / ourwrite_cache_pages()is running.
So I don't think this is quite it. If there are two writebacks racing, either of them will writeprotect the page which is enough for commit to be safe (as the mapped write will then trigger ext4_page_mkwrite() which will wait for the end of running commit in jbd2_journal_get_write_access()). Or am I missing something? But there must be still some race as you can still see occasional checksum changes... So I must be missing something.
Quoted 3 lines
Since the switch to either leg is
PageChecked(), I guess this might be due to clearing it right away inext4_journalled_writepage(), before write-protection.
The write-protection happens in clear_page_dirty_for_io() call so that's before our ->writepage() callback is even called.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Hey Jan,
Quoted 14 lines · Jan Kara
Quoted 14 lines
...
The remaining checksum changes due to write-protect failures _seem_ to be a race between our write-protect with
write_cache_pages()and writeback/sync. But I'm not exactly sure as it's been hard to find the timing/steps for both threads.
The idea is,Our
write_cache_pages()during commit /ext4_journalled_submit_inode_data_buffers()
depends onPAGECACHE_TAG_DIRTYbeing set, forpagevec_lookup_range_tag()to put such pages in the array to be write-protected withclear_page_dirty_for_io().
Quoted 9 lines
Quoted 5 lines
With a debug patch to
dump_stack()callers of__test_set_page_writeback(), that canxas_clear_mark()it, _while_ the page is still going in our call towrite_cache_pages(),
we see this:wb_workfn()->ext4_writepage()->ext4_ext4_bio_write_page(),I guess there was something between
wb_workfn()andext4_writepage(), wasn't there? There should bewrite_cache_pages()...
Yes, you're right, I oversimplified the function path between the first two. :) (I'll use '...' in the future to make that clearer.)
Quoted 6 lines
Quoted 5 lines
i.e., _not_ going through
ext4_journalled_writepage(), which knows/waits on journal.
The other leg ofext4_writepage()_doesn't_, and thus can clear the tag and prevent
write-protect while the journal commit / ourwrite_cache_pages()is running.
Quoted 7 lines
So I don't think this is quite it. If there are two writebacks racing, either of them will writeprotect the page which is enough for commit to be safe (as the mapped write will then trigger
ext4_page_mkwrite()which will wait for the end of running commit injbd2_journal_get_write_access()). Or am I missing something? But there must be still some race as you can still see occasional checksum changes... So I must be missing something.
Actually you're right, I missed that -- either writeback will writeprotect the page.
There's still something else going on, but it's me not you who missed something. :)
Thanks for pointing that out.
Quoted 7 lines
Quoted 3 lines
Since the switch to either leg is
PageChecked(), I guess this might be due to clearing it right away inext4_journalled_writepage(), before write-protection.The write-protection happens in
clear_page_dirty_for_io()call so that's before our ->writepage() callback is even called.
Right, I meant write-protection by the other/racing writeback thread. But please nevermind, as this doesn't seem to be the corner case anymore.
Thanks!
Quoted 4 lines
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
Signature · 1 line
Mauricio Faria de Oliveira
Quoted 9 lines · Jan Kara
Commit
afb585a97f81"ext4: data=journal: write-protect pages onj_submit_inode_data_buffers()") added callsext4_jbd2_inode_add_write()to track inode ranges whose mappings need to get write-protected during transaction commits. However the added calls use wrong start of a range (0 instead of page offset) and so write protection is not necessarily effective. Use correct range start to fix the problem.Fixes afb585a97f81 ("ext4: data=journal: write-protect pages on j_submit_inode_data_buffers()")Signed-off-by Jan Kara <[email protected]>
Thanks, applied.
- Ted