Clear PG_error before reading a page
We've observed that transient disk errors can result in persistent I/O issues at the filesystem level. I can at least put forward the problem with ext2/ext3 and scsci_debug.
Here are the steps to reproduce:
* format an ext3 fs on a SCSI disk simulated by scsci_debug
* mount the fs and create a 10MB file
* unmount/remount the fs
* enable the medium error flag in scsi_debug (please note that I've slightly modified scsi_debug to return a medium error indication when any sector >= 0x1234 is read)
* try to read the file (as expected, got -EIO)
* disable medium error injection in scsi_debug
* try to read the file (keep getting -EIO)
In fact, when the underlying device experiences errors, block_read_full_page() calls ext3_get_block() which correctly returns an I/O error. Consequently, block_read_full_page() set the PG_error flag. However, the problem is that afterwards, when the device no longer returns any errors, PG_error is never cleared and, as a consequence, we are still getting -EIO at the filesystem level.
Granted that block_read_full_page() always tries to fill a full page, I think it should first clear the PG_error flag and set it back if a call to get_block() fails. Besides, we should handle this in block_read_full_page but also in other places that do full page reads. Any comments? I am not able to reproduce the problem with the patch below.
Johann
--
fs/buffer.c +1 -0
@@ -2078,6 +2078,7 @@ int block_read_full_page(struct page *paint fully_mapped =1 ;BUG_ON (!PageLocked (page)); +ClearPageError (page); blocksize =1 << inode->i_blkbits;if (!page_has_buffers (page))create_empty_buffers (page, blocksize,0 );
Quoted 36 lines · On Tue, 15 May 2007 16:37:26 +0200 Johann Lombardi <[email protected]> wrote:
We've observed that transient disk errors can result in persistent I/O issues at the filesystem level. I can at least put forward the problem with ext2/ext3 and
scsci_debug.Here are the steps to reproduce:
* format an ext3 fs on a SCSI disk simulated byscsci_debug
* mount the fs and create a 10MB file
* unmount/remount the fs
* enable the medium error flag inscsi_debug(please note that I've slightly modifiedscsi_debugto return a medium error indication when any sector >=0x1234is read)
* try to read the file (as expected, got-EIO)
* disable medium error injection inscsi_debug
* try to read the file (keep getting-EIO)In fact, when the underlying device experiences errors,
block_read_full_page()callsext3_get_block()which correctly returns an I/O error. Consequently,block_read_full_page()set thePG_errorflag. However, the problem is that afterwards, when the device no longer returns any errors,PG_erroris never cleared and, as a consequence, we are still getting-EIOat the filesystem level.Granted that
block_read_full_page()always tries to fill a full page, I think it should first clear thePG_errorflag and set it back if a call toget_block()fails. Besides, we should handle this inblock_read_full_pagebut also in other places that do full page reads. Any comments? I am not able to reproduce the problem with the patch below.Johann
Signed-off-by Johann Lombardi <[email protected]>--
fs/buffer.c int block_read_full_page(struct page *pa · line 2084
Quoted 7 lines
fs/buffer.c +1
-0@@ -2078,6 +2078,7 @@ int block_read_full_page(struct page *paint fully_mapped =1 ;BUG_ON (!PageLocked (page)); +ClearPageError (page); blocksize =1 << inode->i_blkbits;if (!page_has_buffers (page))create_empty_buffers (page, blocksize,0 );
We need to make sure that this page has PG_uptodate cleared, so that a re-read is forced. And the affected buffer_head, if any, should have buffer_uptodate() cleared.
This change might have horrid interactions with readahead and various application access patterns: if for some reason a dud sector takes 30 seconds of driver futzing to return -EIO and someone (application or kernel) tries to read the same sector tens or hundreds of times, suckiness ensues. This will be hard to test for.
Most reads don't (or shouldn't) go through block_read_full_page(). mpage_readpages() does the heavy lifting.
Quoted 3 lines · Andrew Morton
We need to make sure that this page has
PG_uptodatecleared, so that a re-read is forced. And the affectedbuffer_head, if any, should havebuffer_uptodate()cleared.
ok.
Quoted 5 lines
This change might have horrid interactions with readahead and various application access patterns: if for some reason a dud sector takes 30 seconds of driver futzing to return
-EIOand someone (application or kernel) tries to read the same sector tens or hundreds of times, suckiness ensues. This will be hard to test for.
Yes, I agree. So according to you, what's the best way to address the initial problem (i.e. PG_error never cleared)?
Quoted 2 lines
Most reads don't (or shouldn't) go through
block_read_full_page().mpage_readpages()does the heavy lifting.
Yes, indeed. However, as soon as a call to get_block() fails, do_mpage_readpage() will call block_read_full_page() which will attach buffers to this page.
Consequently, all subsequent reads will go through block_read_full_page().
Johann
Quoted 7 lines · On Tue, 15 May 2007 23:01:24 +0200 Johann Lombardi <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
We need to make sure that this page has
PG_uptodatecleared, so that a re-read is forced. And the affectedbuffer_head, if any, should havebuffer_uptodate()cleared.ok.
Quoted 8 lines
Quoted 5 lines
This change might have horrid interactions with readahead and various application access patterns: if for some reason a dud sector takes 30 seconds of driver futzing to return
-EIOand someone (application or kernel) tries to read the same sector tens or hundreds of times, suckiness ensues. This will be hard to test for.Yes, I agree. So according to you, what's the best way to address the initial problem (i.e.
PG_errornever cleared)?
ooh, hard.
umm, a very safe way would be to free the dang page. Make sure that we have no references on it and that we have a ref on the inode (ie: we're within a syscall which was passed an fd) then run invalidate_mapping_pages()
Quoted 7 lines
Quoted 2 lines
Most reads don't (or shouldn't) go through
block_read_full_page().mpage_readpages()does the heavy lifting.Yes, indeed. However, as soon as a call to
get_block()fails,do_mpage_readpage()will callblock_read_full_page()which will attach buffers to this page.
Consequently, all subsequent reads will go throughblock_read_full_page().
hm, confused. Why is get_block() failing? That has to go and read metadata. If get_block() failed then we don't know what blocks to read to bring this page uptodate, so the pagecache page should remain in state !PageUptodate(), !PageError(). But then, we shouldn't have populated pagecache at that offset at all.
I think I'm missing something here. I suspect you're referring to a mix of reading the blockdev via /dev/hda1 and then using the already-populated pagecache as filesystem metadata, or something?
Is the PageError page part of an S_ISREG file, or is it part of an S_ISBLK file?
Quoted 7 lines · Andrew Morton
Quoted 4 lines
Yes, indeed. However, as soon as a call to
get_block()fails,do_mpage_readpage()will callblock_read_full_page()which will attach buffers to this page.
Consequently, all subsequent reads will go throughblock_read_full_page().hm, confused. Why is
get_block()failing? That has to go and read metadata.
In fact, I am referring to the first part of my test case (i.e. mount the ext3 fs, enable medium errors in scsi_debug and try to read a file from the fs).
So, when I try to read a file, ext3_get_block() needs to read metadata from the disk. However, given that the SCSI disk simulated by scsi_debug reports medium errors, ext3_get_block() returns EIO to the caller (i.e. do_mpage_readpage()). That's why get_block() is failing.
Then, do_mpage_readpage() calls block_read_full_page() (via "goto confused"). block_read_full_page() attaches buffers to this page and calls ext3_get_block() which fails for the same reason as before. Consequently, block_read_full_page() sets the PG_error flag.
Moreover, all subsequent readpage calls will go through block_read_full_page() because the page has now buffers attached.
Basically, my problem is that afterwards, when the device no longer returns any errors, the PG_error flag is never cleared and, as a result, I keep getting -EIO. That's the problem I'd like to address.
Quoted 4 lines
If
get_block()failed then we don't know what blocks to read to bring this page uptodate, so the pagecache page should remain in state !PageUptodate(), !PageError(). But then, we shouldn't have populated pagecache at that offset at all.
Yes, indeed. do_generic_mapping_read() doesn't populate the pagecache if an error occurred.
Still, __do_page_cache_readahead()->read_pages()->ext3_readpages()->mpage_readpages() does populate the pagecache even if the page reads failed.
Quoted 3 lines
I think I'm missing something here. I suspect you're referring to a mix of reading the blockdev via
/dev/hda1and then using the already-populated pagecache as filesystem metadata, or something?
I don't think so, unless I'm missing something.
Quoted 2 lines
Is the PageError page part of an
S_ISREGfile, or is it part of anS_ISBLKfile?
The PageError pages are part of a regular file and have been read through readahead.
Johann
Quoted 8 lines · Johann Lombardi
Quoted 4 lines · Andrew Morton
If
get_block()failed then we don't know what blocks to read to bring this page uptodate, so the pagecache page should remain in state !PageUptodate(), !PageError(). But then, we shouldn't have populated pagecache at that offset at all.
Quoted 4 lines
Yes, indeed.
do_generic_mapping_read()doesn't populate the pagecache if an error occurred.
Still, __do_page_cache_readahead()->read_pages()->ext3_readpages()->mpage_readpages() does populate the pagecache even if the page reads failed.
Doesn't really matter. If the read failed then the page is not uptodate. If it didn't, then it isn't error.
Signature · 1 line
SUSE Labs, Novell Inc.
Quoted 24 lines · On Wed, 16 May 2007 17:39:19 +0200 Johann Lombardi <[email protected]> wrote:
Quoted 7 lines · Andrew Morton
Quoted 4 lines
Yes, indeed. However, as soon as a call to
get_block()fails,do_mpage_readpage()will callblock_read_full_page()which will attach buffers to this page.
Consequently, all subsequent reads will go throughblock_read_full_page().hm, confused. Why is
get_block()failing? That has to go and read metadata.In fact, I am referring to the first part of my test case (i.e. mount the ext3 fs, enable medium errors in
scsi_debugand try to read a file from the fs).So, when I try to read a file,
ext3_get_block()needs to read metadata from the disk. However, given that the SCSI disk simulated byscsi_debugreports medium errors,ext3_get_block()returns EIO to the caller (i.e.do_mpage_readpage()). That's whyget_block()is failing.Then,
do_mpage_readpage()callsblock_read_full_page()(via "goto confused").block_read_full_page()attaches buffers to this page and callsext3_get_block()which fails for the same reason as before. Consequently,block_read_full_page()sets thePG_errorflag.
Moreover, all subsequent readpage calls will go throughblock_read_full_page()because the page has now buffers attached.
Quoted 4 lines
Basically, my problem is that afterwards, when the device no longer returns any errors, the
PG_errorflag is never cleared and, as a result, I keep getting-EIO. That's the problem I'd like to address.
hm, OK. So, where are we up to?
I still worry about the fact that changes in this area could cause the kernel to do a *lot* more IO attempts against failed devices, or failed sectors. We already have a few problems in that area.
What is the actual real-world operational scenario here? Would it be a hotplugged disk? A transient network failure in a SAN? IOW, is it something from which the kernel should automatically recover, or it is a situation in which manual intervention would be better?
Quoted 6 lines · Andrew Morton
Quoted 4 lines
Basically, my problem is that afterwards, when the device no longer returns any errors, the
PG_errorflag is never cleared and, as a result, I keep getting-EIO. That's the problem I'd like to address.hm, OK. So, where are we up to?
Once the errors reported by the underlying device are corrected, we must unmount/remount the filesystem if we want to use it.
In fact, since readahead ignores I/O errors, the pagecache is populated with pages having the PG_error flag set and buffers attached. Since PG_error is then never cleared, we keep getting EIO despite that the underlying device works just fine.
Quoted 4 lines
What is the actual real-world operational scenario here? Would it be a hotplugged disk? A transient network failure in a SAN? IOW, is it something from which the kernel should automatically recover, or it is a situation in which manual intervention would be better?
The real-world operational scenario is a storage system reporting medium errors which can be corrected by a manual intervention.
Johann
Quoted 7 lines · On Thu, 17 May 2007 13:42:50 +0200 Johann Lombardi <[email protected]> wrote:
Quoted 4 lines
What is the actual real-world operational scenario here? Would it be a hotplugged disk? A transient network failure in a SAN? IOW, is it something from which the kernel should automatically recover, or it is a situation in which manual intervention would be better?
The real-world operational scenario is a storage system reporting medium errors which can be corrected by a manual intervention.
So running ioctl(BLKFLSBUF) against the device will fix things up?
Perhaps not, and we should invalidate all the file data as well. The only ways we have of doing that are umount+mount or /proc/sys/vm/drop_caches.
I'm not sure what is the best thing to do here - there are advantages to caching the fact that the device (or some of it) is unreadable.
Quoted 1 line · Andrew Morton
So running ioctl(BLKFLSBUF) against the device will fix things up?
No, 'blockdev --flushbufs /dev/sdb' doesn't help.
Quoted 2 lines
Perhaps not, and we should invalidate all the file data as well. The only ways we have of doing that are umount+mount or
/proc/sys/vm/drop_caches.
'echo > /proc/sys/vm/drop_caches' works. Thanks.
Johann