ext3 -> crash -> fsck -> readlink -> oops
Hi there,
We just suffered some SCSI bus problems on kernel.org that resulted in significant data loss on our mirror volume, but not enough to ditch the whole filesystem and start over (rsync --checksum is a good thing.) However, we keep getting oopses, ostensibly as the result of a garbaged symlink. This filesystem has been run through fsck -y (e2fstools 1.26).
Oops details at:
http://userweb.kernel.org/~hpa/oops-20020516/
The kernel running is 2.4.19-pre8.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 10 lines · H. Peter Anvin
Hi there,
We just suffered some SCSI bus problems on kernel.org that resulted in significant data loss on our mirror volume, but not enough to ditch the whole filesystem and start over (rsync --checksum is a good thing.) However, we keep getting oopses, ostensibly as the result of a garbaged symlink. This filesystem has been run through fsck -y (e2fstools 1.26).
Quoted 6 lines
Oops details at:
http://userweb.kernel.org/~hpa/oops-20020516/The kernel running is 2.4.19-pre8.
I assume that was `fsck -fy'?
You seem have a 4k symlink. I'd have expected fsck to detect that.
It crashed because vfs_readlink ran off the end of the page.
fs/namei.c +1 -1
@@ -1947,7 +1947,7 @@ int vfs_readlink(struct dentry *dentry,if (IS_ERR (link)) goto out; - len =strlen (link); + len =strnlen (link,PAGE_CACHE_SIZE );if (len > (unsigned ) buflen) len = buflen;if (copy_to_user (buffer, link, len))
-
Quoted 7 lines · Andrew Morton
Quoted 1 line
I assume that was `fsck -fy'?
You seem have a 4k symlink. I'd have expected fsck to detect that.
It crashed because
vfs_readlinkran off the end of the page.
Either that... or someone is relying on magic null termination that isn't provided by fsck. Either way, the kernel shouldn't oops... ext3_panic() perhaps, but not oops...
-hpa
Quoted 1 line · Andrew Morton
You seem have a 4k symlink. I'd have expected fsck to detect that.
I just took a look (built a simple testcase) and it does not detect string overflows for long symlinks. It does for fast symlinks, because the data is available right away when it is checking the inode data, but the block data isn't validated at that time, nor later in the run.
fs/namei.c int vfs_readlink(struct dentry *dentry, · line 1953
Quoted 13 lines
It crashed because
vfs_readlinkran off the end of the page.fs/namei.c +1
-1@@ -1947,7 +1947,7 @@ int vfs_readlink(struct dentry *dentry,if (IS_ERR (link)) goto out; - len =strlen (link); + len =strnlen (link,PAGE_CACHE_SIZE );if (len > (unsigned ) buflen) len = buflen;if (copy_to_user (buffer, link, len))
This will fix the oops on behalf of the kernel. I'm just working on a patch to e2fsck to fix that side of things.
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/
Hi,
Review
Quoted 4 lines · Andrew Morton
(quoted diff) +1
-1- len = strlen(link); + len = strnlen(link, PAGE_CACHE_SIZE); if (len > (unsigned) buflen) len = buflen;
Well, if we're going to be testing against buflen anyway, then
namei.c renamed from namei.c.~1~ +1 -3
@@ -1947,9 +1947,7 @@if (IS_ERR (link)) goto out; - len =strlen (link); -if (len > (unsigned ) buflen) - len = buflen; + len =strnlen (link, buflen);if (copy_to_user (buffer, link, len)) len = -EFAULT ; out:
is cleaner.
--Stephen
Quoted 12 lines · Stephen C. Tweedie
Review
Quoted 4 lines · Andrew Morton
(quoted diff) +1
-1- len = strlen(link); + len = strnlen(link, PAGE_CACHE_SIZE); if (len > (unsigned) buflen) len = buflen;Well, if we're going to be testing against buflen anyway, then
namei.c line 1953
Quoted 10 lines
namei.c renamed from namei.c.~1~ +1
-3@@ -1947,9 +1947,7 @@if (IS_ERR (link)) goto out; - len =strlen (link); -if (len > (unsigned ) buflen) - len = buflen; + len =strnlen (link, buflen);if (copy_to_user (buffer, link, len)) len = -EFAULT ; out:
Except that buflen is passed in from userspace, so the user could allocate a 8MB buffer and try to read a broken symlink to get access to kernel memory. How about:
namei.c renamed from namei.c.~1~ +1 -3
@@ -1947,9 +1947,7 @@if (IS_ERR (link)) goto out; - len =strlen (link); -if (len > (unsigned ) buflen) - len = buflen; + len =strnlen (link,min (buflen,PAGE_SIZE ));if (copy_to_user (buffer, link, len)) len = -EFAULT ; out:
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/
Quoted 27 lines · Andrew Morton
Quoted 10 lines · H. Peter Anvin
Hi there,
We just suffered some SCSI bus problems on kernel.org that resulted in significant data loss on our mirror volume, but not enough to ditch the whole filesystem and start over (rsync --checksum is a good thing.) However, we keep getting oopses, ostensibly as the result of a garbaged symlink. This filesystem has been run through fsck -y (e2fstools 1.26).
Quoted 6 lines
Oops details at:
http://userweb.kernel.org/~hpa/oops-20020516/The kernel running is 2.4.19-pre8.
I assume that was `fsck -fy'?
You seem have a 4k symlink. I'd have expected fsck to detect that.
It crashed because
vfs_readlinkran off the end of the page.
fs/namei.c int vfs_readlink(struct dentry *dentry, · line 1953
Quoted 8 lines
fs/namei.c +1
-1@@ -1947,7 +1947,7 @@ int vfs_readlink(struct dentry *dentry,if (IS_ERR (link)) goto out; - len =strlen (link); + len =strnlen (link,PAGE_CACHE_SIZE );if (len > (unsigned ) buflen) len = buflen;if (copy_to_user (buffer, link, len))
OK, I posted a patch to e2fsck on ext2-devel (against Ted's BK repository, but should be OK with 1.27) which should detect and fix (i.e. remove) the symlinks without nul termination that are causing this problem. Let me know how it works (you might want to try 'e2fsck -fn <dev>' first just to see that it detects the problem without actually changing the filesystem.
Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/