Filesystem/kernel bug?
I was attempting to perform an "emerge --sync" as root on my Gentoo system to update the portage tree (the actual meaning of this command is irrelevant, so don't worry if you're not familiar with it) and had it fail while attempting to change directories. I found this to be quite strange, as I was running the synchronization as root. The directory in question was, "/usr/portage/mail-mta/msmtp". Sure enough, manually changing directories failed with a "permission denied". I then CD'ed to "/usr/portage/mail-mta" and executed a "ls -la". To my surprise, there was no directory named "msmtp" in "/usr/portage/mail-mta". I then tried to change to a random directory name, to verify that the error message wasn't just generic. "cd foo" returned "No such file or directory", as expected.
Following is basically an outline of the steps I performed, which lead to my confusion:
cgx-mail / # cd /usr/portage/mail-mta/
cgx-mail mail-mta # ls -la
ls: msmtp: Permission denied
total 4
drwxr-xr-x 16 root root 416 Dec 27 09:47 .
drwxr-xr-x 135 root root 4008 Dec 27 09:48 ..
drwxr-xr-x 3 root root 336 Dec 15 17:09 courier drwxr-xr-x 3 root root 200 Nov 1 16:35 esmtp drwxr-xr-x 3 root root 400 Dec 17 16:06 exim drwxr-xr-x 3 root root 200 Sep 7 07:10 mini-qmail drwxr-xr-x 3 root root 400 Dec 22 08:45 nbsmtp drwxr-xr-x 3 root root 400 Nov 16 04:38 nullmailer drwxr-xr-x 3 root root 328 Dec 23 01:37 postfix drwxr-xr-x 3 root root 480 Dec 15 17:37 qmail drwxr-xr-x 3 root root 352 Jul 20 09:37 qmail-ldap drwxr-xr-x 3 root root 248 Nov 24 17:07 qmail-mysql drwxr-xr-x 3 root root 296 Dec 21 09:11 sendmail drwxr-xr-x 3 root root 240 Dec 25 16:07 ssmtp drwxr-xr-x 3 root root 280 Dec 6 10:16 xmail
cgx-mail mail-mta # cd msmtp
-bash: cd: msmtp: Permission denied
cgx-mail mail-mta # cd foo
-bash: cd: foo: No such file or directory
cgx-mail mail-mta # uname -r
2.6.9-gentoo-r9
cgx-mail mail-mta # cat /etc/fstab
/dev/sda1 /boot reiserfs noatime,notail 1 1 /dev/sda3 / reiserfs noatime 0 0 /dev/sda2 none swap sw 0 0 /dev/fd0 /mnt/floppy auto noauto 0 0 none /proc proc defaults 0 0
At this point, I'm not really sure what's going on.
Can anybody help me out? I've posted any information I thought would be relevant. If you need anything else, feel free to let me know.
Thank you,
Signature · 1 line
Sean
Update:
I found these events --and many similar-- in my log file.
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74887 0x0 SD] Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD] Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]
Does this shed any new light? Does it look like I might have a corrupted file system?
Thanks again,
Signature · 1 line
Sean
Hello
Quoted 23 lines · Sean
Update:
I found these events --and many similar-- in my log file.
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74887 0x0 SD] Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD] Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070:
Quoted 6 lines
reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 748880x0SD]Does this shed any new light? Does it look like I might have a corrupted file system?
Yes. Please try to reiserfsck your filesystem. Latest reiserfsck can be obtained from ftp://ftp.namesys.com/pub/reiserfsprogs/reiserfsprogs-3.6.19.tar.gz.
Quoted 1 line
Thanks again,
Quoted 19 lines · Vladimir Saveliev
Hello
Quoted 14 lines · Sean
Update:
I found these events --and many similar-- in my log file.
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74887 0x0 SD] Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD] Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1 Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck? Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]
Quoted 10 lines
Quoted 4 lines
Does this shed any new light? Does it look like I might have a corrupted file system?
Yes. Please try to reiserfsck your filesystem. Latest reiserfsck can be obtained from ftp://ftp.namesys.com/pub/reiserfsprogs/reiserfsprogs-3.6.19.tar.gz.
Vladimir,
Thank you much for your response. This is a production email server so I had to wait for an appropriate time to shut the system down.
Just a quick summary of what happened. After rebooting the sever and booting from a CD, I ran reiserfsck, which found no corruption in the file system. I then manually checked the directory structure and everything looked fine. I'm not what was wrong, but a reboot apparently corrected whatever it was.
I've noticed a couple of patches for reiserfs in the past month or so on the LKML. I'll check after I send this message to see if I can find the answer for myself, but, are there any major reiserfs bug fixes related to the Linux kernel, which have been corrected since 2.6.9 that I should know about? Also, I'm using a 3ware 8000 series SATA controller, if that's at all relevant.
Thanks again for your help,
Signature · 1 line
Sean
On Wed, Dec 29, 2004 at 07:54:20AM -0500, Fao, Sean wrote:
...
Quoted 11 lines
Vladimir,
Thank you much for your response. This is a production email server so I had to wait for an appropriate time to shut the system down.
Just a quick summary of what happened. After rebooting the sever and booting from a CD, I ran reiserfsck, which found no corruption in the file system. I then manually checked the directory structure and everything looked fine. I'm not what was wrong, but a reboot apparently corrected whatever it was.
I've seen this on ext3 and XFS as well.
Both locally (ext3 and XFS) and NFS exported (XFS).
About to try out 2.6.10 with CVS XFS and patches from the list - but somehow I doubt that the actual problem is in the filesystems.
Signature · 1 line
/ jakob
Quoted 7 lines · Jakob Oestergaard
I've seen this on ext3 and XFS as well.
Both locally (ext3 and XFS) and NFS exported (XFS).
About to try out 2.6.10 with CVS XFS and patches from the list - but somehow I doubt that the actual problem is in the filesystems.
Part of my reason for panic was that I recently had my Linux router kick the bucket. I also used reiserfs on this system and *believe* it was running a 2.6.8 kernel.
The strange thing about this system was that, at first glance, it appeared to have a complete hard disk failure. However, when I rebooted the computer, the system did indeed start to boot. When it came time to mount the root file system, it became apparent that it had become corrupt. I've been running Linux systems for years and this was the first system that I ever experienced a corrupted file system on.
As a result of my success with file systems on Linux in the past, I've been somewhat lazy when partitioning drives lately. I've been creating 100 MB boot, system_ram * 2 swap, and throwing everything else in the root. I might have to go back to my old style to separate /, /home and /usr onto separate file systems.
Signature · 1 line
Sean