Skip to contentSkip to conversation

Filesystem/kernel bug?

DISCUSSION Fao, Sean 3 participants 6 messages Dec 27, 2004 – Dec 29, 2004

lore
Share & tools
lore

Fao, Sean

author
#0 lore raw

I was attempting to perform an "emerge --sync" as root on my Gentoo system to update the portage tree (the actual meaning of this command is irrelevant, so don't worry if you're not familiar with it) and had it fail while attempting to change directories. I found this to be quite strange, as I was running the synchronization as root. The directory in question was, "/usr/portage/mail-mta/msmtp". Sure enough, manually changing directories failed with a "permission denied". I then CD'ed to "/usr/portage/mail-mta" and executed a "ls -la". To my surprise, there was no directory named "msmtp" in "/usr/portage/mail-mta". I then tried to change to a random directory name, to verify that the error message wasn't just generic. "cd foo" returned "No such file or directory", as expected.

Following is basically an outline of the steps I performed, which lead to my confusion:

cgx-mail / # cd /usr/portage/mail-mta/

cgx-mail mail-mta # ls -la
ls: msmtp: Permission denied
total 4
drwxr-xr-x 16 root root 416 Dec 27 09:47 .
drwxr-xr-x 135 root root 4008 Dec 27 09:48 ..

drwxr-xr-x    3 root root  336 Dec 15 17:09 courier
drwxr-xr-x    3 root root  200 Nov  1 16:35 esmtp
drwxr-xr-x    3 root root  400 Dec 17 16:06 exim
drwxr-xr-x    3 root root  200 Sep  7 07:10 mini-qmail
drwxr-xr-x    3 root root  400 Dec 22 08:45 nbsmtp
drwxr-xr-x    3 root root  400 Nov 16 04:38 nullmailer
drwxr-xr-x    3 root root  328 Dec 23 01:37 postfix
drwxr-xr-x    3 root root  480 Dec 15 17:37 qmail
drwxr-xr-x    3 root root  352 Jul 20 09:37 qmail-ldap
drwxr-xr-x    3 root root  248 Nov 24 17:07 qmail-mysql
drwxr-xr-x    3 root root  296 Dec 21 09:11 sendmail
drwxr-xr-x    3 root root  240 Dec 25 16:07 ssmtp
drwxr-xr-x    3 root root  280 Dec  6 10:16 xmail

cgx-mail mail-mta # cd msmtp
-bash: cd: msmtp: Permission denied

cgx-mail mail-mta # cd foo
-bash: cd: foo: No such file or directory

cgx-mail mail-mta # uname -r
2.6.9-gentoo-r9

cgx-mail mail-mta # cat /etc/fstab

/dev/sda1               /boot        reiserfs  noatime,notail        1 1
/dev/sda3               /            reiserfs  noatime               0 0
/dev/sda2               none         swap      sw                    0 0
/dev/fd0                /mnt/floppy  auto      noauto                0 0
none                    /proc        proc      defaults              0 0

At this point, I'm not really sure what's going on.

Can anybody help me out? I've posted any information I thought would be relevant. If you need anything else, feel free to let me know.

Thank you,

Signature · 1 line
Sean
author
#1 lore raw

Update:

I found these events --and many similar-- in my log file.

Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74887 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]

Does this shed any new light? Does it look like I might have a corrupted file system?

Thanks again,

Signature · 1 line
Sean
#2 lore raw

Hello

Quoted 23 lines · Sean

Update:

I found these events --and many similar-- in my log file.

Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
data of [30749 74887 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
reiserfs_read_locked_inode: i/o failure occurred trying to find stat 
data of [30749 74888 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 
does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: 
search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: 
Quoted 6 lines

reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]

Does this shed any new light? Does it look like I might have a corrupted file system?

Yes. Please try to reiserfsck your filesystem. Latest reiserfsck can be obtained from ftp://ftp.namesys.com/pub/reiserfsprogs/reiserfsprogs-3.6.19.tar.gz.

Quoted 1 line

Thanks again,

author
#3 lore raw
Quoted 19 lines · Vladimir Saveliev

Hello

Quoted 14 lines · Sean

Update:

I found these events --and many similar-- in my log file.

Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74887 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]
Dec 26 17:55:43 cgx-mail ReiserFS: warning: is_tree_node: node level 0 does not match to the expected one 1
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-5150: search_by_key: invalid format found in block 13706028. Fsck?
Dec 26 17:55:43 cgx-mail ReiserFS: sda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [30749 74888 0x0 SD]
Quoted 10 lines
Quoted 4 lines

Does this shed any new light? Does it look like I might have a corrupted file system?

Yes. Please try to reiserfsck your filesystem. Latest reiserfsck can be obtained from ftp://ftp.namesys.com/pub/reiserfsprogs/reiserfsprogs-3.6.19.tar.gz.

Vladimir,

Thank you much for your response. This is a production email server so I had to wait for an appropriate time to shut the system down.

Just a quick summary of what happened. After rebooting the sever and booting from a CD, I ran reiserfsck, which found no corruption in the file system. I then manually checked the directory structure and everything looked fine. I'm not what was wrong, but a reboot apparently corrected whatever it was.

I've noticed a couple of patches for reiserfs in the past month or so on the LKML. I'll check after I send this message to see if I can find the answer for myself, but, are there any major reiserfs bug fixes related to the Linux kernel, which have been corrected since 2.6.9 that I should know about? Also, I'm using a 3ware 8000 series SATA controller, if that's at all relevant.

Thanks again for your help,

Signature · 1 line
Sean
#4 lore raw

On Wed, Dec 29, 2004 at 07:54:20AM -0500, Fao, Sean wrote:
...

Quoted 11 lines

Vladimir,

Thank you much for your response. This is a production email server so I had to wait for an appropriate time to shut the system down.

Just a quick summary of what happened. After rebooting the sever and booting from a CD, I ran reiserfsck, which found no corruption in the file system. I then manually checked the directory structure and everything looked fine. I'm not what was wrong, but a reboot apparently corrected whatever it was.

I've seen this on ext3 and XFS as well.

Both locally (ext3 and XFS) and NFS exported (XFS).

About to try out 2.6.10 with CVS XFS and patches from the list - but somehow I doubt that the actual problem is in the filesystems.

Signature · 1 line
 / jakob
author
#5 lore raw
Quoted 7 lines · Jakob Oestergaard

I've seen this on ext3 and XFS as well.

Both locally (ext3 and XFS) and NFS exported (XFS).

About to try out 2.6.10 with CVS XFS and patches from the list - but somehow I doubt that the actual problem is in the filesystems.

Part of my reason for panic was that I recently had my Linux router kick the bucket. I also used reiserfs on this system and *believe* it was running a 2.6.8 kernel.

The strange thing about this system was that, at first glance, it appeared to have a complete hard disk failure. However, when I rebooted the computer, the system did indeed start to boot. When it came time to mount the root file system, it became apparent that it had become corrupt. I've been running Linux systems for years and this was the first system that I ever experienced a corrupted file system on.

As a result of my success with file systems on Linux in the past, I've been somewhat lazy when partitioning drives lately. I've been creating 100 MB boot, system_ram * 2 swap, and throwing everything else in the root. I might have to go back to my old style to separate /, /home and /usr onto separate file systems.

Signature · 1 line
Sean