Skip to contentSkip to conversation

ext3 file system went read-only in 2.6.18.2 (plus hacks)

DISCUSSION Ben Greear 3 participants 5 messages Jan 8, 2007

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

First, this kernel is tainted by one of my proprietary patches and also has my other non-proprietary patches applied, so please ignore as you wish. My patches do not (purposefully) affect the file-system.

The system is a via/c3 system running a fairly stripped down FC5 installation on a 1GB CF disk. We've been using this platform for over a year and this is the first file system error we've seen. It could easily be that the CF is funky, but I thought I'd post the message in case it proves to be something else.

eth1: Promiscuous mode enabled.
eth1: Promiscuous mode enabled.
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99247: rec_len % 4 1 Aborting journal on device hda1.
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99254: rec_len % 4 1 ext3_abort called.
EXT3-fs error (device hda1): ext3_journal_start_sb: Detected aborted journal Remounting filesystem read-only
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99048: rec_len % 4 1 EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99041: directory en1
syslogd: /var/log/messages: Read-only file system
syslogd: /var/log/secure: Read-only file system
syslogd: /var/log/maillog: Read-only file system
syslogd: /var/log/cron: Read-only file system
syslogd: /var/log/spooler: Read-only file system
syslogd: /var/log/boot.log: Read-only file system
syslogd: /var/log/messages: Read-only file system
syslogd: /var/log/secure: Read-only file system
syslogd: /var/log/maillog: Read-only file system
syslogd: /var/log/cron: Read-only file system
syslogd: /var/log/spooler: Read-only file system
syslogd: /var/log/boot.log: Read-only file system
pktgen: pktgen_mark_device marking eth0#5 for removal
pktgen: pktgen_mark_device marking eth0#0 for removal
.....

After restarting and a manual fsck, the system appears to be back to normal.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#1 lore raw
Quoted 6 lines

pktgen: pktgen_mark_device marking eth0#5 for removal
pktgen: pktgen_mark_device marking eth0#0 for removal
.....

After restarting and a manual fsck, the system appears to be back to normal.

Did the fsck find any errors ?

author
#2 lore raw
Quoted 8 lines · Alan
Quoted 6 lines

pktgen: pktgen_mark_device marking eth0#5 for removal
pktgen: pktgen_mark_device marking eth0#0 for removal
.....

After restarting and a manual fsck, the system appears to be back to normal.

Did the fsck find any errors ?

Yes. They are below:

Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/hda1
CT_ROOT contains a file system with errors, check forced.
CT_ROOT: Directory inode 99041, block 0, offset 0: directory corrupted

CT_ROOT: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
[FAILED]

*** An error occurred during the file system check. *** Dropping you to a shell; the system will reboot *** when you leave the shell.
Give root password for maintenance
(or type Control-D to continue):
(Repair filesystem) 1 # fsck.ext3

anaconda-ks.cfg     .gconf/             .serverauth.7145
.bash_history       .gconfd/            .ssh/
.bash_logout        GNUstep/            .tcshrc
.bash_profile       .gtkrc              .vnc/
.bashrc             install.log.syslog  .Xauthority
.cshrc              .lesshst            .xauths0kLo8
.ethereal/          .serverauth.1869    .xinitrc

(Repair filesystem) 1 # fsck.ext3 -a -y /dev/hda1
e2fsck: Only one the options -p/-a, -n or -y may be specified. (Repair filesystem) 2 # fsck.ext3 -y /dev/hda1 e2fsck 1.38 (30-Jun-2005)
CT_ROOT contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 99041, block 0, offset 0: directory corrupted Salvage? yes

First entry 'M-?' (inode=124909) in directory inode 99041 (???) should be '.' Fix? yes

Setting filetype for entry '.' in ??? (99041) to 2. Missing '..' in directory inode 99041.
Fix? yes

Setting filetype for entry '..' in ??? (99041) to 2. Directory inode 99048, block 0, offset 0: directory corrupted Salvage? yes

First entry 'M-?' (inode=129789) in directory inode 99048 (???) should be '.' Fix? yes

Setting filetype for entry '.' in ??? (99048) to 2. Missing '..' in directory inode 99048.
Fix? yes

Setting filetype for entry '..' in ??? (99048) to 2. Directory inode 99247, block 0, offset 0: directory corrupted Salvage? yes

First entry 'M-?' (inode=102335) in directory inode 99247 (???) should be '.' Fix? yes

Setting filetype for entry '.' in ??? (99247) to 2. Missing '..' in directory inode 99247.
Fix? yes

Setting filetype for entry '..' in ??? (99247) to 2. Directory inode 99254, block 0, offset 0: directory corrupted Salvage? yes

First entry 'M-?' (inode=112567) in directory inode 99254 (???) should be '.' Fix? yes

Setting filetype for entry '.' in ??? (99254) to 2. Missing '..' in directory inode 99254.
Fix? yes

Setting filetype for entry '..' in ??? (99254) to 2.
Pass 3: Checking directory connectivity
'..' in /lib/modules/2.6.18.2c3/kernel/drivers/mtd/maps (99041) is <The NULL inode> (0), .Fix? yes

'..' in /lib/modules/2.6.18.2c3/kernel/drivers/mtd/chips (99048) is <The NULL inode> (0),.Fix? yes

'..' in /home/lanforge/DB/day_352 (99247) is <The NULL inode> (0), should be /home/lanfor.Fix? yes

'..' in /home/lanforge/DB/day_353 (99254) is <The NULL inode> (0), should be /home/lanfor.Fix? yes

Pass 4: Checking reference counts
Inode 2 ref count is 19, should be 23. Fix? yes

Inode 63506 ref count is 1, should be 2. Fix? yes

Inode 95836 ref count is 58, should be 56. Fix? yes

Unattached inode 99004
Connect to /lost+found? yes

Inode 99004 ref count is 2, should be 1. Fix? yes

Unattached inode 99005
Connect to /lost+found? yes

Inode 99005 ref count is 2, should be 1. Fix? yes

Unattached inode 99006
Connect to /lost+found? yes

Inode 99006 ref count is 2, should be 1. Fix? yes

Unattached inode 99007
Connect to /lost+found? yes

Inode 99007 ref count is 2, should be 1. Fix? yes

Unattached inode 99008
Connect to /lost+found? yes

Inode 99008 ref count is 2, should be 1. Fix? yes

Unattached inode 99009
Connect to /lost+found? yes

Inode 99009 ref count is 2, should be 1. Fix? yes

Unattached inode 99010
Connect to /lost+found? yes

Inode 99010 ref count is 2, should be 1. Fix? yes

Unattached inode 99011
Connect to /lost+found? yes

Inode 99011 ref count is 2, should be 1. Fix? yes

Unattached inode 99012
Connect to /lost+found? yes

Inode 99012 ref count is 2, should be 1. Fix? yes

Unattached inode 99013
Connect to /lost+found? yes

Inode 99013 ref count is 2, should be 1. Fix? yes

Unattached inode 99014
Connect to /lost+found? yes

Inode 99014 ref count is 2, should be 1. Fix? yes

Unattached inode 99016
Connect to /lost+found? yes

Inode 99016 ref count is 2, should be 1. Fix? yes

Unattached inode 99017
Connect to /lost+found? yes

Inode 99017 ref count is 2, should be 1. Fix? yes

Unattached inode 99019
Connect to /lost+found? yes

Inode 99019 ref count is 2, should be 1. Fix? yes

Inode 99020 ref count is 8, should be 6. Fix? yes

Unattached inode 99021
Connect to /lost+found? yes

Inode 99021 ref count is 2, should be 1. Fix? yes

Unattached inode 99022
Connect to /lost+found? yes

Inode 99022 ref count is 2, should be 1. Fix? yes

Unattached inode 99023
Connect to /lost+found? yes

Inode 99023 ref count is 2, should be 1. Fix? yes

Unattached inode 99248
Connect to /lost+found? yes

Inode 99248 ref count is 2, should be 1. Fix? yes

Unattached inode 99249
Connect to /lost+found? yes

Inode 99249 ref count is 2, should be 1. Fix? yes

Unattached inode 99250
Connect to /lost+found? yes

Inode 99250 ref count is 2, should be 1. Fix? yes

Unattached inode 99251
Connect to /lost+found? yes

Inode 99251 ref count is 2, should be 1. Fix? yes

Unattached inode 99252
Connect to /lost+found? yes

Inode 99252 ref count is 2, should be 1. Fix? yes

Unattached inode 99253
Connect to /lost+found? yes

Inode 99253 ref count is 2, should be 1. Fix? yes

Unattached inode 99255
Connect to /lost+found? yes

Inode 99255 ref count is 2, should be 1. Fix? yes

Unattached inode 99256
Connect to /lost+found? yes

Inode 99256 ref count is 2, should be 1. Fix? yes

Unattached inode 99257
Connect to /lost+found? yes

Inode 99257 ref count is 2, should be 1. Fix? yes

Unattached inode 99258
Connect to /lost+found? yes

Inode 99258 ref count is 2, should be 1. Fix? yes

Unattached inode 99259
Connect to /lost+found? yes

Inode 99259 ref count is 2, should be 1. Fix? yes

Unattached inode 99260
Connect to /lost+found? yes

Inode 99260 ref count is 2, should be 1. Fix? yes

Pass 5: Checking group summary information
Block bitmap differences: +67611 -(79872--79873) -(80952--81007) -(81016--81023) -(810271Fix? yes

CT_ROOT: ***** FILE SYSTEM WAS MODIFIED *****
CT_ROOT: ***** REBOOT LINUX *****
CT_ROOT: 38495/253184 files (3.5% non-contiguous), 227976/253015 blocks (Repair filesystem) 3 # reboot

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#3 lore raw

Well, the filesystem obviously got corrupted. The only question is *why* it got corrupted. It could be memory corruption, or a bug in your propietary kernel patches, or some kind of hardware issue with the CF device. There's really no way to say for sure.

Were there any error messages in the system log from the device driver?

- Ted

author
#4 lore raw
Quoted 4 lines · Theodore Tso

Well, the filesystem obviously got corrupted. The only question is *why* it got corrupted. It could be memory corruption, or a bug in your propietary kernel patches, or some kind of hardware issue with the CF device. There's really no way to say for sure.

Ok. I certainly can't guarantee my code is not somehow causing the problem.

Quoted 2 lines

Were there any error messages in the system log from the device driver?

I looked and did not see anything obvious. The file system was not overly full (49MB free, not counting the reserved space for the root user.)

I appreciate you looking at the report. If we see it again or manage to find some way to reliably reproduce this on other hardware, I will of course let you know.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com