I love Linux!

OT Robert de Bath 4 participants 5 messages May 24, 2003 – May 27, 2003

Share & tools

Robert de Bath

author

2003-05-24 08:40 UTC #0 lore raw

I thought you might be interested in this scrollback log, the complete one is approx 17000 lines; I've no idea what caused the original problem; hopefully something like a cosmic ray hitting a scsi transceiver.

Quoted 11 lines

W1(robert)/$ ls -l /disk/sda4
total 184881842
drwxr-xr-x 4 root root 1024 May 11 18:44 boot
dr--rwSrwT 15848 2962804722 2850042371 995903248 Jan 20 1959 lost+found W1(robert)/$ sudo umount /disk/sda4
W1(robert)/$ sudo e2fsck /dev/sda4
e2fsck 1.27 (8-Mar-2002)
Group descriptors look bad... trying backup blocks... Superblock has a bad ext3 journal (inode 8). Clear<y>? yes

Hmmm.

Quoted 8 lines

*** ext3 journal has been deleted - filesystem is now ext2 only ***
/dev/sda4 was not cleanly unmounted, check forced.
e2fsck: Illegal triply indirect block found while reading bad blocks inode This doesn't bode well, but we'll try to go on...
Pass 1: Checking inodes, blocks, and sizes
Bad block inode has illegal block(s). Clear<y>? yes

OMG!

Quoted 18 lines

Illegal block #0 (1431276975) in bad block inode. CLEARED.
Illegal block #1 (673818087) in bad block inode.  CLEARED.
Illegal block #2 (707372471) in bad block inode.  CLEARED.
Illegal block #3 (608784174) in bad block inode.  CLEARED.
Illegal block #4 (2312398709) in bad block inode.  CLEARED.
Illegal block #5 (1548877642) in bad block inode.  CLEARED.
Illegal block #6 (1635558748) in bad block inode.  CLEARED.
Illegal block #7 (698739426) in bad block inode. CLEARED. Illegal block #8 (1244206925) in bad block inode. CLEARED. Illegal block #9 (3562494532) in bad block inode. CLEARED. Illegal block #10 (1391372882) in bad block inode. CLEARED. Illegal block #11 (1253673108) in bad block inode. CLEARED. Illegal block #-1 (252663295) in bad block inode. CLEARED. Illegal block #-2 (575818347) in bad block inode. CLEARED. Illegal block #-3 (2336762341) in bad block inode. CLEARED. Reserved inode 4 <The ACL data inode> has bad mode. Clear<y>? yes

Quoted 2 lines

Reserved inode 6 <The undelete directory inode> has bad mode. Clear<y>? yes

Wimper wimper

Quoted 28 lines

Illegal block #4 (704842913) in inode 1951. CLEARED.
Illegal block #5 (2746145280) in inode 1951.  CLEARED.
Illegal block #6 (1448955905) in inode 1951.  CLEARED.
Illegal block #8 (3877244673) in inode 1951.  CLEARED.
Illegal block #9 (3309966792) in inode 1951.  CLEARED.
Illegal block #10 (1437152980) in inode 1951. CLEARED. Illegal block #11 (1498273102) in inode 1951. CLEARED. Too many illegal blocks in inode 1951.
Clear inode? yes
Restarting e2fsck from the beginning... /dev/sda4 was not cleanly unmounted, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure
i_faddr for inode 2 (/) is 2184181835, should be zero. Clear? yes
i_frag for inode 2 (/) is 72, should be zero.
Clear? yes
i_fsize for inode 2 (/) is 46, should be zero.
Clear? yes
Entry 'lost+found' in / (2) has deleted/unused inode 11. Clear? yes
Pass 3: Checking directory connectivity
/lost+found not found. Create? yes

Quoted 8 lines

Pass 4: Checking reference counts
Inode 2 ref count is 5, should be 4. Fix? yes
Inode 12 (...) has a bad mode (056231).
Clear? yes
Inode 13 (...) has a bad mode (075610).
Clear? yes

Inode 2! Inode 12! Oh well there goes the data.

Quoted 28 lines

Illegal block #4 (704842913) in inode 1951. CLEARED.
Illegal block #5 (2746145280) in inode 1951.  CLEARED.
Illegal block #6 (1448955905) in inode 1951.  CLEARED.
Illegal block #8 (3877244673) in inode 1951.  CLEARED.
Illegal block #9 (3309966792) in inode 1951.  CLEARED.
Illegal block #10 (1437152980) in inode 1951. CLEARED. Illegal block #11 (1498273102) in inode 1951. CLEARED. Too many illegal blocks in inode 1951.
Clear inode? yes
Restarting e2fsck from the beginning... /dev/sda4 was not cleanly unmounted, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure
i_faddr for inode 2 (/) is 2184181835, should be zero. Clear? yes
i_frag for inode 2 (/) is 72, should be zero.
Clear? yes
i_fsize for inode 2 (/) is 46, should be zero.
Clear? yes
Entry 'lost+found' in / (2) has deleted/unused inode 11. Clear? yes
Pass 3: Checking directory connectivity
/lost+found not found. Create? yes

Quoted 8 lines

Pass 4: Checking reference counts
Inode 2 ref count is 5, should be 4. Fix? yes
Inode 12 (...) has a bad mode (056231).
Clear? yes
Inode 13 (...) has a bad mode (075610).
Clear? yet

Cringe!

...
17000 lines from the top.

Quoted 1 line

/dev/sda4: ***** FILE SYSTEM WAS MODIFIED *****

No shit!

Quoted 9 lines

/dev/sda4: 58/10040 files (12.1% non-contiguous), 15670/40131 blocks W1(robert)/$ sudo e2fsck -y -f /dev/sda4
e2fsck 1.27 (8-Mar-2002)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda4: 58/10040 files (12.1% non-contiguous), 15670/40131 blocks

Hey! 15Mb! That doesn't look bad.

Quoted 3 lines

W1(robert)/$ sudo mount /dev/sda4 -r
W1(robert)/$ ls /disk/sda4
boot lost+found

Oh my.

Quoted 34 lines

W1(robert)/$ cd /disk/sda4
W1(robert)sda4$ ls -l
total 2

drwxr-xr-x    4 root     root         1024 May 11 18:44 boot
drwxr-xr-x    2 root     root         1024 May 24 08:29 lost+found

W1(robert)sda4$ cd boot
W1(robert)boot$ ls -l
total 13289

(quoted diff) +0 -7

-rw-r--r--    1 root     root       221502 Jun 25  2001 System.map-2.2.17
-rw-r--r--    1 root     root       397485 Apr 14  2002 System.map-2.4.18-k6
-rw-r--r--    1 root     root       437466 May  7 00:23 System.map-2.4.20
-rw-r--r--    1 root     root       459497 May 11 00:48 System.map-2.4.20-05102308
-rw-r--r--    1 root     root       424536 Mar 25 10:05 System.map-2.4.20-1-k6
-rw-r--r--    1 root     root          512 Dec 10  2000 boot.0300
-rw-r--r--    1 root     root          512 Sep 24  2000 boot.0800

drwxr-xr-x 2 root root 1024 Sep 15 2002 bootdisk

(quoted diff) +0 -7

-rw-r--r--    1 root     root        37036 Apr 14  2002 config-2.4.18-k6
-rw-r--r--    1 root     root        40107 May  6 23:23 config-2.4.20
-rw-r--r--    1 root     root        40409 May 10 23:11 config-2.4.20-05102308
-rw-r--r--    1 root     root        39538 Mar 22 03:36 config-2.4.20-1-k6
-rw-r--r--    1 root     root      2736128 Sep 15  2002 initrd.img-2.4.18-k6
-rw-r--r--    1 root     root      2871296 May  7 07:22 initrd.img-2.4.20
-rw-r--r--    1 root     root      2887680 May  9 22:04 initrd.img-2.4.20-1-k6

lrwxrwxrwx 1 root root 17 May 11 18:44 initrd.img.old -> initrd.img-2.4.20

(quoted diff) +0 -2

-rw-------    1 root     root       163328 May 11 18:44 map
-rwxrwxr-x    1 root     root          599 May 10 22:32 mklink

drwxr-xr-x 2 root root 1024 Sep 15 2002 old

(quoted diff) +0 -2

-rw-r--r--    1 root     root           20 May  7 00:23 patches-2.4.20
-rw-r--r--    1 root     root           25 May 11 00:48 patches-2.4.20-05102308

lrwxrwxrwx 1 root root 23 May 11 18:44 vmlinuz -> vmlinuz-2.4.20-05102308

(quoted diff) +0 -4

-rw-r--r--    1 root     root       614313 Apr 14  2002 vmlinuz-2.4.18-k6
-rw-r--r--    1 root     root       722227 May  7 00:23 vmlinuz-2.4.20
-rw-r--r--    1 root     root       790421 May 11 00:48 vmlinuz-2.4.20-05102308
-rw-r--r--    1 root     root       638232 Mar 25 10:05 vmlinuz-2.4.20-1-k6

Quoted 2 lines

lrwxrwxrwx 1 root root 14 May 11 18:44 vmlinuz-rd.old -> vmlinuz-2.4.20
W1(robert)boot$

It's all there! Could this possibly mean ...

FX: diff, lilo, reboot, rumble, beep, rumble

Quoted 2 lines

Debian GNU/Linux 3.0 mayday.local
mayday login: _

Wow! I love Linux!

Signature · 3 lines

Rob.                          (Robert de Bath <robert$ @ debath.co.uk>)
                                       <http://www.cix.co.uk/~mayday>
Google Homepage:   http://www.google.com/search?btnI&q=Robert+de+Bath

Daniel Phillips

2003-05-25 01:39 UTC #1 lore raw

Quoted 8 lines · Robert de Bath

It's all there! Could this possibly mean ...
FX: diff, lilo, reboot, rumble, beep, rumble
Quoted 2 lines
Debian GNU/Linux 3.0 mayday.local
mayday login: _
Wow! I love Linux!

You have Ted Ts'o to thank for that, and various other artists to be sure, but Ted's e2fsck is what got your data back.

Regards,

Daniel

Ryan Anderson

2003-05-25 08:49 UTC #2 lore raw

In reply to Robert de Bath

Quoted 1 line · Robert de Bath

Quoted 1 line
*** ext3 journal has been deleted - filesystem is now ext2 only ***

Did you ever put the journal back on?

[...]

Quoted 1 line

Wow! I love Linux!

I do too - but after watching that happen, I'd be checking my backups. :) That's a lot of luck to burn through in one day.

Signature · 2 lines

Ryan Anderson
  sometimes Pug Majere

Robert de Bath

author

2003-05-25 19:34 UTC #3 lore raw

Quoted 4 lines · Ryan Anderson

Quoted 1 line · Robert de Bath
Quoted 1 line
*** ext3 journal has been deleted - filesystem is now ext2 only ***
Did you ever put the journal back on?

No actually I haven't, (by choice) the FS is now read-only too as it's "only" /boot.

Quoted 6 lines

[...]
Quoted 1 line
Wow! I love Linux!
I do too - but after watching that happen, I'd be checking my backups. :) That's a lot of luck to burn through in one day.

Actually I didn't believe it at first, so I recovered the files from backup before I risked a reboot; but when they compared the same as the backup I tried it anyway. I've also checked all the other filesystems and done a fresh (verified!) L0 backup. :)

Anyway I might have balanced out a bit on the luck front; I had one of those 'Oh shit' moments with a 'doze system a couple of days ago. I thought I was wiping the start of a virtual (vmware) disk ...

Signature · 3 lines

Rob.                          (Robert de Bath <robert$ @ debath.co.uk>)
                                       <http://www.cix.co.uk/~mayday>
Google Homepage:   http://www.google.com/search?btnI&q=Robert+de+Bath

Eric

2003-05-27 03:20 UTC #4 lore raw

In reply to Robert de Bath

Quoted 5 lines · Robert de Bath

Quoted 3 lines
Debian GNU/Linux 3.0 mayday.local
mayday login: _
Wow! I love Linux!

Once I acidentally ran e2fsck on a swap partition. It was on a reboot I think. Scared the crap out of me as it found about a billion things wrong with the structures and such. Don't ask me how it didn't complain...but I almost wet myself anf cried when every block had something wrong with it.

Imagine my relief when I realized it was just my swap partition!

----------------------
Eric Bambach
[email protected]
----------------------