Skip to contentSkip to conversation

ext2fs corruption again

DISCUSSION Kristian 6 participants 7 messages Sep 15, 2001 – Sep 16, 2001 Block layer

lore
Share & tools
lore

Kristian

author
#0 lore raw

Hello.

For about 3 weeks I sent a report that I've got very strange kernel error messages.

I changed my harddrive to IBM 75 GB because someone said that IBM's 40 GB harddisks are not very stable.

Today I've got these from the kernel (with the new hd):

Sep 15 10:01:58 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4215
Sep 15 10:01:58 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4217
Sep 15 10:01:59 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4234
Sep 15 10:01:59 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4236
Sep 15 10:01:59 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4239
Sep 15 10:02:03 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4847
Sep 15 10:02:03 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4848
Sep 15 10:02:03 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4852
Sep 15 10:02:03 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4855
Sep 15 10:02:06 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_new_block: Allocating block in system zone - block = 174
Sep 15 10:02:06 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: Freeing blocks in system zones - Block = 179, count = 3
Sep 15 10:02:09 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4839

Then I did an e2fsck on that device (hda5) and the errors occured after the check (and a complete reboot) again:

Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4163
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4166
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4131
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4132
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4155
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4156
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4157
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4161
Sep 15 10:10:38 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 4162
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 716
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 717
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 720
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 723
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 724
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 725
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 726
Sep 15 10:10:43 adlib kernel: EXT2-fs error (device ide0(3,5)): ext2_free_blocks: bit already cleared for block 727

I've written down all e2fsck messages by hand. ;-) And I compared them.

The following messages from e2fsck are always the same even on the old and on the new hd. Here they are:

Duplicate/bad bock(s) in inode: 97: 643 Duplicate/bad bock(s) in inode: 100: 649 Duplicate/bad bock(s) in inode: 101: 650 651 Duplicate/bad bock(s) in inode: 102: 652 Duplicate/bad bock(s) in inode: 103: 653 656 Duplicate/bad bock(s) in inode: 104: 659 660 Duplicate/bad bock(s) in inode: 105: 661 662 663 664 665 666 Duplicate/bad bock(s) in inode: 106: 667 668 Duplicate/bad bock(s) in inode: 107: 669 671 Duplicate/bad bock(s) in inode: 108: 672 673 674 Duplicate/bad bock(s) in inode: 110: 678

Inodes 643-678 are always connected to faults.

The following files are always in connection with these errors:
/var/log/wtmp
/var/log/messages

The old hd was hda: IBM-DTLA-305040, ATA DISK drive. The new is: hda: IBM-DTLA-307075, ATA DISK drive.

hdparm says:
Model=IBM-DTLA-307075, FwRev=TXAOA50C, SerialNo=YSDYSFN9998 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=40 BuffType=DualPortCache, BuffSize=1916kB, MaxMultSect=16, MultSect=8 CurCHS=17475/15/63, CurSects=-78446341, LBA=yes, LBAsects=150136560 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2 udma0 udma1 *udma2
AdvancedPM=yes: disabled (255)
Drive Supports : ATA/ATAPI-5 T13 1321D revision 1 : ATA-2 ATA-3 ATA-4 ATA-5

I currently use linux 2.4.9 and e2fsprogs 1.23 and fileutils-4.1 and a modified RedHat 6.2. These errors only occured with linux>=2.4.5-ac11.

I might say this is definitely an error with ext2 !

Kristian

·· · · reach me :: · ·· ·· · · ·· · ·· · ··· · ·

                         :: http://www.korseby.net
                         :: http://www.tomlab.de

[email protected] ....::

#1 lore raw
Quoted 7 lines · Kristian

Hello.

For about 3 weeks I sent a report that I've got very strange kernel error messages.

I changed my harddrive to IBM 75 GB because someone said that IBM's 40 GB harddisks are not very stable.

Just to get it out of the way, can you open your computer and check what country the disk is manufactured in? There has been some complaints on this list about IBM-disks fabricated in Hungary.

/David Weinehall
_ _
// David Weinehall <[email protected]> /> Northern lights wander \\

//  Project MCA Linux hacker        //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </
#2 lore raw
In reply to Kristian
Quoted 6 lines · David Weinehall
Quoted 5 lines · Kristian

Hello.

For about 3 weeks I sent a report that I've got very strange kernel error messages.

Quoted 6 lines
Quoted 2 lines

I changed my harddrive to IBM 75 GB because someone said that IBM's 40 GB harddisks are not very stable.

Just to get it out of the way, can you open your computer and check what country the disk is manufactured in? There has been some complaints on this list about IBM-disks fabricated in Hungary.

It's not just the disks made in Hungary, I've had 3 IBM drives go bad on me in the last week after 3-4 months of operation 2 15GB 75GXPs made in Thailand (bad sectors), 1 40GB 40GV also made in Thailand (started making bad scratching noise, BIOS wouldn't detect it after that). Still have a number of the 75GXPs in service, but I'm keeping my eye on them.

Kristian's problem looks like it could be hardware problems of some sort leading to corruption.

-Dave

#3 lore raw
In reply to Kristian
Quoted 8 lines · David Rees

It's not just the disks made in Hungary, I've had 3 IBM drives go bad on me in the last week after 3-4 months of operation 2 15GB 75GXPs made in Thailand (bad sectors), 1 40GB 40GV also made in Thailand (started making bad scratching noise, BIOS wouldn't detect it after that). Still have a number of the 75GXPs in service, but I'm keeping my eye on them.

Kristian's problem looks like it could be hardware problems of some sort leading to corruption.

I think so. Someone send me a link where there was described that especially that 75 GB drives are causing such severe corruption.

But that drives seem to have these errors from the beginning. I just putted off the packaging yesterday of that 75 GB drive. Mostly that errors occured when the disk was totally off for a moment and only on my root-partition.

Is it possible to detect which file currently own a specific inode ?

Thanks anyway.

Kristian
·· · · reach me :: · ·· ·· · · ·· · ·· · ··· · ·

                         :: http://www.korseby.net
                         :: http://www.tomlab.de

[email protected] ....::

Signature · 4 lines
·· · · reach me :: · ·· ·· ·  · ·· · ··  · ··· · ·
                         :: http://www.korseby.net
                         :: http://www.tomlab.de
[email protected] ....::
#4 lore raw
In reply to Kristian

Hi!

Quoted 11 lines
Quoted 7 lines

Hello.

For about 3 weeks I sent a report that I've got very strange kernel error messages.

I changed my harddrive to IBM 75 GB because someone said that IBM's 40 GB harddisks are not very stable.

Just to get it out of the way, can you open your computer and check what country the disk is manufactured in? There has been some complaints on this list about IBM-disks fabricated in Hungary.

Install crc loop device, and if disk does silent errors, you'll know.
Pavel

drivers/block/loop.c +108 -3
@@ -69,6 +69,7 @@
 #include <linux/slab.h>
 
 #include <asm/uaccess.h>
+#include <asm/checksum.h>
 
 #include <linux/loop.h>		
 
@@ -107,7 +108,6 @@
 		in = loop_buf;
 		out = raw_buf;
 	}
-
 	key = lo->lo_encrypt_key;
 	keysize = lo->lo_encrypt_key_size;
 	for (i = 0; i < size; i++)
@@ -115,11 +115,106 @@
 	return 0;
 }
 
-static int none_status(struct loop_device *lo, struct loop_info *info)
+#define ID printk(KERN_ERR "crc: info about (%s, %d, %d) ", kdevname(lo->lo_device), real_block, blksize);
+
+
+static int transfer_crc(struct loop_device *lo, int cmd, char *raw_buf,
+		  char *loop_buf, int size, int real_block)
 {
+	struct buffer_head *bh;
+	int blksize = 1024, nsect;	/* Size of block on auxilary media */
+	int cksum;
+	u32 *data;
+	nsect = blksize / 4;
+
+	if (!lo->second_device) {
+		ID; printk( "reading from not-yet-setup crc device can result in armagedon. Dont try again.\n" );
+		return -1;
+	}
+	bh = getblk(lo->second_device, 1+real_block/nsect, blksize);
+	if (!bh) {
+		ID; printk( "getblk returned NULL.\n" );
+		return -1;
+	}
+	if (!buffer_uptodate(bh)) {
+		ll_rw_block(READ, 1, &bh);
+		wait_on_buffer(bh);
+		if (!buffer_uptodate(bh)) {
+			ID; printk(  "could not read block with CRC\n" );
+			goto error;
+		}
+	}
+
+	data = (u32 *) bh->b_data;
+	if (cmd == READ)
+		cksum = csum_partial_copy_nocheck(raw_buf, loop_buf, size, 0);
+	else
+		cksum = csum_partial_copy_nocheck(loop_buf, raw_buf, size, 0);
+
+	if (cmd == READ) {
+		if (le32_to_cpu(data[real_block%nsect]) != cksum) {
+			if (lo->lo_encrypt_key_size == 0) {	/* Normal mode */
+				ID; printk( "wrong checksum reading, is %x, should be %x\n", cksum, 0x1234 );
+				goto error;
+			} else { 
+				ID; printk( "wrong checksum repairing, setting to %x\n", cksum );
+				goto repair;
+			}
+		}
+	} else {
+	repair:
+		data[real_block%nsect] = cpu_to_le32(cksum);
+		mark_buffer_uptodate(bh, 1);
+		mark_buffer_dirty(bh);
+	}
+
+	brelse(bh);
 	return 0;
+error:
+	brelse(bh);
+	return -1;
+
 }
 
+static int ioctl_crc(struct loop_device *lo, int cmd, unsigned long arg)
+{
+	struct file	*file;
+	struct inode	*inode;
+	int error;
+
+	printk( "Entering ioctl_crc\n" );
+	if (cmd != LOOP_CRC_SET_FD)
+		return -EINVAL;
+
+	error = -EBADF;
+	file = fget(arg);
+	if (!file)
+		return -EINVAL;
+
+	error = -EINVAL;
+	inode = file->f_dentry->d_inode;
+	if (!inode) {
+		printk(KERN_ERR "ioctl_crc: NULL inode?!?\n");
+		goto out;
+	}
+
+	if (S_ISBLK(inode->i_mode)) {
+		error = blkdev_open(inode, file);
+		lo->second_device = inode->i_rdev;
+		printk( "loop_crc: Registered device %x\n", lo->second_device );
+		return error;
+	} else {
+	out:
+		fput(file);
+		return -EINVAL;
+	}
+}
+
+static int none_status(struct loop_device *lo, struct loop_info *info)
+{
+	return 0;
+} 
+
 static int xor_status(struct loop_device *lo, struct loop_info *info)
 {
 	if (info->lo_encrypt_key_size <= 0)
@@ -139,10 +234,19 @@
 	init: xor_status
 }; 	
 
+struct loop_func_table crc_funcs = { 
+	number: LO_CRYPT_CRC,
+	transfer: transfer_crc,
+	init: none_status,
+	ioctl: ioctl_crc
+}; 	
+
 /* xfer_funcs[0] is special - its release function is never called */ 
 struct loop_func_table *xfer_funcs[MAX_LO_CRYPT] = {
 	&none_funcs,
-	&xor_funcs  
+	&xor_funcs,
+	NULL, NULL, NULL, NULL, NULL,
+	&crc_funcs,
 };
 
 #define MAX_DISK_SIZE 1024*1024*1024
@@ -728,6 +832,7 @@
 	lo->transfer = NULL;
 	lo->ioctl = NULL;
 	lo->lo_device = 0;
+	lo->second_device = 0;
 	lo->lo_encrypt_type = 0;
 	lo->lo_offset = 0;
 	lo->lo_encrypt_key_size = 0;
include/linux/loop.h +3 -0
@@ -28,6 +28,7 @@
 	int		lo_number;
 	int		lo_refcnt;
 	kdev_t		lo_device;
+	kdev_t		second_device;
 	int		lo_offset;
 	int		lo_encrypt_type;
 	int		lo_encrypt_key_size;
@@ -119,6 +120,7 @@
 #define LO_CRYPT_BLOW     4
 #define LO_CRYPT_CAST128  5
 #define LO_CRYPT_IDEA     6
+#define LO_CRYPT_CRC	  7
 #define LO_CRYPT_DUMMY    9
 #define LO_CRYPT_SKIPJACK 10
 #define MAX_LO_CRYPT	20
@@ -150,5 +152,6 @@
 #define LOOP_CLR_FD	0x4C01
 #define LOOP_SET_STATUS	0x4C02
 #define LOOP_GET_STATUS	0x4C03
+#define LOOP_CRC_SET_FD 0x4C04
 
 #endif
Signature · 2 lines
I'm [email protected]. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [email protected]
#5 lore raw
In reply to Kristian
Quoted 1 line · Pavel Machek

Install crc loop device, and if disk does silent errors, you'll know.

Where do you store the CRCs? It appears that they are written to another block device. Also, how do you initialize the CRC table for an existing filesystem?

What would make this considerably more useful is to be able to write the CRCs into a regular file, as it would be a bit of a pain to have a partition for each CRC loop device to store the CRCs in.

Otherwise, it looks very useful, and could be handy in tracking down reports like this where it is unclear where the data corruption is.

Cheers, Andreas

Signature · 3 lines
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
#6 lore raw
In reply to Kristian

Hi!

Quoted 5 lines
Quoted 1 line

Install crc loop device, and if disk does silent errors, you'll know.

Where do you store the CRCs? It appears that they are written to another block device. Also, how do you initialize the CRC table for an existing filesystem?

You just read the device in special mode. That forces it to recompute crc-s.

Quoted 3 lines

What would make this considerably more useful is to be able to write the CRCs into a regular file, as it would be a bit of a pain to have a partition for each CRC loop device to store the CRCs in.

By -o loop, you can turn regular file into blockdevice ;-).

Quoted 2 lines

Otherwise, it looks very useful, and could be handy in tracking down reports like this where it is unclear where the data corruption is.

Pavel

Signature · 2 lines
I'm [email protected]. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [email protected]