uncorrectable ext2 errors

Hi!

A few days ago, I reported some problems with a ext2 filesystem which I cannot repair with e2fsck. Now I got some new observations.

To summarize the problem: e2fsck reports block bitmap differences, but telling it to repair these doesn't help, another e2fsck run reports the same differences.

Now I took a copy of the metadata with e2image -r, and ran e2fsck on that. Surprisingly, now e2fsck was able to repair the differences.

With this corrected filesystem, I took a look at what e2fsck did change on the filesystem. Interestingly, the errors are very systematic. It's mainly sequences of 4096 bytes which should be all 255, but are 3,0,0,0,300,7,0,... (rest are 0s). These blocks start at the following positions:

6630000001
6638000001
6640000001
6648000001
6748000001
6750000001
6758000001
6760000001
6768000001
6790000001
6798000001
67A0000001
67A8000001
67B0000001
67B8000001
67C0000001
67C8000001
67D0000001
67D8000001
67E0000001
67E8000001

To me, this looks like some strange problem in a layer below the filesystem. So it may be LVM, ide, promise ide driver, or a hardware fault.

Do you have any idea how to further diagnose this?

Jan

Is this the root filesystem that is being checked? If yes, then you probably need to either reboot after the fsck is complete (before mounting RW), or run the fsck from a rescue disk/CD.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

No, it's not the root filesystem, but I usually check it while it's mounted r/o, because the check takes nearly an hour and I want to give the users at least read access to their files.

But because I assumed that this may cause problems, I did try it with a completly unmonted filesystem once, without success.

Can you explain how a r/o mounted file system can cause problems? Perhaps there is still some connection to my problem.

Jan

No it shouldn't cause problems, only that you should umount the filesystem before you mount it read/write because the fsck might have changed something, and the kernel will have the old information still in memory, and might cause MORE corruption, than before.

So, just umount after fsck and you should be set (at least for that part)...

[...]

Small but important correction: make all these positions 6630000000
6638000000
[...]

I was unaware that cmp -l starts counting at 1 instead of 0...

Jan

Now I pvmoved one PE which contains areas of the filesystem with errors to a different drive. Surprise: e2fsck is now able to correct these errors. Block bitmap differences in other PEs are still not fixed.

This strengthens my assumption that this is not a e2fsck problem, but an LVM or IDE one.

Are there any known problems with huge hard disks (250GB) on Promise IDE?

Jan

Further investigating this, I found the following posting:
http://lkml.org/lkml/2002/9/6/22

And indeed, even with 2.4.21, /proc/ide/ide2/hdf/settings shows that 'address' is 0, meaning that LBA48 is turned of, if I understand this correctly. But that would mean that a 250GB hard disk should not work as expected, right? May that explain my disk corruption problems?

Jan

Funny. Just yesterday or so I was going through old IDE notes and discarding all that was outdated. Quoting from memory:

Andre: PDC 20265 pukes in 48-bit DMA clean in PIO primary - secondary is OK.

Last September or so there was a long discussion about a filesystem that was destroyed. But what I recall is that in the end it turned out not to be a hardware problem, but a precedence problem - two missing parentheses in the driver.

Google will tell you all, I suppose. Search for Promise and Isely.

Yes, thanks, I found these mails, and they may describe exactly the symptoms I saw on my server. So perhaps the fixes have not been (correctly) applied?

I only saw the mails from Mike Isely, but no 'official' response. Do you remember if the patches got accepted by one of the maintainers? Andre?

Jan

I see no kernel version in your post, that would be the first thing of interest. Next, look at this addressing variable via /proc. It it is zero, then you are hit by something avoided by the patch I sketched yesterday evening or so. Otherwise we must look further.

It looks like all your corrupted places are at a multiple of 2^27.

Also, I see that you do e2fsck on a mounted filesystem. Terrible.

Andries

Sorry - I mentioned it in an earlier post with a different subject. It's plain 2.4.21.

It is 0, yes. May it be caused by the following lines in pdc202xx_old.c?

        if (hwif->pci_dev->device == PCI_DEVICE_ID_PROMISE_20265)
                hwif->addressing = (hwif->channel) ? 0 : 1;

:-)
I know. But it's mounted read only, and to be sure I tried it without mounting the file system, as well.
But the most important prove that it's not a fs problem is that writing to one partition changed contents of a different partition.

Jan

OK. So, this means that you cannot access past the 2^28 sector boundary.

So, you can address at most 137 GB of your disk.

Did you say that it was 250 GB?

Exactly. And it's reported as 250GB, and I can access parts of the disk behind the 137 GB limit without an error message, but it looks like writing to these parts, it silently overwrites content at the beginning of the drive. Like it just discards the upper bits of the address or something like that.

Jan

Yes, that it what it does.

Look at my post from yesterday or so with Subject: [PATCH sketch] More IDE stuff

ide-disk.c: __ide_do_rw_disk() issues read/write requests to the disk. It does

        if (drive->addressing == 1)             /* 48-bit LBA */
                return lba_48_rw_disk(drive, rq, (unsigned long long) block);
        if (drive->select.b.lba)                /* 28-bit LBA */
                return lba_28_rw_disk(drive, rq, (unsigned long) block);
        return chs_rw_disk(drive, rq, (unsigned long) block);

with checking the size of block.
And init_idedisk_capacity() does not check addressing.

In my above post I gave a patch for 2.6.0-test3.
But the IDE code for 2.4 and 2.6 is very similar, so the patch, once applied to 2.6 should also be backported to 2.4.

Andries

It should also issue LBA28 if the size of th range and the end block fall under the LBA28 limit because thst saves you valuable I/O time.

Jens had patches for that but I don't know where they went in 2.6

Yep, but its other issue.

Jens and me agreed that they should be included after taskfile IO is integrated as the only IO path. Otherwise there are too many corner cases to deal with.

--bartlomiej

That is something different. The patches I gave (I gave patches didnt I?) limit the total capacity for large disks if the controller doesnt speak lba48.

That is necessary to avoid very unpleasant surprises.

But saving some time during I/O, yes, maybe I suggested Jens what to do and he did it. Hmm. I don't see the code anymore.
Google(Jens Andries rq_lba48) gives the patch.
Ah, I see. His patch went in - I see Changeset 1.1046.31.30 if that means anything, - and Bartlomiej asked for it to be reverted again. So, Jens's patches are lost for the moment. Must remember to resurrect them.

No, you didn't. You gave some whining and patch sketch. :-)

...when taskfile IO is the only IO path not earlier.

Jan, did removing offending lines from pdc202xx_old.c help?

--bartlomiej

I do not want to try it on our server, but I have a similar mainboard and just bought a 160GB harddisk to try it on another computer.

Unfortunately, it doesn't seem to work well. The hard disk only gives 20MB/s on sequential read (Seagate Baraccuda 160GB should be faster, right?), and hdparm -I /dev/hde completely locks up the whole computer - no console message at all, only hard reset helps. I need to get this running before I can try kernel patches for the LBA48 stuff.

By the way: That computer reports a
00:11.0 RAID bus controller: Promise Technology, Inc. 20265 (rev 02) while the server has a
00:11.0 Unknown mass storage controller: Promise Technology, Inc. 20265 (rev 02)

(Numeric Class IDs are 0104 and respectively 0180, vendor and device codes are the same on both computers)

Do you think this will make any difference? IIRC, the FastTrack series has some RAID features, but you can safely ignore them and use it just as a simple ATA100 controller.

Jan

Looks like this problem has been described here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=104250818527780&w=2

I'll try the patch this evening...

Why does promise IDE always have some strange problems?

Jan

That sounds about right for UDMA33, which is what you'd get without the fix I sent Marcelo a few days ago

But I've not yet patched the kernel, and while booting it says:

Aug 13 09:13:59 sirith kernel: hde: 312581808 sectors (160042 MB) w/8192KiB Cache, CHS=19457/255/63, UDMA(100)

hdparm -i also says:
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5

So I think it should use UDMA(100), which should give more than 20MB/s (Seagate says the drive does 32-58MB/s sustained transfer rate)

Jan

Ah sorry, I just had a look at the patch, and it seems like I'd get UDMA33 even though UDMA100 is reported, without that patch. So yes, this explains the slow speed.

Jan

You meant the .id -> .present change, right? I applied this patch, and still get only 20MB/s.

But I also have some good news: The patch by Ross Biro, available from
http://marc.theaimsgroup.com/?l=linux-kernel&m=104250818527780&w=2
actually fixed the hdparm -I crash.

Jan

Thanks I'd missed that bug fix somehow

Just remember to say Y to "Special FastTrak feature".

--bartlomiej

Ah, yes, it was that one. Yes, my patch turned into a mess of rejects after your layout changes of earlier patches from that series. But that does not matter in the least. You are intelligent, careful and precise, will not have any problem reconstructing the correct patch from my detailed description.

Here is the most important part (limiting the disk capacity to 137GB if LBA48 is not available) as a patch agains 2.4.21. The goto is not nice, but I didn't want to do a major rewrite.

Note that the function contains a bogous if-clause:

	if (id->cfs_enable_2 & 0x0400) {
	[...]
		return;
	} 
	[...]
	if ((id->command_set_2 & 0x0400) && (id->cfs_enable_2 & 0x0400)) {
	[...]

The second if-condition can never be true. But I didn't want to touch that stuff.

@@ -1196,7 +1196,7 @@
 		drive->bios_cyl		= drive->cyl;
 		drive->capacity48	= capacity_2;
 		drive->capacity		= (unsigned long) capacity_2;
-		return;
+		goto check_capacity48;
 	/* Determine capacity, and use LBA if the drive properly supports it */
 	} else if ((id->capability & 2) && lba_capacity_is_ok(id)) {
 		capacity = id->lba_capacity;

@@ -1228,6 +1228,15 @@
 		drive->sect = 63;
 		drive->cyl = (unsigned long)(drive->capacity48) / (drive->head * drive->sect);
 	}
+
+check_capacity48:
+	/* Limit disk size to 137GB if LBA48 addressing is not supported */
+	if (drive->addressing == 0 && drive->capacity48 > (1ULL)<<28) {
+		printk("%s: cannot use LBA48 - capacity reset "
+			"from %llu to %llu\n",
+			drive->name, drive->capacity48, (1ULL)<<28);
+		drive->capacity48 = (1ULL)<<28;
+	}
 }
 
 static unsigned long idedisk_capacity (ide_drive_t *drive)

Major rewrite is already present in 2.6.0-test3 and there is more to go.

Already removed from 2.6.

Ok, now I tried a kernel without the
"hwif->addressing = (hwif->channel) ? 0 : 1" line in pdc202xx_old.c, and yes, now the 160GB drive works without aliasing parts above 137GB back to the beginning of the drive.

Some simple tests (reading and writing a few 100MB of data) didn't show any nasty side effects. If you want me to do additional tests, please tell me. At the moment the drive doesn't contain any data, so I can even try dangerous things.

Jan

Thanks! Can you do some more testing?
LBA-48 should be (was before 2.4.20 according to previous bugreports) working just fine, but just to make sure.

I will remove these offending lines from 2.6.x if there are no problems.

--bartlomiej

Ok, after I compiled the driver with special udma feature and special fasttrak feature, I got decent transfer rates (40-50MB/s) from the drive.

I let a process running which continously copied a big (700MB) file, removed the old copy, umounted an mounted the filesystem, and checked the md5sum of the file. I didn't see a single corruption in >160 copies.

At the same time, a second process was generating load by compiling kernels on a different partition on the same drive. One partition was below the 137GB boundary, one was above it.

These tests were done with patched 2.4.21.

Jan

Yes. I did some googling.

http://www.ussg.iu.edu/hypermail/linux/kernel/0209.0/0000.html
and
http://www.ussg.iu.edu/hypermail/linux/kernel/0209.0/0898.html

are essential.

It looks LBA-48 was disabled on PDC20265 as a "workaround", because there was a stupid in Promise LBA-48 support.
Bug was fixed, but "workaround" disabling LBA-48 was never removed.

You can remove these two lines and see if it helps (but it may corrupt your fs even more if it doesn't).

Because of this bug you are hitting another bug which Andries has described recently.

--bartlomiej