Blockbusting news, results are in

In order of importance instead of chronology:

In the presence of friends who are disk drive engineers at Toshiba, I tried to read the file containing the bad block, we listened to the disk drive do auto-retries, we watched Linux record the I/O failure in the system log, we saw the "cp" program report an I/O error, and we observed that the drive did not reallocate the bad block during reads.

Next, I tried to write the bad block. The following command is not the first one that I tried but it was the first one to actually try writing the bad block:
dd if=/dev/zero of=/dev/hda bs=512 seek=19021881 count=1
We listened to the disk drive do auto-retries, we watched Linux record the I/O failure in the system log, we saw the "dd" command report an I/O error, and we observed that the drive did not reallocate the bad block during writes. (The bad block number is 19021882.)

After a few other experiments, I used smartctl to direct the drive to do a long self-test. When it completed, we observed that the drive had self-diagnosed a read failure on the same bad sector number as always, and we observed that the drive did not reallocate the bad block during long self-tests.

Does anyone need more?

A partial solution could be to stop using Toshiba drives, but I don't think this will be a complete answer. Toshiba is not the only maker whose disk drives get bad blocks. We do not know if Toshiba is the only maker whose firmware refuses to reallocate bad blocks when permanent errors are detected, because the makers aren't saying.

File systems must maintain lists of bad blocks and prevent ordinary file operations from ever using those sector numbers.

Someone pointed out that this technique will not work for swap partitions. I agree. The "mkswap" command needs to test every sector in the swap partition and warn the user if the partition will be unusable.

Now for the less important stuff.

After many hours of "find"ing and "cp"ing files to /dev/null, the bad block was detected to be in file
/usr/share/locale/es/LC_MESSAGES/bfd.mo
So indeed, this file had been written once and was not intended to be written again, and could easily be restored from a source of good data. But I was really startled by this, because I don't use Spanish locales. The only locales I use are Japanese and English. So why did this file even get read, even while I was doing kernel compiles and stuff like that? After all, the reason the bad block was getting logged in the system log was that the file was getting read.

I "mv"ed the file to file /badblockhere and used rpm with --replacepkgs to reinstall binutils from SuSE's 8.2 distribution. Then copied the new correct file /usr/share/locale/es/LC_MESSAGES/bfd.mo to file /goodfilehere. This preparation made it easy to do experiments with my Toshiba friends when they visited.

My first attempt to write the bad block (after the read experiments) was:
dd if=/goodfilehere of=/badblockhere
But this did not even try to write to the bad block. The drive did not try to do any auto-retries, there were no errors in the system log, and the dd command output a success message. Next, a repeat of a read attempt that used to fail:
cp /badblockhere /dev/null
succeeded. So I guess that the when the dd command is told to output to an ordinary file, it does not overwrite its output file, it creates a new file and then renames it to replace the old file. (Too bad it couldn't do the same when I ran this command:
dd if=/dev/zero of=/dev/hda bs=512 seek=19021881 count=1
and write a new disk drive to replace the old one ^u^)

And now that block is in free space somewhere, waiting for Linux and the Reiser filesystem to allocate it when creating or expanding some future file.

The bad block can still be detected. This fails as always:
dd if=/dev/hda of=/dev/null bs=512 skip=19021881 count=1
(The bad block number is 19021882.)

By the way, Toshiba's US subsidiary has indications on their web site that they provide warranty service on their products, but that they have reduced the warranty period from three years to one year. This was a smart move by Toshiba's US subsidiary. If their disk drives start to develop bad blocks after two years, then customers don't discover how bad Toshiba's firmware is until two years have passed, and now they can't even make claims to get firmware fixed.

Toshiba's head office is even smarter. In Japanese they refuse entirely to provide warranty service to end users. Customers have to send defective disk drives back up through the sales channel. Well, lucky customers who bought the disk drive as part of a notebook computer probably get one year's warranty from the vendor of the notebook computer, so if they're lucky enough to learn about Toshiba's firmware within a year then they can send their entire computer back for some length of time to get warranty service. But anyone who went to Akihabara and bought the drive by itself from a parts store, the store probably offers one week or one month to replace a failing drive if it was dead on arrival. In these cases a customer who learns about Toshiba's firmware after two weeks or five weeks gets screwed.

My disk drive was made at Toshiba's factory in Gifu prefecture on September 13, 2001. Since that time the factory has closed and this model has been discontinued.

But Toshiba isn't the only maker who isn't saying how bad their firmware is. We need those bad block lists. They are as necessary as they ever were.

I'm not sure why this is a news flash. When I was at Sun a 2GB drive cost us $4000. I think we sold them for $6000. You can't buy a 2GB drive today nor a 20GB drive. A 200GB drive costs $160. That's 100 times bigger for 25 times less money, or a net increase of price/capacity of 2500. In the same period of time, CPUs have not kept up though they are close.

You're suprised that drives are unreliable? Please. You are getting unbelievable value from those drives and you demanded it. Price is the only way people make purchasing decisions, that's why DEC got out of the drive business, then HP did, and then IBM did. They couldn't afford to compete with the cutrate junk that we call drives today.

I'm not blaming you, I'm as bad as the next guy, I buy based on price as well but I have no illusions that what I am buying is reliable. The drives we put into servers here go through a couple weeks of all bit patterns being changed and even then we don't depend on them, everything is backed up.

I've told you guys over and over that you need to CRC the data in user space, we do that in our backup scripts and it tells us when the drives are going bad. So we don't get burned and you wouldn't either if you did the same thing.

Drives are amazingly cheap, it's a miracle that they work at all, don't be so suprised when they don't.

---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Hi;

I think you may be missing the point he is trying to make
in order to take your hobby horse for a spin;) He is trying to claim, that he has a disk that is not dying, that has a bad sector that he cant get remapped, and thus, there needs to be support for bad blocks in the filesystem layer. (in the face of the argument that modern disks make filesystem support of bad blocks irrelevant.)
As a side note, I also have a 6gig disk, which a few
years ago was, ahem, bumped during a write. It now has a handful of screwy sectors, that I cant get rid of, even after doing the stuff Norman describes. I used the -c option to e2fsck, and its been doing great ever since-- a few years of use without more bad sectors.

Paul
[email protected]

First you have to make Linux have a direct path back to the application layer which owns the request. Then you can attempt a filesystem remapping code war.

Well basically there are ways to force invoke the remap but 99% of the people can not and will not go through the hassle. So I am not going to spend time explaining each and every vendor mode.

That is what people in media forensics get paid to do.

Why do the CRC in user space, that requires modifying every one of 7000+ applications (if I understand you correctly, which is far from a sure thing;-) )?

Write a reiser4 CRC file plugin. It would take a weekend, and most of the work would be cut and pasting from the default file plugin..

I understand why you do it in BK, but for user space as a whole user space is the wrong place.

I think the fs driver layer might be the wrong thing too; maybe it'd be best to do the CRC and/or checksumming at the block layer?

At the very least, I see a lack of genericity with respect to making it a plugin to a specific fs. I'm going to try not to delve too far into specifics, as my knowledge in these areas is limited, but I'd welcome any corrections of misunderstandings I might have about feasibility, value, or importance of these things, and even techhical misconceptions.

Jens, I apologize if advance if this is just another lame flamewar best bitbucketed as opposed to answered.

Thanks.

-- wli

Hi!

I think that's best place.

Here's first attempt at implementation:
http://www.ussg.iu.edu/hypermail/linux/kernel/0004.3/0487.html

Pavel

When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

Wow, that was a while ago. Well, I'm impressed.

-- wli

Yes, because you can then use it on ST-506 disks if you really want to, and never see a bad block at the filesystem level. It also means that people with drives that do defect management to their satisfaction don't need the overhead of the defect management in software layer, however small it may be.

John.

Or even better, do it on the disk controller strapped to the physical disk so you can hide the fact that CRCs add data overhead for every block making them longer than 2^n and can use a CRC with optimised for the type of errors most common on the media. Wait, that is where it is already being done.

What is apparently missing is better handling of the uncorrectable errors. Specifically the ability to pass the errors and warnings up to the OS for evaluation and for the OS to be able to request a block remap or to undo a block remap.

I'd guess that most of Hans' errors are not coming from spinning media but from tranmission errors on the HD cables and system busses, and from undetected memory errors.

________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt

Why this suggestion keeping coming up, I have no idea. If you take the idea to it's extreme, it's basically saying that we should off-load all processing on to the host. Although there has been a move towards dumb peripherals in recent years, (E.G. software modems), I have seen almost no even vaguely convincing arguments other than cost as to why they are superior, (lower latency has been mentioned with regard to software modems - I fail to see the benefit, although I suppose it might exist for games players). Apart from some data recovery applications, I don't see how it is possible to do anything really useful simply by adding the ability to pass some warnings and errors up to the OS, without giving the OS access to all of the data that the drive firmware has access to.

Obviously drives with completely open and free firmware would be great, but that is not likely to happen in the near future, so for the time being, if you don't like the way drives handle defect management, complain to the manufactuers. I am satisfied with the way Maxtor disks handle defect management, both Eric's explainations and my own observations.

John.

I'm not suggesting the drive off-load it. I'm only suggesting that there be a mechanism for the host to be more involved if the host is capable.

The problem that began this thread is a perfect example. A bad block that the drive firmware apparently will not remap calls for the ability to explicitly instruct the drive to remap it. In some cases it might be good to be able to let the host countermand a remap if the disk reports overtemp.

No disagreement here.

________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt

I am not getting any errors;-), but what you say sounds likely to be true of the reiserfs users who see errors, and I trust Larry's account of his users.

Hans

Sorry, i meant Larry.

________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt

Hi!

Even better, use crc loop method, and do checks at block device level. I had patch to implement that...
Pavel

When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

this would not get the memory errors that Larry was talking about as well..... which is not to discount your patch, sounds like a reasonable patch.....

Hans

End to end data integrity checking is a requirement. Otherwise errors happen silently and you rarely if ever see them. And the error checking must be end to end because you cannot trust the other layers to work properly 100% of the time.

However to actually track down errors to root causes of errors, the closer you can have your error checking to the hardware the better. So having CRC data or similar in the filesystem for both the metadata and the file information is a good thing.

Error checking should not be necessary for casual files that you don't really care about but for times when you care about the integrity of your data the application should be checking it.

Eric

On Sun, 2003-10-19 11:16:42 +0900, Norman Diamond <[email protected]> wrote in message <1c6401c395e7$16630d00$3eee4ca5@DIAMONDLX60>:

Maybe the drive can't remap the block because there's no free space in the remap area available any more...

In this case, the real problem is that the drive doesn't tell in advance that is already remapped some blocks.

MfG, JBG

   Jan-Benedict Glaw       [email protected]    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));