Skip to contentSkip to conversation

2.6.0-test11 data loss

DISCUSSION Keith Lea 8 participants 10 messages Dec 24, 2003 – Dec 25, 2003

lore
Share & tools
lore

Keith Lea

author
#0 lore raw

Hello, I'm not subscribed to this list. This is not a help request, and not really a bug report, I just thought someone should know about this.

I installed the 2.6.0-beta11-mm kernel last week, and the other day my computer locked up (this is normal on my laptop with every kernel version I've tried, this isn't the problem I'm posting about). When I restarted, many, many files that had been open when it locked up were filled with garbage, or the contents of totally unrelated files. For example, my syslog contained some KDE header file code, and /sbin/modprobe contained 82kb of data that seemed like random noise. I think each file was the same size as it was originally, just with different data, but I'm not sure.

The corruption happened on two separate partitions on a single IDE laptop drive, and both were ReiserFS 3.6 partitions. I don't know if this is a kernel bug or a Reiser bug or something else, but I thought the kernel developers should know about this, and be on the lookout for similar things (hopefully with more informative bug reports than mine). I'm sorry I don't have more information, but if anyone wants to know more about my system I'd be glad to help.

-Keith Lea

#1 lore raw

Hi,

I've been hit by the same problem but using 2.6.0 . As you described, garbage in files (eg. /etc/modules.conf, ...).

2.6.0, Slackware 9.1

Quoted 3 lines

The corruption happened on two separate partitions on a single IDE laptop drive, and both were ReiserFS 3.6 partitions. I don't know if this is a kernel bug or a Reiser bug or something else, but I thought

I don't think this is a reiserfs bug. This was my first thought and after first hitting this bug, I've moved all my partitions from reiserfs to jfs. But I've also had this problem with it... Now I'm back to 2.4.23, and everything works fine.

Gergely

#2 lore raw

Hello

Quoted 5 lines · Gergely Tamas

I've been hit by the same problem but using 2.6.0 . As you described, garbage in files (eg. /etc/modules.conf, ...).

2.6.0, Slackware 9.1

Quoted 8 lines
Quoted 3 lines

The corruption happened on two separate partitions on a single IDE laptop drive, and both were ReiserFS 3.6 partitions. I don't know if this is a kernel bug or a Reiser bug or something else, but I thought

I don't think this is a reiserfs bug. This was my first thought and after first hitting this bug, I've moved all my partitions from reiserfs to jfs. But I've also had this problem with it... Now I'm back to 2.4.23, and everything works fine.

Because of the numerous reboots and hangs I've seen with experimental patches I've also seen this, but it's not reiserFS fault. The problem is that most drives have write caching enabled and not all of them are safe with this. If you disable it with hdparm (hdparm -W 0 /dev/hd*) you'll find that open files during a hard reset or power outage will prevent those open files from being corrupted.

Con

#3 lore raw
Quoted 7 lines · Con Kolivas
Quoted 4 lines · Gergely Tamas

I don't think this is a reiserfs bug. This was my first thought and after first hitting this bug, I've moved all my partitions from reiserfs to jfs. But I've also had this problem with it... Now I'm back to 2.4.23, and everything works fine.

Because of the numerous reboots and hangs I've seen with experimental patches I've also seen this, but it's not reiserFS fault. The problem is that most drives have write caching enabled and not all of them are safe with this. If you disable it with hdparm (hdparm -W 0 /dev/hd*) you'll find that open files during a hard reset or power outage will prevent those open files from being corrupted.

Write cache off will not prevent a file from being corrupted, however, it should limit the corruption to a single disk operation.

I don't see how the behavior you describe could be the drive's fault...

The user stated that their system hard locked, then they went and rebooted it, and following the reboot they had corruption... From this, there are a few possibilities:

1. The drive had been given the commands to write the data prior to the hang.

If this was the case, the drive would happilly keep writing the data it had been given and was caching in the background, even while you continued to send (or stopped sending) data for a new command over the interface. An IDE interface lockup or system lockup will not prevent the drive from flushing the remainder of its write cache. (Only possible exception might be faulty handling of a hard reset, but all drives today will flush their cache when they see the reset, prior to processing it.) Unless the user yanked power within a few hundred milliseconds of the write command, I think it is unlikely that cached data already in the drive wasn't flushed properly.

2. The drive was in the middle of a command writing important data during the hang.

In this case, yes, your file you were writing would probably be corrupt on the media, but nothing more. Drives detect power loss, and immediately disable write-gate and park the actuator. If they don't get the actuator parked before they run out of back-EMF from the momentum of the platter(s), the head will stick to the media and you'll probably need a chisel to get that drive to spin again.

3. The drive hadn't yet been issued the commands for the data that was eventually corrupted.

I find this to be the most likely case, and is a situation where the filesystem thinks objects were moved but those updates were not correctly sent to the disk (due to the hang?), so it might think they're in the old location or something. (I'm not a filesystem wizard so if I'm way off-base, my apologies)

It seems to me that the problem occurred at a higher system level than the disk, and disabling the write cache on the drive (besides being a *HUGE* performance loser) will only make the window for failure smaller, not eliminate it entirely.

Unless you are using *really* old hard drives, the write caching in today's drives is really quite good and definitely should be usable. Sure, it makes things less safe in power events, but system lockups shouldn't affect the drive's ability to flush its cache. Note too that Gergely reported that the problem went away on his 2.4.23 system. I don't believe that to be a small data point.

--eric, posting from home

Signature · 2 lines
Eric D. Mudama
[email protected]
#4 lore raw
Quoted 62 lines · Eric D. Mudama
Quoted 6 lines · Con Kolivas
Quoted 4 lines · Gergely Tamas

I don't think this is a reiserfs bug. This was my first thought and after first hitting this bug, I've moved all my partitions from reiserfs to jfs. But I've also had this problem with it... Now I'm back to 2.4.23, and everything works fine.

Quoted 6 lines

Because of the numerous reboots and hangs I've seen with experimental patches I've also seen this, but it's not reiserFS fault. The problem is that most drives have write caching enabled and not all of them are safe with this. If you disable it with hdparm (hdparm -W 0 /dev/hd*) you'll find that open files during a hard reset or power outage will prevent those open files from being corrupted.

Write cache off will not prevent a file from being corrupted, however, it should limit the corruption to a single disk operation.

I don't see how the behavior you describe could be the drive's fault...

The user stated that their system hard locked, then they went and rebooted it, and following the reboot they had corruption... From this, there are a few possibilities:

1. The drive had been given the commands to write the data prior to the hang.

If this was the case, the drive would happilly keep writing the data it had been given and was caching in the background, even while you continued to send (or stopped sending) data for a new command over the interface. An IDE interface lockup or system lockup will not prevent the drive from flushing the remainder of its write cache. (Only possible exception might be faulty handling of a hard reset, but all drives today will flush their cache when they see the reset, prior to processing it.) Unless the user yanked power within a few hundred milliseconds of the write command, I think it is unlikely that cached data already in the drive wasn't flushed properly.

2. The drive was in the middle of a command writing important data during the hang.

In this case, yes, your file you were writing would probably be corrupt on the media, but nothing more. Drives detect power loss, and immediately disable write-gate and park the actuator. If they don't get the actuator parked before they run out of back-EMF from the momentum of the platter(s), the head will stick to the media and you'll probably need a chisel to get that drive to spin again.

3. The drive hadn't yet been issued the commands for the data that was eventually corrupted.

I find this to be the most likely case, and is a situation where the filesystem thinks objects were moved but those updates were not correctly sent to the disk (due to the hang?), so it might think they're in the old location or something. (I'm not a filesystem wizard so if I'm way off-base, my apologies)

It seems to me that the problem occurred at a higher system level than the disk, and disabling the write cache on the drive (besides being a *HUGE* performance loser) will only make the window for failure smaller, not eliminate it entirely.

Quoted 6 lines

Unless you are using *really* old hard drives, the write caching in today's drives is really quite good and definitely should be usable. Sure, it makes things less safe in power events, but system lockups shouldn't affect the drive's ability to flush its cache. Note too that Gergely reported that the problem went away on his 2.4.23 system. I don't believe that to be a small data point.

I hardly said it was the correct solution; just what worked for me, as I had exactly the same issue going 2.4->2.6. I can't even recall if write caching was actually on in 2.4, and my write performance under video capture has not shown any detriment. The filesystem gods should comment. Merry Christmas.

Con

#5 lore raw
In reply to Eric D. Mudama
Quoted 8 lines · Eric D. Mudama

It seems to me that the problem occurred at a higher system level than the disk, and disabling the write cache on the drive (besides being a *HUGE* performance loser) will only make the window for failure smaller, not eliminate it entirely.

You should only use write caching in kernels where write cache flushing is supported. Chris, which ones are those, could you remind us?

Signature · 1 line
Hans
#6 lore raw
In reply to Gergely Tamas
Quoted 4 lines · On Dec-24 2003, Wed, 23:22 +0100 Gergely Tamas <[email protected]> wrote:

I've been hit by the same problem but using 2.6.0 . As you described, garbage in files (eg. /etc/modules.conf, ...).

2.6.0, Slackware 9.1

Count me in.

IBM ThinkPad T40p (PIIX IDE HDD access)
slackware-current
linux-2.6.0
reiserfs-3.6

I can reproduce the problem anytime simply by terminating an XDM session.

- complete freeze
- blank screen
- can't see an oops
- nothing in the logs
- kernel won't panic (tried w/ the morsecode panics patch)
- typically corrupted files (random garbage in the middle):
/lib/modules/2.6.0/modules.dep /var/adm/messages

I'm hesitant in blaming hdd write cache since there's no power outage involved (also I've never seen this before w/ 2.4).

Signature · 1 line
Tomas Szepe <[email protected]>
#7 lore raw

I'm getting a system freeze but no file corruption. The freeze happens randomly after all rc.d scripts run. The freeze seems to happen slightly at a "later" time when I applied the 2.6.0-mm1 patch(I was able to login and startx) whereas before the freeze happened before/while logging in.

My boot parameters usually look like this:

BOOT_IMAGE=Linux-2.6.0 ro root=303 apm=on acpi=off

IBM Thinkpad T22
linux-2.6.0 | linux-2.6.0-mm1
slackware 9.1

bash-2.05b# lspci
00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 03)
00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 03)
00:02.0 CardBus bridge: Texas Instruments PCI1450 (rev 03) 00:02.1 CardBus bridge: Texas Instruments PCI1450 (rev 03) 00:03.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 0c)
00:03.1 Serial controller: Lucent Microelectronics LT WinModem (rev 01) 00:05.0 Multimedia audio controller: Cirrus Logic CS 4614/22/24 [CrystalClear SoundFusion Audio Accelerator] (rev 01)
00:07.0 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02) 00:07.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01) 00:07.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01) 00:07.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 03) 01:00.0 VGA compatible controller: S3 Inc. 86C270-294 Savage/IX-MV (rev
13)

Now that I went back and tested the kernel(mm1) with the following parameters, the system hasn't freezed yet. I'll report if anything goes wrong.

BOOT_IMAGE=Linux-2.6.0 ro root=303 idebus=66 ide0=ata66 ide1=ata66 ide2=ata66 apm=on acpi=off

Thank you

Quoted 3 lines · Tomas Szepe

On Dec-24 2003, Wed, 23:22 +0100
Gergely Tamas <[email protected]> wrote:

#8 lore raw
In reply to Keith Lea
Quoted 13 lines · Keith Lea

Hello, I'm not subscribed to this list. This is not a help request, and not really a bug report, I just thought someone should know about this.

I installed the 2.6.0-beta11-mm kernel last week, and the other day my computer locked up (this is normal on my laptop with every kernel version I've tried, this isn't the problem I'm posting about). When I restarted, many, many files that had been open when it locked up were filled with garbage, or the contents of totally unrelated files. For example, my syslog contained some KDE header file code, and /sbin/modprobe contained 82kb of data that seemed like random noise. I think each file was the same size as it was originally, just with different data, but I'm not sure.

Quoted 7 lines

The corruption happened on two separate partitions on a single IDE laptop drive, and both were ReiserFS 3.6 partitions. I don't know if this is a kernel bug or a Reiser bug or something else, but I thought the kernel developers should know about this, and be on the lookout for similar things (hopefully with more informative bug reports than mine). I'm sorry I don't have more information, but if anyone wants to know more about my system I'd be glad to help.

I know this is not the answer you're looking for but, could you please test again using 2.6.0 or 2.6.0-mm1? 2.6.0-test11 is a bit ouf of date now that 2.6.0 has gone gold.

#9 lore raw
In reply to Keith Lea
Quoted 8 lines · Keith Lea

Hello, I'm not subscribed to this list. This is not a help request, and not really a bug report, I just thought someone should know about this.

I installed the 2.6.0-beta11-mm kernel last week, and the other day my computer locked up (this is normal on my laptop with every kernel version I've tried, this isn't the problem I'm posting about). When I restarted, many, many files that had been open when it locked up were filled with garbage, or the contents of totally unrelated files. For example, my syslog contained some KDE header file code, and /sbin/modprobe contained 82kb of data that seemed like random noise. I think each file was the same size as it was originally, just with different data, but I'm not sure.

The corruption happened on two separate partitions on a single IDE laptop drive, and both were ReiserFS 3.6 partitions. I don't know if this is a kernel bug or a Reiser bug or something else, but I thought the kernel developers should know about this, and be on the lookout for similar things (hopefully with more informative bug reports than mine). I'm sorry I don't have more information, but if anyone wants to know more about my system I'd be glad to help.

(quoted diff) +0 -2
-Keith Lea
-
Quoted 6 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

please read about the difference between metadata journaling, data journaling, and atomic filesystems, and all will become clear. also note the ordered writes option for version 3.6 of reiserfs which is probably what you want until atomic reiser4 is fully stable.

Signature · 1 line
Hans