Skip to contentSkip to conversation

Software RAID1 deadlock in 2.6.25 kernels

DISCUSSION Neil Brown 9 participants 18 messages Jun 27, 2008 – Jul 8, 2008 1 message not in the archive MD RAID

lore
Share & tools
lore

Neil Brown

author
#1 lore raw
In reply to a message not in the archive
Quoted 1 line · On Friday June 27, [email protected] wrote:

Hello Neil and Linus,

I've dropped Linus from cc as he won't be interested, and added linux-raid.

Quoted 12 lines

I am contacting both of you because you have submitted md raid code
to the 2.6.25 kernels. I have found an apparent deadlock condition using software RAID1 with SUSE 11.0 and Fedora 9, both which use the 2.6.25 kernels. Configuration is as follows:

Dell Precision 450 with single Xeon 2.4 GHz processor
(hyperthreaded, looks like a dual core)
2 Seagate 120GB IDE drives
sda1,sdb1 - 1GB swap
sda2,sdb2 - 100MB, RAID1, md0, /boot
sda3,sdb3 - balance of disk, RAID1, md1 3 LVM filesystems

Quoted 2 lines

System locks up after running a short time. Had it hang once during
installation. Tried both Reiserfs and EXT3.

We need more details about the lockup. Does the whole system lock up, or just the array.
Can you get
alt-sysrq-T
output? That might be useful.

Quoted 2 lines

Are either of you aware of this issue? If not, I can file a bugzilla.

No I'm not aware of such an issue.
I prefer to avoid bugzilla, lets just discuss it here.

NeilBrown

#2 lore raw
Quoted 8 lines · Neil Brown
Quoted 2 lines

System locks up after running a short time. Had it hang once during
installation. Tried both Reiserfs and EXT3.

We need more details about the lockup. Does the whole system lock up, or just the array.
Can you get
alt-sysrq-T
output? That might be useful.

Also, try to set up netconsole and boot with nmi_watchdog=2. I've now 3 completely different HW setups where 2.6.25 hangs due to HPET issues.

Gabor

Signature · 4 lines
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------
#3 lore raw
Quoted 13 lines · Gabor Gombas
Quoted 9 lines · Neil Brown
Quoted 3 lines

System locks up after running a short time. Had it hang once during
installation. Tried both Reiserfs and EXT3.

We need more details about the lockup. Does the whole system lock up, or just the array.
Can you get
alt-sysrq-T
output? That might be useful.

Quoted 6 lines

Also, try to set up netconsole and boot with nmi_watchdog=2. I've now 3 completely different HW setups where 2.6.25 hangs due to HPET issues.

Gabor

Due to time constraints, I will be unable to do much on this until the weekend.

In the meantime, a more detailed description of the symptoms:

When the system hangs, the mouse movement is tracked across the screen and I can ping the node. There is no response to clicking on a mouse button or trying to type anything into a field. It also does not respond to ssh or to <CTRL><ALT><F1>.

I may also try and reproduce it on different hardware to rule out a graphics problem if you wish.

Mike

#4 lore raw

Hi,

Quoted 7 lines · Mike McCarthy
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

System locks up after running a short time. Had it hang once
during installation. Tried both Reiserfs and EXT3.

When the system hangs, the mouse movement is tracked across the screen and I can ping the node. There is no response to clicking on a mouse button or trying to type anything into a field. It also does not respond to ssh or to <CTRL><ALT><F1>.

Maybe it's a totally different issue, but I also noticed system lockups, that started after I converted the system to Software-RAID1. However, in my case the lockups only occur after 3-10 days uptime. One day I was able to capture a couple of syslog entries:

| Jun 12 09:50:47 tardis kernel: hdg: lost interrupt
| Jun 12 09:50:47 tardis kernel: hdg: drive_cmd: status=0x51 { DriveReady SeekComplete Error }
| Jun 12 09:50:47 tardis kernel: hdg: drive_cmd: error=0x04 { DriveStatusError }
| Jun 12 09:50:47 tardis kernel: ide: failed opcode was: 0xb0
| Jun 12 09:51:07 tardis kernel: hdg: dma_timer_expiry: dma status == 0x21
| (2 x WD2500SB-01RFA0 on a PDC20276 (MBFastTrak133))

The HDD LED is permanently on.

Cheers,
MB

Signature · 4 lines
Michael Bussmann <[email protected]>
BOFH excuse #136:

Daemons loose in system.
#5 lore raw
Quoted 17 lines · Michael Bussmann

Hi,

Quoted 7 lines · Mike McCarthy
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

System locks up after running a short time. Had it hang once
during installation. Tried both Reiserfs and EXT3.

When the system hangs, the mouse movement is tracked across the screen and I can ping the node. There is no response to clicking on a mouse button or trying to type anything into a field. It also does not respond to ssh or to <CTRL><ALT><F1>.

Maybe it's a totally different issue, but I also noticed system lockups, that started after I converted the system to Software-RAID1. However, in my case the lockups only occur after 3-10 days uptime. One day I was able to capture a couple of syslog entries:

Quoted 9 lines
| Jun 12 09:50:47 tardis kernel: hdg: lost interrupt
| Jun 12 09:50:47 tardis kernel: hdg: drive_cmd: status=0x51 { DriveReady SeekComplete Error }
| Jun 12 09:50:47 tardis kernel: hdg: drive_cmd: error=0x04 { DriveStatusError }
| Jun 12 09:50:47 tardis kernel: ide: failed opcode was: 0xb0
| Jun 12 09:51:07 tardis kernel: hdg: dma_timer_expiry: dma status == 0x21
| (2 x WD2500SB-01RFA0 on a PDC20276 (MBFastTrak133))

The HDD LED is permanently on.

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#6 lore raw
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

Mike

#7 lore raw
Quoted 5 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

Quoted 9 lines

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I am running 2.6.25 and have run every 'fix' iteration to now currently 2.6.25.9, I use RAID1 on two hosts and have not seen any issues on either.

That error looks HW related, you should run some heavy disk I/O processes for a few days / see if you can get it to re-occur on 2.6.22?

Or.. does it occur immedaitely after booting 2.6.25?

Justin.

#9 lore raw
In reply to Richard Scobie
Quoted 10 lines · Richard Scobie
Quoted 7 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

FC9 running on RAID 1 (ata_piix) for the last 3 weeks or so, with no trouble.

Same here. I haven't had any issues with RAID 1 on Fedora 9.

So the issue isn't biting everyone but there is always the chance something is wrong with a specific controller or configuration.

Michael

#8 lore raw
In reply to Mike McCarthy
Quoted 7 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

FC9 running on RAID 1 (ata_piix) for the last 3 weeks or so, with no trouble.

Regards,

Richard

#11 lore raw
In reply to Mike McCarthy
Quoted 6 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

Given heavy 2.6.25 use, my guess is still that the root cause of this is hardware, and that the change in disk code either triggers the hardware problem, or handles it differently. Are you by any chance running NCQ on your system?

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#12 lore raw
Quoted 10 lines · Bill Davidsen
Quoted 6 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

Given heavy 2.6.25 use, my guess is still that the root cause of this is hardware, and that the change in disk code either triggers the hardware problem, or handles it differently. Are you by any chance running NCQ on your system?

No. This system and the drives pre-date NCQ. I think NCQ is only implemented in SATA and these are IDE drives. Sometime over the weekend, I am going to reload SUSE 11 and try to do some more debugging.

BTW: It's back to 10.3 (kernel 2.6.22) running happily with a VMware server thrashing away at the disks.

Mike

#10 lore raw
In reply to [email protected]
Quoted 10 lines · On Mon, Jun 30, 2008 at 1:19 PM, [email protected] <[email protected]>…
Quoted 9 lines · Richard Scobie
Quoted 7 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

2.6.22 is running fine. The problems are in the 2.6.25 kernel (FC9 and SUSE 11.0)

Quoted 7 lines
Quoted 2 lines

FC9 running on RAID 1 (ata_piix) for the last 3 weeks or so, with no trouble.

Same here. I haven't had any issues with RAID 1 on Fedora 9.

So the issue isn't biting everyone but there is always the chance something is wrong with a specific controller or configuration.

FWIW, me too. I have three different Fedora 9 systems running the latest Fedora kernels based on 2.6.25 which are running software RAID 1 without any issues at all.

-Dave

#13 lore raw
In reply to Mike McCarthy

Hi,

Quoted 8 lines · Mike McCarthy
Quoted 7 lines · Bill Davidsen
Quoted 6 lines · Mike McCarthy
Quoted 5 lines · Bill Davidsen

Wonder if hardware or software is happening, sounds like an mishandled hardware error, but I'm guessing. I have a server with RAID1 and Fedora 2.6.22.14-72.fc6PAE kernel, up 72 days, no problems.

I have a number of 2.6.25.7-9 machines with SW-RAID1 that are running flawlessly so far.

Quoted 6 lines
Quoted 5 lines

Given heavy 2.6.25 use, my guess is still that the root cause of this is hardware, and that the change in disk code either triggers the hardware problem, or handles it differently. Are you by any chance running NCQ on your system?

No. This system and the drives pre-date NCQ. I think NCQ is only

Same here. In my case the lockups are totally random and not related to heavy disc i/o. Actually most lockups occur when the system was quite idle.

So far I tried
- Replaced IDE cables
- kernel upgrades up to 2.6.25.9
- removed one drive from the RAID, thus running in degraded mode
- disabled CPU frequency scaling
- put one drive on the PDC20276, the other on the ICH4 (82801DB)

Maybe my hardware _is_ broken, but I'll try some other settings anyway (including using RTC again for ztdummy instead of HPET, disabling NOHZ etc).

Cheers,
MB

Signature · 1 line
Michael Bussmann <[email protected]>
#14 lore raw
Quoted 6 lines · Michael Bussmann

So far I tried
- Replaced IDE cables
- kernel upgrades up to 2.6.25.9
- removed one drive from the RAID, thus running in degraded mode
- disabled CPU frequency scaling
- put one drive on the PDC20276, the other on the ICH4 (82801DB)

Set up netconsole or serial console, boot with nmi_watchdog=2, and wait for a lockup.

Gabor

Signature · 4 lines
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------
#15 lore raw
Quoted 11 lines · Gabor Gombas
Quoted 6 lines · Michael Bussmann

So far I tried
- Replaced IDE cables
- kernel upgrades up to 2.6.25.9
- removed one drive from the RAID, thus running in degraded mode
- disabled CPU frequency scaling
- put one drive on the PDC20276, the other on the ICH4 (82801DB)

Set up netconsole or serial console, boot with nmi_watchdog=2, and wait for a lockup.

Some motivation: I have a feeling that the HPET problem in 2.6.25 is actually biting people other than me. But if you do not enable the NMI watchdog and do not have a way to save the resulting Oops, then you will never know what hit you. SysRq+W will show tasks blocked in places related to disk I/O, but that's not the real reason, so messing with the disks will not help you in this case.

And this also explains why machines w/o HPET (or HPET being disabled by the BIOS or otherwise unaffected for yet unknown reasons) can run 2.6.25.x just fine.

Gabor

Signature · 4 lines
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------
#16 lore raw
In reply to Mike McCarthy
Quoted 7 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Given heavy 2.6.25 use, my guess is still that the root cause of this is hardware, and that the change in disk code either triggers the hardware problem, or handles it differently. Are you by any chance running NCQ on your system?

No. This system and the drives pre-date NCQ. I think NCQ is only implemented in SATA and these are IDE drives. Sometime over the weekend, I am going to reload SUSE 11 and try to do some more debugging.

BTW: It's back to 10.3 (kernel 2.6.22) running happily with a VMware server thrashing away at the disks.

This has recycled back to the top of my todo list, I have a server in mothballs with IDE drives, I'll pull it out, upgrade to FC9 current (non-rawhide) and see if I have any problems. It's off due to lack of need, not really obsolete, so it's a fair test. O'll put a dew hundred GB of raid-1 and beat on it.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#17 lore raw
Quoted 11 lines · Bill Davidsen
Quoted 7 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Given heavy 2.6.25 use, my guess is still that the root cause of this is hardware, and that the change in disk code either triggers the hardware problem, or handles it differently. Are you by any chance running NCQ on your system?

No. This system and the drives pre-date NCQ. I think NCQ is only implemented in SATA and these are IDE drives. Sometime over the weekend, I am going to reload SUSE 11 and try to do some more debugging.

BTW: It's back to 10.3 (kernel 2.6.22) running happily with a VMware server thrashing away at the disks.

This has recycled back to the top of my todo list, I have a server in mothballs with IDE drives, I'll pull it out, upgrade to FC9 current (non-rawhide) and see if I have any problems. It's off due to lack of need, not really obsolete, so it's a fair test. O'll put a dew hundred GB of raid-1 and beat on it.

I was going to get back to you all today and let you know what I found. On Thursday, I rebuilt the system with SUSE 11 but before I did I went over all of the BIOS settings. The second IDE drive was set to "NONE" instead of "AUTO". Well, the installation went without the previous hitch of having to manually install grub on the first boot after the install. It has also been running since then without issue.

Is it that simple? Could that be all that was wrong? What doesn't make sense is how 10.3 (kernel 2.6.22) never had an issue. Perhaps without the BIOS reporting the second drive, the later kernel chose the wrong parameters setting it up and they didn't match what was set up by the BIOS for the first drive?

Mike

#18 lore raw
Quoted 8 lines · Mike McCarthy
Quoted 7 lines · Bill Davidsen
Quoted 6 lines · Mike McCarthy
Quoted 3 lines · Bill Davidsen

Given heavy 2.6.25 use, my guess is still that the root cause of this is hardware, and that the change in disk code either triggers the hardware problem, or handles it differently. Are you by any chance running NCQ on your system?

No. This system and the drives pre-date NCQ. I think NCQ is only implemented in SATA and these are IDE drives. Sometime over the weekend, I am going to reload SUSE 11 and try to do some more debugging.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

BTW: It's back to 10.3 (kernel 2.6.22) running happily with a VMware server thrashing away at the disks.

This has recycled back to the top of my todo list, I have a server in mothballs with IDE drives, I'll pull it out, upgrade to FC9 current (non-rawhide) and see if I have any problems. It's off due to lack of need, not really obsolete, so it's a fair test. O'll put a dew hundred GB of raid-1 and beat on it.

I was going to get back to you all today and let you know what I found. On Thursday, I rebuilt the system with SUSE 11 but before I did I went over all of the BIOS settings. The second IDE drive was set to "NONE" instead of "AUTO". Well, the installation went without the previous hitch of having to manually install grub on the first boot after the install. It has also been running since then without issue.

Is it that simple? Could that be all that was wrong? What doesn't make sense is how 10.3 (kernel 2.6.22) never had an issue. Perhaps without the BIOS reporting the second drive, the later kernel chose the wrong parameters setting it up and they didn't match what was set up by the BIOS for the first drive?

Oh well, I needed to upgrade that system, I'll push testing down the week a day or two. It could well have been that simple.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark