Skip to contentSkip to conversation

2.4.22-pre3 and reiserfs boot problem

DISCUSSION Stephan von Krawczynski 5 participants 18 messages Jul 6, 2003 – Jul 10, 2003 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hello,

I just tried 2.4.22-pre3 and found out I cannot boot my test box any more. It halts at:

reiserfs: found format "3.6" with standard journal

on a partition located on aic7xxx based hd.

Booting the box with pre2 works perfectly well. Anything I should try? What information is needed?

Regards,
Stephan

#1 lore raw
Quoted 11 lines · Stephan von Krawczynski

Hello,

I just tried 2.4.22-pre3 and found out I cannot boot my test box any more. It halts at:

reiserfs: found format "3.6" with standard journal

on a partition located on aic7xxx based hd.

Booting the box with pre2 works perfectly well. Anything I should try? What information is needed?

Same config here (reiserfs 3.6.x, aic7xxx drive), works without problems. Is reiserfs or aic7xxx compiled as a module in your setup? If so did you remember to install the new modules and mkinitrd (if required).

Is reiserfs on your root drive? If not can you please boot into single user mode, enable sysrq, try the mount again and get the decoded output from sysrq-p and sysrq-t if it hangs.

-chris

#2 lore raw
Quoted 10 lines · On 06 Jul 2003 14:13:44 -0400 Chris Mason <[email protected]> wrote:
Quoted 9 lines · Stephan von Krawczynski

Hello,

I just tried 2.4.22-pre3 and found out I cannot boot my test box any more. It halts at:

reiserfs: found format "3.6" with standard journal

on a partition located on aic7xxx based hd.

Quoted 7 lines
Quoted 2 lines

Booting the box with pre2 works perfectly well. Anything I should try? What information is needed?

Same config here (reiserfs 3.6.x, aic7xxx drive), works without problems. Is reiserfs or aic7xxx compiled as a module in your setup? If so did you remember to install the new modules and mkinitrd (if required).

Hello Chris,

no, there is no modules involved here. The problem arises not on my root partition (which happens to be reiserfs, too), that seems to be ok. Problems seem to come up on sda4, a simple data partition.

Quoted 5 lines

Is reiserfs on your root drive? If not can you please boot into single user mode, enable sysrq, try the mount again and get the decoded output from sysrq-p and sysrq-t if it hangs.

-chris

I am going to try with single user tomorrow morning.

Regards,
Stephan

#3 lore raw
In reply to Chris Mason

Hello Chris,

I have to correct myself regarding the problem. It does _not_ arise on aic-driven scsi-disk, but on 3ware-driven RAID5 with 320 GB data. I tried to mount it by hand, but that does not work either. It does not look like there is a lot of work going on on the hds while mounting (no LEDs).
Can anyone reproduce with equal setup?
I try to come up with further information...

Signature · 2 lines
Regards,
Stephan
#4 lore raw
In reply to Chris Mason
Quoted 6 lines · On 06 Jul 2003 14:13:44 -0400 Chris Mason <[email protected]> wrote:

[...]
Is reiserfs on your root drive? If not can you please boot into single user mode, enable sysrq, try the mount again and get the decoded output from sysrq-p and sysrq-t if it hangs.

-chris

Hello Chris,

I tried to produce some useful output but failed. Additionals I found:

- pre3-ac1 has the same problem
- the box _hangs_ in fact, no sysrq is working.
(you need hw-reset to revive the box)
- I can see no disk activity on the 3ware RAID in question
- It always shows up, completely reproducable
- It shows during boot and during single- or multiuser (mount from console)

Regards,
Stephan

#5 lore raw
Quoted 12 lines · Stephan von Krawczynski
Quoted 6 lines · On 06 Jul 2003 14:13:44 -0400 Chris Mason <[email protected]> wrote:

[...]
Is reiserfs on your root drive? If not can you please boot into single user mode, enable sysrq, try the mount again and get the decoded output from sysrq-p and sysrq-t if it hangs.

-chris

Hello Chris,

Quoted 8 lines

I tried to produce some useful output but failed. Additionals I found:

- pre3-ac1 has the same problem
- the box _hangs_ in fact, no sysrq is working.
(you need hw-reset to revive the box)
- I can see no disk activity on the 3ware RAID in question
- It always shows up, completely reproducable
- It shows during boot and during single- or multiuser (mount from console)

Step one is to figure out if the problem is reiserfs or 3ware. Instead of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null and see if you still hang.

This will read the FS metadata and should generate enough io to trigger the hang if it is a 3ware problem.

(I'm on vacation for a few days, so Oleg is cc'd)

-chris

#6 lore raw

Hello!

Quoted 7 lines · Chris Mason
Quoted 7 lines
Quoted 3 lines

Is reiserfs on your root drive? If not can you please boot into single user mode, enable sysrq, try the mount again and get the decoded output from sysrq-p and sysrq-t if it hangs.

I tried to produce some useful output but failed. Additionals I found:
- pre3-ac1 has the same problem
- the box _hangs_ in fact, no sysrq is working.
(you need hw-reset to revive the box)

This complete hang is _very_ suspicious. Usually you cannot achieve such results without touching hardware.

Quoted 1 line
Quoted 1 line

- I can see no disk activity on the 3ware RAID in question

After the lockup? This is kind of expected ;)

Quoted 7 lines
Quoted 2 lines

- It always shows up, completely reproducable
- It shows during boot and during single- or multiuser (mount from console)

Step one is to figure out if the problem is reiserfs or 3ware. Instead of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null and see if you still hang.
This will read the FS metadata and should generate enough io to trigger the hang if it is a 3ware problem.

Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get journal replayed. This is in case access pattern matters.

Bye,
Oleg

#7 lore raw
Quoted 6 lines · On Wed, 9 Jul 2003 17:48:37 +0400 Oleg Drokin <[email protected]> wrote:

Hello!

Quoted 3 lines · Chris Mason
Quoted 3 lines
Quoted 3 lines

Is reiserfs on your root drive? If not can you please boot into single user mode, enable sysrq, try the mount again and get the decoded output from sysrq-p and sysrq-t if it hangs.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

I tried to produce some useful output but failed. Additionals I found:
- pre3-ac1 has the same problem
- the box _hangs_ in fact, no sysrq is working.
(you need hw-reset to revive the box)

This complete hang is _very_ suspicious. Usually you cannot achieve such results without touching hardware.

Well, I did a few more tries and it looks like this:
- enter mount command
- short blinking on the RAID disks
- about 1/2 second later box hangs

Quoted 5 lines
Quoted 5 lines

Step one is to figure out if the problem is reiserfs or 3ware. Instead of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null and see if you still hang.
This will read the FS metadata and should generate enough io to trigger the hang if it is a 3ware problem.

Ok, I tried this. debugreiserfs runs without any problems. Disks show quite an activity, the whole thing lasts 1-2 minutes.

mount afterwards shows the same hang.

Quoted 2 lines

Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get journal replayed. This is in case access pattern matters.

I can try that, too. What do you expect to see?

Regards,
Stephan

#8 lore raw

Hello!

Quoted 8 lines · Stephan von Krawczynski
Quoted 5 lines
Quoted 5 lines

Step one is to figure out if the problem is reiserfs or 3ware. Instead of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null and see if you still hang.
This will read the FS metadata and should generate enough io to trigger the hang if it is a 3ware problem.

Ok, I tried this. debugreiserfs runs without any problems. Disks show quite an activity, the whole thing lasts 1-2 minutes.
mount afterwards shows the same hang.

Hm.

Quoted 3 lines
Quoted 2 lines

Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get journal replayed. This is in case access pattern matters.

I can try that, too. What do you expect to see?

Well, it will either hang or not, I think.
It it won't hang, this will complicate matters.
Then next step would be probably to try and mount the partition from usermodelinux if you are able to conduct such a test.
I am still pretty skeptical about the possibility that recent reiserfs changes broke stuff.

Bye,
Oleg

#9 lore raw
Quoted 25 lines · On Wed, 9 Jul 2003 18:11:11 +0400 Oleg Drokin <[email protected]> wrote:

Hello!

Quoted 8 lines · Stephan von Krawczynski
Quoted 5 lines
Quoted 5 lines

Step one is to figure out if the problem is reiserfs or 3ware. Instead of mounting the filesystem, run debugreiserfs -d /dev/xxxx > /dev/null and see if you still hang.
This will read the FS metadata and should generate enough io to trigger the hang if it is a 3ware problem.

Ok, I tried this. debugreiserfs runs without any problems. Disks show quite an activity, the whole thing lasts 1-2 minutes.
mount afterwards shows the same hang.

Hm.

Quoted 3 lines
Quoted 2 lines

Or if this one suceeds, then may be reiserfsck --check /dev/xxxx to get journal replayed. This is in case access pattern matters.

I can try that, too. What do you expect to see?

Well, it will either hang or not, I think.
It it won't hang, this will complicate matters.
Then next step would be probably to try and mount the partition from usermodelinux if you are able to conduct such a test.
I am still pretty skeptical about the possibility that recent reiserfs changes broke stuff.

Quoted 2 lines

Bye,
Oleg

ok, I did the reiserfsck and it works flawlessly. No errors no problems no hang.
I tried mount afterwards and it still hangs.
Is there some recent change around the mount procedure itself? maybe it is really unrelated to reiserfs and 3ware...

Regards,
Stephan

PS to Marcelo:
There is a problem with 2.4.22-pre3. I cannot mount a reiserfs data-partition with 320 GB size located on a 3ware RAID. It just hangs the box, during init or any runlevel I tried. It is completely reproducable, but debugreiserfs on the partition and reiserfsck both show no problems at all ...
The things mounts flawlessly under 2.4.22-pre2 and below.

#10 lore raw
Quoted 10 lines · Stephan von Krawczynski

On Wed, 9 Jul 2003 18:11:11 +0400
ok, I did the reiserfsck and it works flawlessly. No errors no problems no hang.
I tried mount afterwards and it still hangs.
Is there some recent change around the mount procedure itself? maybe it is really unrelated to reiserfs and 3ware...

Regards,
Stephan

Quoted 6 lines

PS to Marcelo:
There is a problem with 2.4.22-pre3. I cannot mount a reiserfs data-partition with 320 GB size located on a 3ware RAID. It just hangs the box, during init or any runlevel I tried. It is completely reproducable, but debugreiserfs on the partition and reiserfsck both show no problems at all ...
The things mounts flawlessly under 2.4.22-pre2 and below.

There are no 3ware changes in pre3. So it must be reiserfs or something else. Lets try reverting the reiserfs patches to see if they are the cause?

Attached are files rei1, rei2, and rei3 (all gzip compressed).

They are the three reiserfs changesets which have been included in -pre3.

Can you please revert them (with patch -R) and try to reproduce the problem?

Thanks

#11 lore raw
Quoted 7 lines · On Wed, 9 Jul 2003 14:18:37 -0300 (BRT) Marcelo Tosatti…
Quoted 6 lines

PS to Marcelo:
There is a problem with 2.4.22-pre3. I cannot mount a reiserfs data-partition with 320 GB size located on a 3ware RAID. It just hangs the box, during init or any runlevel I tried. It is completely reproducable, but debugreiserfs on the partition and reiserfsck both show no problems at all ... The things mounts flawlessly under 2.4.22-pre2 and below.

Quoted 5 lines

There are no 3ware changes in pre3. So it must be reiserfs or something else. Lets try reverting the reiserfs patches to see if they are the cause?

Attached are files rei1, rei2, and rei3 (all gzip compressed).

I reverted all three patches and the problem stays just the same. I guess this makes it a lot likely that the problem lies elsewhere.

If you want me to try others, just send them to me like these three.

Thanks for your support
Stephan

#12 lore raw
Quoted 14 lines · Stephan von Krawczynski
Quoted 11 lines · On Wed, 9 Jul 2003 14:18:37 -0300 (BRT) Marcelo Tosatti…
Quoted 6 lines

PS to Marcelo:
There is a problem with 2.4.22-pre3. I cannot mount a reiserfs data-partition with 320 GB size located on a 3ware RAID. It just hangs the box, during init or any runlevel I tried. It is completely reproducable, but debugreiserfs on the partition and reiserfsck both show no problems at all ... The things mounts flawlessly under 2.4.22-pre2 and below.

There are no 3ware changes in pre3. So it must be reiserfs or something else. Lets try reverting the reiserfs patches to see if they are the cause?

Quoted 6 lines
Quoted 1 line

Attached are files rei1, rei2, and rei3 (all gzip compressed).

I reverted all three patches and the problem stays just the same. I guess this makes it a lot likely that the problem lies elsewhere.

If you want me to try others, just send them to me like these three.

Stephan,

First of all, thanks a lot for your help. Not everyone is willing to debug/test problems like you do. This is very important for us.

Well, we now know reiserfs patches in 2.4.22-pre are not the problem.

2.4.21 is OK (does not crash when mounting) correct?

#13 lore raw
Quoted 10 lines · On Thu, 10 Jul 2003 08:54:02 -0300 (BRT) Marcelo Tosatti…
Quoted 1 line

If you want me to try others, just send them to me like these three.

Stephan,

First of all, thanks a lot for your help. Not everyone is willing to debug/test problems like you do. This is very important for us.

Well, we now know reiserfs patches in 2.4.22-pre are not the problem.

2.4.21 is OK (does not crash when mounting) correct?

Everything up to and including 2.4.22-pre2 is working without problems. Under 2.4.22-pre3 I can reiserfsck the partition and see i/o going on for minutes without troubles.
But I cannot mount it, the box completely hangs about 1 second after mount try. I see the drive leds blinking shortly and that's it.

Regards,
Stephan

#14 lore raw
In reply to Marcelo Tosatti
Quoted 4 lines · On Wed, 9 Jul 2003 14:18:37 -0300 (BRT) Marcelo Tosatti…

Can you please revert them (with patch -R) and try to reproduce the problem?

Thanks

In addition I can tell you that the problem is also visible with -pre4

Regards,
Stephan

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw
Quoted 7 lines · On Wed, 9 Jul 2003 18:40:15 +0300 Ville Herva <[email protected]> wrote:
Quoted 5 lines
Quoted 1 line

ok, I did the reiserfsck and it works flawlessly. No errors no problems no

hang.
I tried mount afterwards and it still hangs.
Is there some recent change around the mount procedure itself? maybe it is really unrelated to reiserfs and 3ware...

Is it just this partition or any reiserfs fs on 3ware?

Hm, unfortunately I can't tell, I have no other partition available on 3ware
...

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines · Oleg Drokin

Then next step would be probably to try and mount the partition from usermodelinux if you are able to conduct such a test.

It it possible to mount raw partitions with UML?

Hm, I never tried UML. I really wonder if there is nobody else with 3ware and reiserfs available for re-checking 2.4.22-pre3. Only to see if this is a singular problem or reproducable elsewhere.

Regards,
Stephan

#17 lore raw
In reply to Stephan von Krawczynski
Quoted 11 lines · Stephan von Krawczynski

I tried to produce some useful output but failed. Additionals I found:

- pre3-ac1 has the same problem
- the box _hangs_ in fact, no sysrq is working.
(you need hw-reset to revive the box)
- I can see no disk activity on the 3ware RAID in question
- It always shows up, completely reproducable
- It shows during boot and during single- or multiuser (mount from console)

Regards,
Stephan

Which mainboard do you use ?
I'm having endless pain with a 3ware raid and Tyan mainboards, so much I really really want to replace it with anything else that is stable (probably a single Intel P4).

v

#18 lore raw
Quoted 6 lines · On Wed, 9 Jul 2003 16:01:40 +0200 Vincent Touquet <[email protected]>…

Which mainboard do you use ?
I'm having endless pain with a 3ware raid and Tyan mainboards, so much I really really want to replace it with anything else that is stable (probably a single Intel P4).

v

Sorry, the problem only arises with 2.4.22-pre3, and nothing before. So it is definitely software-related.

Regards,
Stephan