2.4.22-pre3 and reiserfs boot problem

Having read the whole thread, I came up with a few ideas.

The following patches in -pre3 could perhaps have to do something with your problems:
o (resend) collected semaphore fixes and semtimedop
o Fix potential IO hangs and increase interactiveness during heavy IO
o fix false sharing of mm info
o fix up semops and return, allow timedop
o mremap VM_LOCKED move_vma
o remove io_apic_modify - this doesnt work on some APICs
o small setup-pci cleanups

Since your hangs are not even traceable with SysRq, please try to boot with nmi_watchdog=1 and if that doesn't work (dmesg will complain) nmi_watchdog=2. About 15 seconds after the hang your box should print a backtrace.

As a last resort you could mount the fs from UML.

I suggest you try the nmi_watchdog thing first.

Carl-Daniel

http://www.hailfinger.org/

I have currently nmi_watchdog=1 which works (NMI interrupts show up during normal operation), but there is no backtrace visible or producable during the freeze, sorry.

Regards,
Stephan

How is that possible? If the NMI watchdog works but doesn't fire, the lockup should respond to SysRq-T. Could you please try SysRq-T *before* the hang just to verify that it would work?

Regards,
Carl-Daniel

http://www.hailfinger.org/

Well, the thing I don't really understand about SysRq-X is that the output is visible by dmesg-command, but I cannot see anything on the console (single user tested). This means it may well work in the hang-case, but as I cannot execute dmesg I will never see it...

Regards,
Stephan

Hi,

Apologies for chipping in, but I saw something similar to what was described in the thread. I'm running 2.4.22-pre3-ac1 with the FreeS/WAN 2.0.1 patches and noticed last night that when booting this kernel, if an ext3 filesystem had exceeded its mount count and required checking, the e2fsck process would hang sometime during the fsck and the system would become unresponsive, but SysRq would still work. Alt-SysRq-P would show e2fsck and some register details. I did not note them down, but booting 2.4.21-rc7-ac1 and letting that kernel check the filesystem would work. Booting back into 2.4.22-pre3-ac1 would then also work.

This might or might not be related to the original problem. I do use nmi_watchdog=1, NMI count is 1 presently, so I guess that works. The ram is memtested, so that is not an issue, heavy filesystem usage works normally, it was just e2fsck that would not work. I have not tried -pre2 or -pre4 yet, but that is on the cards.

If there is anything I can try, let me know.

Anders Karlsson <[email protected]>
Trudheim Technology Limited

Was there any disk activity after it became unresponsive? If not, please provide a (partially) decoded SysRq-T. I'm only interested in the decoded stack trace of the hung process (it should have a "D" after the process name).

Carl-Daniel

http://www.hailfinger.org/

Right, I have collected the details that was asked for. Carl-Daniel suggested a way of tricking the system into doing an fsck without tampering with the filesystem itself. Please find below the data I copied out and decoded. If there is anything else I can do, let me know. If there is any data about my system you require, let me know.

Output from Alt SysRq P:

Pid: 160, comm: fsck.ext3
EIP: 0010:[<c01b0f43>] CPU: 0 EFLAGS: 00000246 Not Tainted
EAX: 00000000 EBX: 00000000 ECX: c0338c00 EDX: 00000007
ESI: c0338c00 EDI: eee68000 EBP: eee69de8 DS: 0018 ES: 0018
CR0: 80050033 CR2: 400dcff0 CR3: 2ee6b000 CR4: 000006d0

Call trace: [<c01b1584>] [<c01b1bfb>] [<c01b1cca>] [<c0141643>]

[<c014178f>] [<c01417a4>] [<c0141887>] [<c01464e1>] [<c0141b61>] [<c0108f83>]

Output from Alt SysRq T:

fsck.ext3 D current 3808 160 149 (NOTLB)

Call Trace: [<c014178f>] [<c01417a4>] [<c0141887>] [<c01464e1>]

[<c0141b61>] [<c0108f83>]

ksymoops 2.4.8 on i686 2.4.22-pre3-ac1. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.22-pre3-ac1/ (default) -m /boot/System.map-2.4.22-pre3-ac1 (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

Warning (compare_maps): mismatch on symbol zeroes , ipsec says f23fdf00, /lib/modules/2.4.22-pre3-ac1/kernel/net/ipsec/ipsec.o says f23fdde0. Ignoring /lib/modules/2.4.22-pre3-ac1/kernel/net/ipsec/ipsec.o entry
Pid: 160, comm: fsck.ext3
EIP: 0010:[<c01b0f43>] CPU: 0 EFLAGS: 00000246 Not Tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EAX: 00000000 EBX: 00000000 ECX: c0338c00 EDX: 00000007
ESI: c0338c00 EDI: eee68000 EBP: eee69de8 DS: 0018 ES: 0018 Warning (Oops_set_regs): garbage 'DS: 0018 ES: 0018' at end of register line ignored
CR0: 80050033 CR2: 400dcff0 CR3: 2ee6b000 CR4: 000006d0

Call trace: [<c01b1584>] [<c01b1bfb>] [<c01b1cca>] [<c0141643>]

[<c014178f>] [<c01417a4>] [<c0141887>] [<c01464e1>] [<c0141b61>] [<c0108f83>]
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; c01b1584 <__make_request+167/71a>
Trace; c01b1bfb <generic_make_request+c4/13a>
Trace; c01b1cca <submit_bh+59/a0>
Trace; c0141643 <write_locked_buffers+2a/36>
Trace; c014178f <write_some_buffers+140/142>
Trace; c01417a4 <write_unlocked_buffers+13/1d>
Trace; c0141887 <sync_buffers+1a/75>
Trace; c01464e1 <__block_fsync+2f/6c>
Trace; c0141b61 <sys_fsync+93/de>
Trace; c0108f83 <system_call+33/38>

4 warnings issued. Results may not be reliable.

Anders Karlsson <[email protected]>
Trudheim Technology Limited