How to send a break?
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on the serial line, using echo.
I mean like this:
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0
Thanks,
Janos
Quoted 5 lines · Haar János
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on the serial line, using echo.
Quoted 7 lines
I mean like this:
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0
Is there a reason you can't use "echo -n m > /proc/sysrq-trigger"?
Jim.
Re: How to send a break? - dump from frozen 64bit linux
----- Original Message -----
From: "Jim Crilly" <[email protected]>
To: "Haar János" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 28, 2006 1:43 AM
Subject: Re: How to send a break?
Quoted 4 lines · Haar János
Quoted 3 lines
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on
the
Quoted 11 lines
Quoted 9 lines
serial line, using echo.
I mean like this:
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0Is there a reason you can't use "echo -n m >
/proc/sysrq-trigger"?
Yes, i want to dump my frequently frozen remote server, automatically, if it is possible. (using null-modem cable, and another server.)
Anyway, i made it this time by hand.
Here is the dump:
http://download.netcenter.hu/bughunt/20060528/log.txt (400KB!)
Can somebody tell me, what is wrong exactly?
Anyway, i interested about, how can i -a single user- interpret these dump to made error reporting more useful?
The problem:
I used one stable system on 32bit, but i need to switch to X86_64, because the nbd cannot use >2TB devices on 32bit.
I have reinstall the RH 9.0 to FC 5.0, and recompiled everything what i used to serving. (the OS is only for external tasks.)
But on 64bit, my system becomes unstable.
Sometimes it is frozen, but no error message at all!
I can see, the crond-jobs is hangs too, but syslog can post messages, but no valuable at all.
(I cannot login with ssh.)
Thats why i need to dump with serial console.
Can somebody help me? :-)
Here is the "normal" dmesg message:
http://download.netcenter.hu/bughunt/20060528/dmesg.txt (21KB)
Thanks,
Janos
Quoted 7 lines
Jim.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: How to send a break? - dump from frozen 64bit linux
Quoted 3 lines · Haar János
Can somebody tell me, what is wrong exactly?
I can't tell you exactely what's wrong unfortunately, but after looking at your dump & dmesg I notice two things that might be worth trying to change :
1) You seem to be running without any swap space at all. I't usually a good idea always to have some swap configured - try adding a swap file.
(note: I don't think this will help with your current problem, it's just a good thing to do generally).
2) You should try the latest stable kernel. Currently that's 2.6.16.18
(http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.18.tar.bz2).
There have been lots of fixes added since 2.6.15.x and perhaps you are lucky that whatever is giving you trouble has already been fixed in that kernel.
Quoted 3 lines
Anyway, i interested about, how can i -a single user- interpret these dump to made error reporting more useful?
You can find some info in Documentation/sysrq.txt & Documentation/oops-tracing.txt .
As for posting good error/bug reports, please read the REPORTING-BUGS file in the root of the kernel source dir.
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Re: How to send a break? - dump from frozen 64bit linux
----- Original Message -----
From: "Jesper Juhl" <[email protected]>
To: "Haar János" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 28, 2006 6:17 PM
Subject: Re: How to send a break? - dump from frozen 64bit linux
Quoted 8 lines · Haar János
Quoted 3 lines
Can somebody tell me, what is wrong exactly?
I can't tell you exactely what's wrong unfortunately, but after looking at your dump & dmesg I notice two things that might be worth trying to change :
Quoted 5 lines
1) You seem to be running without any swap space at all. I't usually a good idea always to have some swap configured - try adding a swap file.
(note: I don't think this will help with your current problem, it's just a good thing to do generally).
Thanks for the idea!
I did thinking of it allready, but dropped, because:
I can only use swap _file_ in this config, and swapping into file is relatively slow.
I am affraid, it will be slow down this system for some cases. The system (programs) is relatively small next to the used buffers/caches, and the kernel will swap out the rarely used programs to be able free up memory for caching.
I think this is not too good idea on this system, what have allready 4GB of memory.
The minimum free space is changeable thanks to VM.
Are you sure?
Quoted 6 lines
2) You should try the latest stable kernel. Currently that's 2.6.16.18
(http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.18.tar.bz2).
There have been lots of fixes added since 2.6.15.x and perhaps you are lucky that whatever is giving you trouble has already been fixed in that kernel.
Hmm.
Last time, when i try the 2.6.16.x, i have lost close to 4000 users home, and documents on XFS filesystem!
(a lot of directory have renamed to "/*" like this one: "/ost+found" in the root.)
I don't want to try it again! :-)
Quoted 3 lines
Quoted 3 lines
Anyway, i interested about, how can i -a single user- interpret these
dump
Quoted 6 lines
Quoted 2 lines
to made error reporting more useful?
You can find some info in
Documentation/sysrq.txt&Documentation/oops-tracing.txt.
As for posting good error/bug reports, please read the REPORTING-BUGS file in the root of the kernel source dir.
Thanks, i will read this.
Anyway, this 64bit hanging issue is reproducible on my system. (normally about daily, but if i try to trigger it, can be about 3-4x daily.) If somebody is interested, please let me know, and i will send any useful infos to debugging this! :-)
Cheers,
Janos
Quoted 11 lines
Signature · 9 lines
-- Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How to send a break? - dump from frozen 64bit linux
On 28/05/06, Haar János <[email protected]> wrote:
[snip]
Quoted 2 lines
I can only use swap _file_ in this config, and swapping into file is relatively slow.
Not so. With a 2.4.x kernel swap files were slower than swap partitions, but with the 2.6 kernel a swap file is just as fast as a swap partition.
[snip]
Quoted 7 lines
Quoted 6 lines
2) You should try the latest stable kernel. Currently that's 2.6.16.18
(http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.18.tar.bz2).
There have been lots of fixes added since 2.6.15.x and perhaps you are lucky that whatever is giving you trouble has already been fixed in that kernel.
Quoted 7 lines
Hmm.
Last time, when i try the 2.6.16.x, i have lost close to 4000 users home, and documents on XFS filesystem!
(a lot of directory have renamed to "/*" like this one: "/ost+found" in the root.)
I don't want to try it again! :-)
That sounds like a pretty serious bug.
Are you sure it was caused by the kernel?
Did you report the bug to LKML & the XFS maintainers so it can get fixed?
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Re: How to send a break? - dump from frozen 64bit linux
Quoted 9 lines · Jesper Juhl
On 28/05/06, Haar János <[email protected]> wrote:
[snip]Quoted 2 lines
I can only use swap _file_ in this config, and swapping into file is relatively slow.
Not so. With a 2.4.x kernel swap files were slower than swap partitions, but with the 2.6 kernel a swap file is just as fast as a swap partition.
what made this possible? any pointers please?
Re: How to send a break? - dump from frozen 64bit linux
In reply to Haar János[cut]
Quoted 8 lines
Quoted 7 lines
2) You should try the latest stable kernel. Currently that's 2.6.16.18
(http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.16.18.tar.bz2).
There have been lots of fixes added since 2.6.15.x and perhaps you are lucky that whatever is giving you trouble has already been fixed in that kernel.
This time i try the 2.6.16.18 kernel, but the issue is the same!
Here is the logs:
http://download.netcenter.hu/bughunt/20060530/dump.txt (The frozen system, 540KB)
http://download.netcenter.hu/bughunt/20060530/261618-good.txt (After reboot, the working system, 300KB, uptime 54 min)
http://download.netcenter.hu/bughunt/20060530/dmesg.txt (The boot dmesg file)
Can somebody tell me, whats wrong?
It seems like some part of the fs died.
(The "top", "watch df" hangs on the ssh window, in the "mc" the line is moving, but if i try to step in/out from/to dir, it hangs too, ping reply is working. )
I use only 3 fs:
- the root FS on NFS.
- one XFS mount point from sata drive (200GB)
- one huge XFS mount point from NBD. (14TB)
Cheers,
Janos
Re: How to send a break? - dump from frozen 64bit linux
Quoted 4 lines · Janos Haar
http://download.netcenter.hu/bughunt/20060530/dump.txt (The frozen system, 540KB)
Can somebody tell me, whats wrong?
kblockd/1 D ffff81011f641778 0 25 19 26 24 (L-TLB)ffff81011f641778 0000000000000000 0000000000000009 ffff81011f735358
ffff81011f735140 ffff81011fc79100 000014a00f9a0ef2 00000000000410dd
0000000102866d40 ffff810003900280
Call Trace: <ffffffff8026d72a>{xfs_qm_shake+135} <ffffffff804e6046>{__mutex_lock_slowpath+424}
<ffffffff804e62e4>{mutex_lock+41} <ffffffff8026d72a>{xfs_qm_shake+135}
<ffffffff80157cfd>{shrink_slab+100} <ffffffff801584d9>{try_to_free_pages+372}
<ffffffff80153c3f>{__alloc_pages+432} <ffffffff8046aef3>{tcp_sendmsg+1373}
<ffffffff804848ad>{inet_sendmsg+70} <ffffffff8043f619>{sock_sendmsg+270}
<ffffffff8013d3e0>{autoremove_wake_function+0} <ffffffff80440db3>{kernel_sendmsg+61}
<ffffffff8802c111>{:nbd:sock_xmit+273} <ffffffff8015195d>{mempool_alloc_slab+17}
<ffffffff80169b1b>{poison_obj+39} <ffffffff8015195d>{mempool_alloc_slab+17}
<ffffffff80169c11>{cache_alloc_debugcheck_after+235}
<ffffffff8015195d>{mempool_alloc_slab+17} <ffffffff802da471>{as_remove_queued_request+267}
<ffffffff8802c472>{:nbd:nbd_send_req+517} <ffffffff8802c712>{:nbd:do_nbd_request+329}
<ffffffff802d9b45>{as_work_handler+46} <ffffffff80139d30>{run_workqueue+168}
<ffffffff802d9b17>{as_work_handler+0} <ffffffff8013a27f>{worker_thread+0}
<ffffffff8013a383>{worker_thread+260} <ffffffff80123fa4>{default_wake_function+0}
<ffffffff8013a27f>{worker_thread+0} <ffffffff8013d29f>{kthread+219}
<ffffffff8012590d>{schedule_tail+70} <ffffffff8010bba6>{child_rip+8}
<ffffffff8013d1c4>{kthread+0} <ffffffff8010bb9e>{child_rip+0}Half the processes on the box seem wedged at that same mutex_lock. I can't seem to find an xfs_qm_shake in my source tree though.
Re: How to send a break? - dump from frozen 64bit linux
Quoted 3 lines · Janos Haar
Quoted 2 lines
http://download.netcenter.hu/bughunt/20060530/dump.txt (The frozen
system,
Quoted 5 lines
Quoted 3 lines
540KB)
Can somebody tell me, whats wrong?
kblockd/1 D
ffff81011f6417780 25 19 26 24
(L-TLB)
Quoted 4 lines
ffff81011f6417780000000000000000 0000000000000009ffff81011f735358ffff81011f735140 ffff81011fc79100 000014a00f9a0ef2 00000000000410dd 0000000102866d40 ffff810003900280 Call Trace: <ffffffff8026d72a>{xfs_qm_shake+135}
<ffffffff804e6046>{__mutex_lock_slowpath+424}
Quoted 1 line
<ffffffff804e62e4>{mutex_lock+41}<ffffffff8026d72a>{xfs_qm_shake+135}
Quoted 1 line
<ffffffff80157cfd>{shrink_slab+100}<ffffffff801584d9>{try_to_free_pages+372}
Quoted 1 line
<ffffffff80153c3f>{__alloc_pages+432}<ffffffff8046aef3>{tcp_sendmsg+1373}
Quoted 1 line
<ffffffff804848ad>{inet_sendmsg+70}<ffffffff8043f619>{sock_sendmsg+270}
Quoted 1 line
<ffffffff8013d3e0>{autoremove_wake_function+0}<ffffffff80440db3>{kernel_sendmsg+61}
Quoted 1 line
<ffffffff8802c111>{:nbd:sock_xmit+273}<ffffffff8015195d>{mempool_alloc_slab+17}
Quoted 1 line
<ffffffff80169b1b>{poison_obj+39}<ffffffff8015195d>{mempool_alloc_slab+17}
Quoted 2 lines
<ffffffff80169c11>{cache_alloc_debugcheck_after+235}
<ffffffff8015195d>{mempool_alloc_slab+17}<ffffffff802da471>{as_remove_queued_request+267}
Quoted 1 line
<ffffffff8802c472>{:nbd:nbd_send_req+517}<ffffffff8802c712>{:nbd:do_nbd_request+329}
Quoted 1 line
<ffffffff802d9b45>{as_work_handler+46}<ffffffff80139d30>{run_workqueue+168}
Quoted 1 line
<ffffffff802d9b17>{as_work_handler+0}<ffffffff8013a27f>{worker_thread+0}
Quoted 1 line
<ffffffff8013a383>{worker_thread+260}<ffffffff80123fa4>{default_wake_function+0}
Quoted 2 lines
<ffffffff8013a27f>{worker_thread+0} <ffffffff8013d29f>{kthread+219}
<ffffffff8012590d>{schedule_tail+70}<ffffffff8010bba6>{child_rip+8}
Quoted 4 lines
<ffffffff8013d1c4>{kthread+0} <ffffffff8010bb9e>{child_rip+0}Half the processes on the box seem wedged at that same
mutex_lock. I can't seem to find anxfs_qm_shakein my source tree though.
The XFS, what i use is the default on the 2.6.16.18.
Anyway, the 2.6.16.18 is unpatched, i can use it from the original source. This kernel already know what i need.
I only use this external module:
e1000-7.0.33
The XFS parts:
acl-2.2.34
attr-2.4.28
dmapi-2.2.3
xfsdump-2.2.33
xfsprogs-2.7.11
Sorry, but i cannot understand the mutex and lock, this is bad thing in this dump? :-)
Anyway, this issue since then i step from i686 to X86_64!
I have upgrade the OS from rh9.0 to FC 5, and i had recompile to 64bit ALL of the services, what i need to use.
(Kernel, apache, mysql+lib+client,pure-ftpd, nbd-client, xfs, php+libs)
Why doing this?
Because i need to use >2TB nbd-devices, and the nbd-client refused to use them on 32bit. :-(
After i upgrade to X86_64, i have upgrade my huge device from 8TB to 14TB. (And on XFS, there is no way to shrink back...)
Another useful info:
If this issue happens, i always use the reset button, or the sysreq-boot with the serial cable.
During the reboot, the rc script runs the xfs_repair on my 2 general device, and i can see, both are clean!
This shows, the kernel can flush the buffers using nbd, and sata (libata)! The NFS-ROOT can not to be unclean. :-)
Cheers,
Janos
Re: How to send a break? - dump from frozen 64bit linux
In reply to [email protected]Added all those listed in the MAINTAINERS file for XFS.
Quoted 29 lines · On Tue, 2006-05-30 at 15:03 -0400, [email protected] wrote:
Quoted 4 lines · Janos Haar
http://download.netcenter.hu/bughunt/20060530/dump.txt (The frozen system, 540KB)
Can somebody tell me, whats wrong?
kblockd/1 D
ffff81011f6417780 25 19 26 24 (L-TLB)ffff81011f6417780000000000000000 0000000000000009ffff81011f735358ffff81011f735140 ffff81011fc79100 000014a00f9a0ef2 00000000000410dd 0000000102866d40 ffff810003900280 Call Trace: <ffffffff8026d72a>{xfs_qm_shake+135} <ffffffff804e6046>{__mutex_lock_slowpath+424} <ffffffff804e62e4>{mutex_lock+41} <ffffffff8026d72a>{xfs_qm_shake+135} <ffffffff80157cfd>{shrink_slab+100} <ffffffff801584d9>{try_to_free_pages+372} <ffffffff80153c3f>{__alloc_pages+432} <ffffffff8046aef3>{tcp_sendmsg+1373} <ffffffff804848ad>{inet_sendmsg+70} <ffffffff8043f619>{sock_sendmsg+270} <ffffffff8013d3e0>{autoremove_wake_function+0} <ffffffff80440db3>{kernel_sendmsg+61} <ffffffff8802c111>{:nbd:sock_xmit+273} <ffffffff8015195d>{mempool_alloc_slab+17} <ffffffff80169b1b>{poison_obj+39} <ffffffff8015195d>{mempool_alloc_slab+17} <ffffffff80169c11>{cache_alloc_debugcheck_after+235} <ffffffff8015195d>{mempool_alloc_slab+17} <ffffffff802da471>{as_remove_queued_request+267} <ffffffff8802c472>{:nbd:nbd_send_req+517} <ffffffff8802c712>{:nbd:do_nbd_request+329} <ffffffff802d9b45>{as_work_handler+46} <ffffffff80139d30>{run_workqueue+168} <ffffffff802d9b17>{as_work_handler+0} <ffffffff8013a27f>{worker_thread+0} <ffffffff8013a383>{worker_thread+260} <ffffffff80123fa4>{default_wake_function+0} <ffffffff8013a27f>{worker_thread+0} <ffffffff8013d29f>{kthread+219} <ffffffff8012590d>{schedule_tail+70} <ffffffff8010bba6>{child_rip+8} <ffffffff8013d1c4>{kthread+0} <ffffffff8010bb9e>{child_rip+0}
Quoted 2 lines
Half the processes on the box seem wedged at that same
mutex_lock. I can't seem to find anxfs_qm_shakein my source tree though.
What everyone is waiting for is being blocked here:
kswapd0 D ffff81011fe03c38 0 297 1 1287 19 (L-TLB)ffff81011fe03c38 0000000000000004 000000000000000a ffff81011f92ba68
ffff81011f92b850 ffffffff805a23a0 0000149f99fa7d7c 000000000003bcde
000000002f2c46e0 ffff81008bc37180
Call Trace: <ffffffff804e5522>{schedule_timeout+34}
<ffffffff80269f87>{xfs_qm_dqunpin_wait+220} <ffffffff80140e74>{debug_mutex_free_waiter+141}
<ffffffff80123fa4>{default_wake_function+0} <ffffffff80268ca5>{xfs_qm_dqflush+70}
<ffffffff8026d7a7>{xfs_qm_shake+260} <ffffffff80157cfd>{shrink_slab+100}
<ffffffff8015801e>{balance_pgdat+559} <ffffffff801582e8>{kswapd+283}
<ffffffff8013d3e0>{autoremove_wake_function+0} <ffffffff804e6a80>{_spin_unlock_irq+9}
<ffffffff8012590d>{schedule_tail+70} <ffffffff8010bba6>{child_rip+8}
<ffffffff801581cd>{kswapd+0} <ffffffff8010bb9e>{child_rip+0}Seems that the kswapd0 has the lock in questing and has put itself to sleep waiting to be woken up. I don't know the xfs code very well, but the kswapd0 seems to be in this function:
xfs_qm_dqunpin_wait(
xfs_dquot_t *dqp)
{
SPLDECL(s);
ASSERT(XFS_DQ_IS_LOCKED(dqp));
if (dqp->q_pincount == 0) {
return;
}
/*
* Give the log a push so we don't wait here too long.
*/
xfs_log_force(dqp->q_mount, (xfs_lsn_t)0, XFS_LOG_FORCE);
s = XFS_DQ_PINLOCK(dqp);
if (dqp->q_pincount == 0) {
XFS_DQ_PINUNLOCK(dqp, s);
return;
}
sv_wait(&(dqp->q_pinwait), PINOD,
&(XFS_DQ_TO_QINF(dqp)->qi_pinlock), s);
}Where sv_wait is:
#define sv_wait(sv, pri, lock, s) \
_sv_wait(sv, lock, TASK_UNINTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT)
And our macro hell goes further ...
static inline void _sv_wait(sv_t *sv, spinlock_t *lock, int state,
unsigned long timeout)
{
DECLARE_WAITQUEUE(wait, current);
add_wait_queue_exclusive(&sv->waiters, &wait);
__set_current_state(state);
spin_unlock(lock);
schedule_timeout(timeout);
remove_wait_queue(&sv->waiters, &wait);
}So it is now waiting to be woken up by something that calls:
xfs_qm_dquot_logitem_unpin which seems to be the function to wake it up.
And decyphering all the macro crap it seems that the function that wakes it up is xfs_trans_chunk_committed, or xfs_trans_uncommit.
The above xfs_qm_dqunpin_wait still looks awfully racy, and the xfs_log_force, which I'm assuming wakes up whoever is suppose to wake up kswapd0, doesn't have a return code check. So if it failed to do whatever the hell it's doing (that code gives me a headache), it looks like this guy might sleep forever holding a lock that will prevent others from freeing kernel memory.
Well that's about all I can figure out.
Good luck,
-- Steve
XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 1 line · Steven Rostedt
Added all those listed in the
MAINTAINERSfile for XFS.
Thanks Steve.
Quoted 4 lines · On Tue, 2006-05-30 at 15:03 -0400, [email protected] wrote:
Quoted 3 lines · Janos Haar
Half the processes on the box seem wedged at that same
mutex_lock. I can't seem to find anxfs_qm_shakein my source tree though.
Its in fs/xfs/quota/xfs_qm.c.
Quoted 6 lines
kswapd0 D
ffff81011fe03c380 297 1 1287 19 (L-TLB)ffff81011fe03c380000000000000004000000000000000affff81011f92ba68ffff81011f92b850 ffffffff805a23a0 0000149f99fa7d7c 000000000003bcde 000000002f2c46e0 ffff81008bc37180 Call Trace: <ffffffff804e5522>{schedule_timeout+34} <ffffffff80269f87>{xfs_qm_dqunpin_wait+220} <ffffffff80140e74>{debug_mutex_free_waiter+141}
So, we're waiting here on a synchronisation variable that'll be released once the dquot metadata buffer write completes.
Quoted 4 lines
So it is now waiting to be woken up by something that calls:
xfs_qm_dquot_logitem_unpinwhich seems to be the function to wake it up.
Mhmm, that'd be called by the I/O completion handler on the buffer containing that dquot.
Quoted 2 lines
And decyphering all the macro crap it seems that the function that wakes it up is
xfs_trans_chunk_committed, orxfs_trans_uncommit.
Right (the former, at this point in the code).
Quoted 3 lines
The above
xfs_qm_dqunpin_waitstill looks awfully racy, and thexfs_log_force, which I'm assuming wakes up whoever is suppose to wake up kswapd0, doesn't have a return code check. So if it failed to do
The logforce isn't race-critical here - its ensuring writeout of previously logged buffers is started before we go to sleep waiting for the driver to wake us up when its done.
An earlier I/O error on the journal is the only thing the log force can return as an error there, which isnt useful at that point anyway (we're in a kernel thread trying to free mem).
Quoted 1 line
whatever the hell it's doing (that code gives me a headache), it looks
Heh, likewise. I have voodoo dolls of one or two of the early XFS folks that I like to poke with needles occasionally.. :)
Quoted 2 lines
like this guy might sleep forever holding a lock that will prevent others from freeing kernel memory.
It will sleep until the previously initiated buffer write is done. AFAICT, we aren't seeing the I/O completion here for some reason... which points more to a possible device driver or h/ware issue (that is the usual root cause of this sort of hang, anyway).
cheers.
Signature · 1 line
Nathan
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
----- Original Message -----
From: "Nathan Scott" <[email protected]>
To: "Janos Haar" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Wednesday, May 31, 2006 6:38 AM
Subject: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 8 lines · Steven Rostedt
Quoted 1 line
Added all those listed in the
MAINTAINERSfile for XFS.Thanks Steve.
Quoted 3 lines · On Tue, 2006-05-30 at 15:03 -0400, [email protected] wrote:
Quoted 2 lines · Janos Haar
Half the processes on the box seem wedged at that same
mutex_lock. I
can't
Quoted 5 lines
Quoted 1 line
Quoted 1 line
seem to find an
xfs_qm_shakein my source tree though.Its in
fs/xfs/quota/xfs_qm.c.Quoted 1 line
kswapd0 D
ffff81011fe03c380 297 1 1287 19
(L-TLB)
Quoted 2 lines
Quoted 2 lines
ffff81011fe03c380000000000000004000000000000000affff81011f92ba68ffff81011f92b850 ffffffff805a23a0 0000149f99fa7d7c
000000000003bcde
Quoted 3 lines
Quoted 3 lines
000000002f2c46e0 ffff81008bc37180 Call Trace: <ffffffff804e5522>{schedule_timeout+34} <ffffffff80269f87>{xfs_qm_dqunpin_wait+220}
<ffffffff80140e74>{debug_mutex_free_waiter+141}
Quoted 37 lines
So, we're waiting here on a synchronisation variable that'll be released once the dquot metadata buffer write completes.
Quoted 4 lines
So it is now waiting to be woken up by something that calls:
xfs_qm_dquot_logitem_unpinwhich seems to be the function to wake it up.Mhmm, that'd be called by the I/O completion handler on the buffer containing that dquot.
Quoted 2 lines
And decyphering all the macro crap it seems that the function that wakes it up is
xfs_trans_chunk_committed, orxfs_trans_uncommit.Right (the former, at this point in the code).
Quoted 3 lines
The above
xfs_qm_dqunpin_waitstill looks awfully racy, and thexfs_log_force, which I'm assuming wakes up whoever is suppose to wake up kswapd0, doesn't have a return code check. So if it failed to doThe logforce isn't race-critical here - its ensuring writeout of previously logged buffers is started before we go to sleep waiting for the driver to wake us up when its done.
An earlier I/O error on the journal is the only thing the log force can return as an error there, which isnt useful at that point anyway (we're in a kernel thread trying to free mem).
Quoted 1 line
whatever the hell it's doing (that code gives me a headache), it looks
Heh, likewise. I have voodoo dolls of one or two of the early XFS folks that I like to poke with needles occasionally.. :)
Quoted 2 lines
like this guy might sleep forever holding a lock that will prevent others from freeing kernel memory.
Quoted 6 lines
It will sleep until the previously initiated buffer write is done. AFAICT, we aren't seeing the I/O completion here for some reason... which points more to a possible device driver or h/ware issue (that is the usual root cause of this sort of hang, anyway).
cheers.
Hey, i think i found something.
My quota on my huge device is broken.
(inferno -- 18014398504855404 0 0 18446744073709551519
0 0)
I cant found a way to re-initialize it.
But anyway, at this point i dont need it, trying to disable the quota usage. We will see....
Thanks a lot!
Janos
Quoted 8 lines
Signature · 7 lines
-- Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 3 lines
Hey, i think i found something.
My quota on my huge device is broken.
That should not be a problem. I ran into that "problem" too but had no lockups back then (2.6.16-rc1).
Quoted 3 lines
(inferno -- 18014398504855404 0 0 18446744073709551519
0 0)
I cant found a way to re-initialize it.
Reinit:
quotaoff /mntpt
umount /mntpt
mount /mntpt
Quoted 2 lines
But anyway, at this point i dont need it, trying to disable the quota usage. We will see....
Jan Engelhardt
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
----- Original Message -----
From: "Jan Engelhardt" <[email protected]>
To: "Janos Haar" <[email protected]>
Cc: "Nathan Scott" <[email protected]>; <[email protected]>; <[email protected]>
Sent: Wednesday, May 31, 2006 11:54 PM
Subject: Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 6 lines
Quoted 3 lines
Hey, i think i found something.
My quota on my huge device is broken.That should not be a problem. I ran into that "problem" too but had no lockups back then (2.6.16-rc1).
09:21:36 up 23:05, 1 user, load average: 13.45, 13.14, 13.11
This looks like fixed with disable the quota usage.
The system hangs more often, when i use a script what heavily uses chown and chgrp and chmod.
Thats why i think, to disable the quota.
At this point it looks like fixed.
Quoted 2 lines
Quoted 2 lines
(inferno -- 18014398504855404 0 0
18446744073709551519
Quoted 8 lines
Quoted 2 lines
0 0)
I cant found a way to re-initialize it.Reinit:
quotaoff /mntpt
umount /mntpt
mount /mntpt
Thanks! :-)
Janos
Quoted 2 lines
Quoted 2 lines
But anyway, at this point i dont need it, trying to disable the quota
usage.
Quoted 10 lines
Quoted 1 line
We will see....
Jan Engelhardt
Signature · 5 lines
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 8 lines
Quoted 5 lines
Reinit:
quotaoff /mntpt
umount /mntpt
mount /mntptThanks! :-)
Too bad XFS does not reinit quota on these commands:
qutoaoff /mp
quotaon /mp
Yes, it would lock the filesystem for a moment, but that's better than trying to unmount it under someone having inodes open!
Jan Engelhardt
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 6 lines · Jan Engelhardt
Quoted 6 lines
Quoted 5 lines
Reinit:
quotaoff /mntpt
umount /mntpt
mount /mntpt
Quoted 6 lines
Quoted 2 lines
Thanks! :-)
Too bad XFS does not reinit quota on these commands:
qutoaoff /mp
quotaon /mp
Hmm, remount would be saner if we wanted to take that approach...
Quoted 2 lines
Yes, it would lock the filesystem for a moment, but that's better than trying to unmount it under someone having inodes open!
But its not just a moment, a quotacheck needs to scan every inode in the filesystem (on disk) to correctly account for all space/inode usage. Its not something to be encouraging people to do frequently, and it would also be very difficult to correctly implement (while the filesystem is actively being modified I mean).
cheers.
Signature · 1 line
Nathan
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 7 lines
Quoted 4 lines
Too bad XFS does not reinit quota on these commands:
qutoaoff /mp
quotaon /mpHmm, remount would be saner if we wanted to take that approach...
quotacheck would be sanest :) But the struct super_block->remount is probably the best idea in kernel space.
Quoted 6 lines
Quoted 2 lines
Yes, it would lock the filesystem for a moment, but that's better than trying to unmount it under someone having inodes open!
But its not just a moment, a quotacheck needs to scan every inode in the filesystem (on disk) to correctly account for all space/inode usage.
Yeah right, XFS was designed for large systems rather than for just my 262188 files. (The latter of which completes in an "adequate" time of a few secs.)
Quoted 2 lines
Its not something to be encouraging people to do frequently,
Certainly not, but XFS has the advange of bulkstat for quota scanning. quotacheck on vfsv0 quota databases always takes longer IMO.
Quoted 3 lines
and it would also be very difficult to correctly implement (while the filesystem is actively being modified I mean).
Noted.
Jan Engelhardt
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
In reply to Janos HaarQuoted 5 lines · Janos Haar
Hey, i think i found something.
My quota on my huge device is broken.
(inferno -- 18014398504855404 0 0 18446744073709551519
0 0)
Hmm, that is interesting. I guess you don't know whether this accounting problem happened before you rebooted or whether it only just got this way (after journal recovery)?
Quoted 3 lines
I cant found a way to re-initialize it.
But anyway, at this point i dont need it, trying to disable the quota usage. We will see....
Jan's recipe was spot on, do that.
cheers.
Signature · 1 line
Nathan
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
---- Original Message -----
From: "Nathan Scott" <[email protected]>
To: "Janos Haar" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Thursday, June 01, 2006 11:58 PM
Subject: Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 5 lines · Janos Haar
Quoted 4 lines
Hey, i think i found something.
My quota on my huge device is broken.
(inferno -- 18014398504855404 0 0
18446744073709551519
Quoted 5 lines
Quoted 1 line
0 0)
Hmm, that is interesting. I guess you don't know whether this accounting problem happened before you rebooted or whether it only just got this way (after journal recovery)?
In my system, this huge device is difficult.
I often need to reboot, and run xfs_repair, to make it clean. (nodes hangs, reboots, etc...)
On the beginning, i use the xfs_repair without any options, but it requires to do a mount/umount the mtp before.
The problem is, i often get an error message, (dump) during the journal recovery, and after i cannot run the xfs_repair from script, because it needs the log done by mount.
Now is my default reboot option is xfs_repair -L, so i dont know, this happens before, or after, sorry.
Quoted 3 lines
Quoted 3 lines
I cant found a way to re-initialize it.
But anyway, at this point i dont need it, trying to disable the quota
usage.
Quoted 3 lines
Quoted 1 line
We will see....
Jan's recipe was spot on, do that.
The qouta stop solves the hangs problem.
This is a bug?
Cheers,
Janos
Quoted 10 lines
cheers.
Signature · 6 lines
Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 6 lines · Janos Haar
---- Original Message -----
Quoted 5 lines · Janos Haar
Quoted 4 lines
Hey, i think i found something.
My quota on my huge device is broken.
(inferno -- 18014398504855404 0 0
Quoted 8 lines
18446744073709551519
Quoted 5 lines
Quoted 1 line
0 0)
Hmm, that is interesting. I guess you don't know whether this accounting problem happened before you rebooted or whether it only just got this way (after journal recovery)?
In my system, this huge device is difficult.
Can you describe your hardware a bit more? (and send xfs_info output too please).
Quoted 2 lines
I often need to reboot, and run
xfs_repair, to make it clean. (nodes hangs, reboots, etc...)
Ehrm, hmm, that smells fishy... does this device have a write cache enabled by any chance?
Quoted 2 lines
Now is my default reboot option is
xfs_repair-L, so i dont know, this happens before, or after, sorry.
Oh, thats bad, all bets are off then - you really cant go doing that routinely, thats an "in emergency only" big red button - it throws away the contents of the journal, and will pretty much guarantee filesystem corruption.
But, it sounds alot like you may have a big hardware reliability issue there, which is going to make it difficult to distinguish any software problems. However, if you find a way to reproduce that quota accounting problem (above), I'm all ears.
cheers.
Signature · 1 line
Nathan
Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
----- Original Message -----
From: "Nathan Scott" <[email protected]>
To: "Janos Haar" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Friday, June 02, 2006 1:43 AM
Subject: Re: XFS related hang (was Re: How to send a break? - dump from frozen 64bit linux)
Quoted 10 lines · Janos Haar
Quoted 9 lines
---- Original Message -----
Quoted 5 lines · Janos Haar
Quoted 4 lines
Hey, i think i found something.
My quota on my huge device is broken.
(inferno -- 18014398504855404 0 018446744073709551519
Quoted 2 lines
Quoted 1 line
0 0)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Hmm, that is interesting. I guess you don't know whether this accounting problem happened before you rebooted or whether it only just got this way (after journal recovery)?
In my system, this huge device is difficult.
Can you describe your hardware a bit more? (and send
xfs_infooutput too please).
[root@X64 ~]# xfs_info /mnt/md0
meta-data=/dev/md31 isize=256 agcount=2600, agsize=1240024
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=3223457536, imaxpct=25
= sunit=1 swidth=4 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=16384 blocks=0, rtextents=0(I used the xfs_grow 2 times)
The hw:
I use 4 nodes, each has 3.3TiB (RAID4 array), and serves NBD.
On the concentrator the RAID0 makes one 12.9TiB from 4x3.3TiB nbd device. The strip is 4kb. (tested, and optimal for performance)
Quoted 2 lines
Quoted 2 lines
I often need to reboot, and run
xfs_repair, to make it clean. (nodes
hangs,
Quoted 4 lines
Quoted 1 line
reboots, etc...)
Ehrm, hmm, that smells fishy... does this device have a write cache enabled by any chance?
Yes, you have right!
I know, this is a big chance to corrupt the fs, but i need strongly the write caching!
This quota corruption is from that case too....
Quoted 6 lines
Quoted 3 lines
Now is my default reboot option is
xfs_repair-L, so i dont know, this happens before, or after, sorry.Oh, thats bad, all bets are off then - you really cant go doing that routinely, thats an "in emergency only" big red button -
:-) Yes, i know.
But on my case, the service is much more important than the data inside the fs.
I run a huge "free web storage", and i hate that thing, when i get up, and can see, the system stops on the automatic reboot, and down for few hours...
Quoted 1 line
8-(
If it can reboot normally, and drop some MB or GB, this is a "little lose" for me.
Quoted 2 lines
it throws away the contents of the journal, and will pretty much guarantee filesystem corruption.
Anyway, if i remove the he -L, the boot hangs on mount about 8 from10 times. The ~1GB lose of 4K strip can pretty much damage the journal too..... If it can repair the fs (2 times from 10), it is often uncompleted, and some minutes or hours lated i get the XFS_FORCE_SHUTDOWN message thanks to the corruption....
(I planned to use an external log, but at this time i dont trust too much the journal recovery....
And if the concentrator finish the flush, the journal notes that it is completed, but the node can hang during write, and drop the data anyway.)
Quoted 5 lines
But, it sounds alot like you may have a big hardware reliability issue there, which is going to make it difficult to distinguish any software problems. However, if you find a way to reproduce that quota accounting problem (above), I'm all ears.
Sorry, but i cant.
Additionally, i allready have shut down the quota, and i cannot reproduce the "bad quota related hang" problem.
Thanks a lot!
Janos
Quoted 10 lines
cheers.
Signature · 6 lines
Nathan - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Followup to: <01b701c6818d$4bcd37b0$1800a8c0@dcccs> By author: =?iso-8859-2?Q?Haar_J=E1nos?= <[email protected]> In newsgroup: linux.dev.kernel
Quoted 6 lines
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on the serial line, using echo.
Quoted 7 lines
I mean like this:
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0
You can't use it using echo, however, you can do it using Perl:
perl -e 'use POSIX; tcsendbreak(1,0);' > /dev/ttyS0
-hpa
Quoted 7 lines · On Sat, 27 May 2006, [iso-8859-2] Haar János wrote:
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on the serial line, using echo.
I mean like this:
Quoted 8 lines
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0Thanks,
Janos
Can't you use /proc/sysrq-trigger?
Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.4 on an i686 machine (5592.73 BogoMips). New book: http://www.AbominableFirebug.com/
_
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 15 lines · linux-os (Dick Johnson)
Quoted 12 lines · On Sat, 27 May 2006, [iso-8859-2] Haar János wrote:
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on the serial line, using echo.
I mean like this:
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0
Quoted 5 lines
Quoted 3 lines
Thanks,
JanosCan't you use
/proc/sysrq-trigger?
That can be tricky if the other end of /dev/ttyS0 is plugged into a debugging serial port on an embedded system where you don't have easy access to a shell.
Or for that matter, if you're trying to talk to the serial port on a non-embedded system, which is too far into OOM thrashing for you to be able to get a usable shell prompt.....
----- Original Message -----
From: <[email protected]>
To: "linux-os (Dick Johnson)" <[email protected]>
Cc: "Haar János" <[email protected]>; <[email protected]>
Sent: Monday, May 29, 2006 5:35 PM
Subject: Re: How to send a break?
Quoted 6 lines · linux-os (Dick Johnson)
Quoted 3 lines · On Sat, 27 May 2006, [iso-8859-2] Haar János wrote:
Hello, list,
I wish to know, how to send a "BREAK" to trigger the sysreq functions on
the
Quoted 9 lines
Quoted 9 lines
serial line, using echo.
I mean like this:
#!/bin/bash
echo "?BREAK?" >/dev/ttyS0
sleep 2
echo "m" >/dev/ttyS0
Quoted 7 lines
Quoted 3 lines
Thanks,
JanosCan't you use
/proc/sysrq-trigger?That can be tricky if the other end of
/dev/ttyS0is plugged into a
debugging
Quoted 1 line
serial port on an embedded system where you don't have easy access to a
shell.
Quoted 1 line
Or for that matter, if you're trying to talk to the serial port on a
non-embedded
Quoted 2 lines
system, which is too far into OOM thrashing for you to be able to get a usable shell prompt.....
This is for debugging an frozen X86_64 system! :-)
Thanks,
Janos