Skip to contentSkip to conversation

unkillable processes using samba, xfs and lvm2 snapshots (k 2.6.10)

DISCUSSION Gildas LE NADAN 6 participants 7 messages Dec 28, 2004 – Jan 5, 2005

lore
Share & tools
lore
#0 lore raw

Hi,

I experience hangs on samba processes on a filer using xfs over lvm2 as data partitions, when there is active snapshots of the xfs partitions.

I have a clone of the production server (same software, same hardware) where the situation can be reproduced perfectly.

Testings showed that the result was the same, whether the snapshots were mounted or not : smbd processes are locked and unkillable while the machine is normaly working otherwise, except software reboot is impossible and hardware reset is needed.

I noticed Brad Fitzpatrick's case in kernel 2.6.10 changelog (http://lkml.org/lkml/2004/11/14/98) and tested kernel 2.6.10 today without success.

Configuration is the following :
- supermicro m/b with dual Xeon 2,8Ghz (SMT is active)
- 1 GB ram,
- adaptec u320 raid controler
- kernel 2.6.10
- debian sarge
- samba 3
- LVM2
- XFS with quota turned on

All software are from debian sarge packages, except the kernel.

I'm not able to determine if the problem is more xfs, device mapper or samba related, and was not able to do extensive testings (using a different filesystem, testing with a different daemon, etc...), but SMT/SMP testings showed that this is not a SMP/SMT related problem.

I've compiled the kernel with the debugging options, so I might provide additional informations if needed as in Brad's case.

Sincerely,
Gildas LE NADAN
(Please CC me as I didn't suscribe to LKML)

#1 lore raw
Quoted 2 lines · Gildas LE NADAN

I experience hangs on samba processes on a filer using xfs over lvm2 as data partitions, when there is active snapshots of the xfs partitions.

A trick is to enable alt-sysrq and press alt-sysrq-t (I think) which spams your syslog with backtraces of all processes currently running, including the ones stuck in 'D' state (ps aux | grep " D ").

If you isolate these backtraces and send them to this list, they will enable developers to help you. Make sure you add 'includes backtrace' in your Subject.

Quoted 4 lines

Testings showed that the result was the same, whether the snapshots were mounted or not : smbd processes are locked and unkillable while the machine is normaly working otherwise, except software reboot is impossible and hardware reset is needed.

For maximum usefulness, make your setup as simple as possible and reproduce.

Good luck - I personally can't help you in any real way, except to help you get the debugging information that is needed.

Signature · 2 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
#2 lore raw

Re: unkillable processes using samba, xfs and lvm2 snapshots (k 2.6.10) [includes backtrace]

Quoted 10 lines
Quoted 2 lines

I experience hangs on samba processes on a filer using xfs over lvm2 as data partitions, when there is active snapshots of the xfs partitions.

A trick is to enable alt-sysrq and press alt-sysrq-t (I think) which spams your syslog with backtraces of all processes currently running, including the ones stuck in 'D' state (ps aux | grep " D ").

If you isolate these backtraces and send them to this list, they will enable developers to help you. Make sure you add 'includes backtrace' in your Subject.

OK, this is what I get after provoking the problem on the test server (copying 1Go of data is enough to trigger the problem) :

# ps afx | grep smbd
 2279 ?        Ss     0:00 /usr/sbin/smbd -D
 2288 ?        S      0:00  \_ /usr/sbin/smbd -D
 2447 ?        D      0:01  \_ /usr/sbin/smbd -D
 2487 pts/0    S+     0:00  |               \_ grep smbd
#  killall -9 smbd
# ps afx | grep smbd
 2554 pts/0    S+     0:00  |               \_ grep smbd
 2447 ?        D      0:01 /usr/sbin/smbd -D

I did a "echo t > /proc/sysrq-trigger" and tried to clean the resulting logs a bit before sending. Hope this gives enough info, otherwise I kept the whole log so I can send whatever part is needed

SysRq : Show State
sibling
task PC pid father child younger older
...
xfslogd/0 S 00000004 0 218 11 220 216 (L-TLB)
f7eecf44 00000046 f7eecf34 00000004 00000002 f60ef53c c0427ba0 f60ef5a8

        00000282 c01017cc 00000000 f7f28974 f7f2896c 00000000 c170f020 
00000000
        00000c41 ff6027e0 00000005 00000286 f7eb9530 f7eb96b0 f7eecf94 
00000002
 Call Trace:
  [__up+28/32] __up+0x1c/0x20
  [worker_thread+565/608] worker_thread+0x235/0x260
  [pagebuf_iodone_work+0/80] pagebuf_iodone_work+0x0/0x50
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfslogd/1     S 00000004     0   219     10           221   217 (L-TLB)
 f7c82f44 00000046 f7c82f30 00000004 00000001 ffffffff f7eb9020 35a49146
        00000000 f7eb9020 c170f020 f7eb9020 00000000 c1717a00 c1717020 
00000001
        000008ae 0395f3e5 00000000 c171705c f7c7e020 f7c7e1a0 00000001 
f7f289dc
 Call Trace:
  [worker_thread+565/608] worker_thread+0x235/0x260
  [schedule+1132/3360] schedule+0x46c/0xd20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfslogd/2     S 00000004     0   220     11           222   218 (L-TLB)
 f7eedf44 00000046 f7eedf34 00000004 00000004 00000000 f7c1f020 c01f4a99
        f714b13c 00000000 00000000 f7f28a74 f7f28a6c 00000000 c171f020 
00000002
        00000d47 6261fbfd 00000074 00000286 f7eb9020 f7eb91a0 f7eedf94 
00000008
 Call Trace:
  [xfs_buf_iodone_callbacks+361/368] xfs_buf_iodone_callbacks+0x169/0x170
  [worker_thread+565/608] worker_thread+0x235/0x260
  [pagebuf_iodone_work+0/80] pagebuf_iodone_work+0x0/0x50
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfslogd/3     S 00000004     0   221     10           223   219 (L-TLB)
 f7c84f44 00000046 f7c84f34 00000004 00000003 ffffffff f7c27a40 35a47b19
        00000000 03969a3b 03969a3b 00000000 f7c84f28 c0116200 c1727020 
00000003
        00000ef7 0396cbec 00000000 00000286 f7c83a40 f7c83bc0 f7c84f94 
00000004
 Call Trace:
  [activate_task+144/176] activate_task+0x90/0xb0
  [worker_thread+565/608] worker_thread+0x235/0x260
  [schedule+1132/3360] schedule+0x46c/0xd20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfsdatad/0    S 00000004     0   222     11           224   220 (L-TLB)
 f7f06f44 00000046 f7f06f30 00000004 00000002 ffffffff f7c83530 35a48050
        00000000 f7c83530 c1717020 f7c83530 00000000 c170fa00 c170f020 
00000000
        00000897 0397ce5a 00000000 c170f05c f7f05a40 f7f05bc0 00000002 
f7f28550
 Call Trace:
  [worker_thread+565/608] worker_thread+0x235/0x260
  [schedule+1132/3360] schedule+0x46c/0xd20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfsdatad/1    S 00000004     0   223     10           225   221 (L-TLB)
 f7c85f44 00000046 f7c85f30 00000004 00000001 ffffffff f7f05530 35a47efe
        00000000 f7f05530 c170f020 f7f05530 00000000 c1717a00 c1717020 
00000001
        000008f9 0398532d 00000000 c171705c f7c83530 f7c836b0 00000001 
f7f285d0
 Call Trace:
  [worker_thread+565/608] worker_thread+0x235/0x260
  [schedule+1132/3360] schedule+0x46c/0xd20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfsdatad/2    S 00000004     0   224     11           903   222 (L-TLB)
 f7f07f44 00000046 f7f07f34 00000004 00000004 ffffffff f7c1f020 35a493ed
        00000000 03985b99 03985b99 00000000 f7f07f28 c0116200 c171f020 
00000002
        00000d7c 03988e25 00000000 00000286 f7f05530 f7f056b0 f7f07f94 
00000008
 Call Trace:
  [activate_task+144/176] activate_task+0x90/0xb0
  [worker_thread+565/608] worker_thread+0x235/0x260
  [schedule+1132/3360] schedule+0x46c/0xd20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfsdatad/3    S 00000004     0   225     10           902   223 (L-TLB)
 f7c87f44 00000046 f7c87f34 00000004 00000003 ffffffff f7c27a40 35a49032
        00000000 0398e44a 0398e44a 00000000 f7c87f28 c0116200 c1727020 
00000003
        000010b0 0399175f 00000000 00000286 f7c83020 f7c831a0 f7c87f94 
00000004
 Call Trace:
  [activate_task+144/176] activate_task+0x90/0xb0
  [worker_thread+565/608] worker_thread+0x235/0x260
  [schedule+1132/3360] schedule+0x46c/0xd20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [worker_thread+0/608] worker_thread+0x0/0x260
  [kthread+186/192] kthread+0xba/0xc0
  [kthread+0/192] kthread+0x0/0xc0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfsbufd       S 00000004     0   226      1           815   213 (L-TLB)
 f7f08f78 00000046 f7f08f68 00000004 00000001 00000000 f7c1f530 c02c1edb
        f7f70e64 00000000 f7eaa944 c0264f5f 00000004 c04f99e8 c1717020 
00000001
        00000134 6d117e89 0000009e c0125879 f7f05020 f7f051a0 00000000 
00000001
 Call Trace:
  [elv_next_request+27/256] elv_next_request+0x1b/0x100
  [kobject_put+31/48] kobject_put+0x1f/0x30
  [__mod_timer+249/320] __mod_timer+0xf9/0x140
  [schedule_timeout+117/208] schedule_timeout+0x75/0xd0
  [process_timeout+0/16] process_timeout+0x0/0x10
  [dm_unplug_all+39/64] dm_unplug_all+0x27/0x40
  [blk_backing_dev_unplug+0/32] blk_backing_dev_unplug+0x0/0x20
  [pagebuf_daemon+118/512] pagebuf_daemon+0x76/0x200
  [pagebuf_daemon+0/512] pagebuf_daemon+0x0/0x200
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
...
 xfssyncd      S 00000004     0  1361      1          1362  1360 (L-TLB)
 f756ef74 00000046 f756ef64 00000004 00000002 f5735568 c0427ba0 f5735568
        f756ef2c 0000022e 00000031 f714be3c f5735568 00000000 c170f020 
00000000
        000034a1 58c63e72 00000098 c0125879 f6fb6a40 f6fb6bc0 00000000 
00000002
 Call Trace:
  [__mod_timer+249/320] __mod_timer+0xf9/0x140
  [schedule_timeout+117/208] schedule_timeout+0x75/0xd0
  [process_timeout+0/16] process_timeout+0x0/0x10
  [xfssyncd+134/480] xfssyncd+0x86/0x1e0
  [xfssyncd+0/480] xfssyncd+0x0/0x1e0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
 xfssyncd      S 00000004     0  1362      1          2233  1361 (L-TLB)
 f6accf74 00000046 f6accf64 00000004 00000002 f60f4360 c0427ba0 f60efc3c
        c050ad58 c023a3ee 00000031 f60efc3c f6d6ccd0 00000000 c170f020 
00000000
        00001568 6080d951 00000098 c0125879 f6a95530 f6a956b0 00000000 
00000002
 Call Trace:
  [pagebuf_rele+46/240] pagebuf_rele+0x2e/0xf0
  [__mod_timer+249/320] __mod_timer+0xf9/0x140
  [schedule_timeout+117/208] schedule_timeout+0x75/0xd0
  [process_timeout+0/16] process_timeout+0x0/0x10
  [xfssyncd+134/480] xfssyncd+0x86/0x1e0
  [xfssyncd+0/480] xfssyncd+0x0/0x1e0
  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
...
 smbd          S 00000004     0  2279      1  2288    2285  2277 (NOTLB)
 f5110ea4 00000082 f5110e90 00000004 00000002 c013ed74 f6770020 c042cd80
        000000d0 f6770020 c1717020 f6770020 00000000 c170fa00 c170f020 
00000000
        0000b4c6 78f5a598 0000006d c170f05c f779a530 f779a6b0 00000002 
f5d37028
 Call Trace:
  [__alloc_pages+484/928] __alloc_pages+0x1e4/0x3a0
  [schedule_timeout+199/208] schedule_timeout+0xc7/0xd0
  [tcp_poll+52/400] tcp_poll+0x34/0x190
  [handle_mm_fault+344/384] handle_mm_fault+0x158/0x180
  [add_wait_queue+29/80] add_wait_queue+0x1d/0x50
  [pipe_poll+52/128] pipe_poll+0x34/0x80
  [do_select+401/736] do_select+0x191/0x2e0
  [__pollwait+0/208] __pollwait+0x0/0xd0
  [sys_select+731/1456] sys_select+0x2db/0x5b0
  [syscall_call+7/11] syscall_call+0x7/0xb
...
 smbd          D 00000004     0  2447   2279                2288 (NOTLB)
 f6736bbc 00000082 f6736bac 00000004 00000002 00000000 c0427ba0 00000000
        f6770020 c0118350 00000000 00000000 c17ff080 00000007 c170f020 
00000000
        00008e96 5602776d 00000071 00000000 f6770020 f67701a0 c023a197 
00000002
 Call Trace:
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [pagebuf_associate_memory+103/400] pagebuf_associate_memory+0x67/0x190
  [schedule_timeout+199/208] schedule_timeout+0xc7/0xd0
  [xlog_sync+630/1216] xlog_sync+0x276/0x4c0
  [xlog_state_release_iclog+91/272] xlog_state_release_iclog+0x5b/0x110
  [add_wait_queue_exclusive+26/80] add_wait_queue_exclusive+0x1a/0x50
  [xlog_state_sync+602/656] xlog_state_sync+0x25a/0x290
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [xlog_assign_tail_lsn+73/128] xlog_assign_tail_lsn+0x49/0x80
  [default_wake_function+0/32] default_wake_function+0x0/0x20
  [xfs_log_force+132/144] xfs_log_force+0x84/0x90
  [xfs_trans_commit+631/1008] xfs_trans_commit+0x277/0x3f0
  [xfs_trans_dup+191/208] xfs_trans_dup+0xbf/0xd0
  [xfs_itruncate_finish+593/1072] xfs_itruncate_finish+0x251/0x430
  [xfs_setattr+3578/4128] xfs_setattr+0xdfa/0x1020
  [linvfs_setattr+258/384] linvfs_setattr+0x102/0x180
  [kmem_cache_alloc+114/192] kmem_cache_alloc+0x72/0xc0
  [linvfs_setattr+0/384] linvfs_setattr+0x0/0x180
  [notify_change+334/400] notify_change+0x14e/0x190
  [do_truncate+147/208] do_truncate+0x93/0xd0
  [fget+73/96] fget+0x49/0x60
  [sys_ftruncate64+204/304] sys_ftruncate64+0xcc/0x130
  [sys_open+108/144] sys_open+0x6c/0x90
  [syscall_call+7/11] syscall_call+0x7/0xb
#3 lore raw
In reply to Gildas LE NADAN
Quoted 36 lines · Gildas LE NADAN

Hi,

I experience hangs on samba processes on a filer using xfs over lvm2 as data partitions, when there is active snapshots of the xfs partitions.

I have a clone of the production server (same software, same hardware) where the situation can be reproduced perfectly.

Testings showed that the result was the same, whether the snapshots were mounted or not : smbd processes are locked and unkillable while the machine is normaly working otherwise, except software reboot is impossible and hardware reset is needed.

I noticed Brad Fitzpatrick's case in kernel 2.6.10 changelog
(http://lkml.org/lkml/2004/11/14/98) and tested kernel 2.6.10 today without success.

Configuration is the following :
- supermicro m/b with dual Xeon 2,8Ghz (SMT is active)
- 1 GB ram,
- adaptec u320 raid controler
- kernel 2.6.10
- debian sarge
- samba 3
- LVM2
- XFS with quota turned on

All software are from debian sarge packages, except the kernel.

I'm not able to determine if the problem is more xfs, device mapper or samba related, and was not able to do extensive testings (using a different filesystem, testing with a different daemon, etc...), but SMT/SMP testings showed that this is not a SMP/SMT related problem.

Quoted 6 lines

I've compiled the kernel with the debugging options, so I might provide additional informations if needed as in Brad's case.

Sincerely,
Gildas LE NADAN
(Please CC me as I didn't suscribe to LKML)

I have a somewhat similar case here, samba processses are unkillable, but I can do a software reboot. Something is also killing amandad, and I lost the backup of this machine last night. The amanda logs are bereft of any info and I've no clue that its happened except a message from amanda that the client access timed out on this machine. This was while running 2.6.10-rc3-mm1-V0.33-04 which ran stably for 8 days previously.

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.
#4 lore raw
Quoted 7 lines · Gene Heskett

I have a somewhat similar case here, samba processses are unkillable, but I can do a software reboot. Something is also killing amandad, and I lost the backup of this machine last night. The amanda logs are bereft of any info and I've no clue that its happened except a message from amanda that the client access timed out on this machine. This was while running 2.6.10-rc3-mm1-V0.33-04 which ran stably for 8 days previously.

I have the same problem (2.6.10-rc3 running 7 days without problems) and i had D state mc, smbd and lsof:
(there is something about nfs in the call tree, it _might_ be that i halted a nfs server the day before without unmounting it on the system with the problem)

Dec 31 18:55:20 core kernel: nfs warning: mount version older than kernel
Jan  1 06:25:38 core kernel: nfs: server igor3 not responding, still trying
Jan  1 17:29:15 core kernel: nfs: server igor3 not responding, still trying
Jan  1 18:05:12 core kernel:       (NOTLB)
Jan  1 18:05:12 core kernel: c843deb4 00200086 f68a6580 c04c0150 000274ab c18e57e0 587f3ab2 000274ab 
Jan  1 18:05:12 core kernel:        00003875 5880d98d 000274ab d696d560 d696d6bc 00000014 d696d560 c843c000 
Jan  1 18:05:12 core kernel:        ffffe000 c011cca2 d696d560 ecc030e0 00040005 c011d00f ffffffff d696d9d4 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c011cca2>] finish_stop+0x42/0x90
Jan  1 18:05:12 core kernel:  [<c011d00f>] get_signal_to_deliver+0x19f/0x2b0
Jan  1 18:05:12 core kernel:  [<c0102388>] do_signal+0x98/0x130
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c01059b2>] sys_ptrace+0xb2/0x610
Jan  1 18:05:12 core kernel:  [<c0102455>] do_notify_resume+0x35/0x38
Jan  1 18:05:12 core kernel:  [<c0102596>] work_notifysig+0x13/0x15
Jan  1 18:05:12 core kernel: mc            D C04C0120     0 21514  30032 21516               (NOTLB)
Jan  1 18:05:12 core kernel: f0269da8 00000082 d1edba20 c04c0120 00000000 00000292 f7de72a4 f0269da8 
Jan  1 18:05:12 core kernel:        000f8c0f 36b16015 00026d31 d1edba20 d1edbb7c df1cbe94 df1cbda0 f0269dcc 
Jan  1 18:05:12 core kernel:        df1cbeb8 c01d4a65 c1b25560 f7de72a4 c01af638 00000000 d6aba200 00000000 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c01d4a65>] nfs_wait_on_inode+0xd5/0x1e0
Jan  1 18:05:12 core kernel:  [<c01af638>] ext3_mark_iloc_dirty+0x28/0x40
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c01545a7>] follow_mount+0x57/0xa0
Jan  1 18:05:12 core kernel:  [<c01d4f64>] __nfs_revalidate_inode+0x74/0x360
Jan  1 18:05:12 core kernel:  [<c0160120>] update_atime+0xd0/0xe0
Jan  1 18:05:12 core kernel:  [<c0154e6d>] link_path_walk+0x73d/0xb60
Jan  1 18:05:12 core kernel:  [<c01bd3c9>] journal_stop+0x149/0x200
Jan  1 18:05:12 core kernel:  [<c01d52ea>] nfs_revalidate_inode+0x4a/0x70
Jan  1 18:05:12 core kernel:  [<c01d4bd0>] nfs_getattr+0x60/0xa0
Jan  1 18:05:12 core kernel:  [<c01509f9>] vfs_getattr+0x39/0xa0
Jan  1 18:05:12 core kernel:  [<c0150aaf>] vfs_stat+0x4f/0x60
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c015a1ff>] fifo_open+0x13f/0x26f
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: mozilla-bin   S C04C0120     0  2810   2994          2859       (NOTLB)
Jan  1 18:05:12 core kernel: e6135f10 00200086 dde78a00 c04c0120 e6135fa0 cb94fa98 c012ff73 c1125b60 
Jan  1 18:05:12 core kernel:        00002623 41070976 00028179 dde78a00 dde78b5c 00000000 7fffffff e6135f68 
Jan  1 18:05:12 core kernel:        7fffffff c0389a35 c01536c4 e4cb7d80 cb1db8e0 e6135fa0 00000145 e491a420 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c012ff73>] __get_free_pages+0x33/0x40
Jan  1 18:05:12 core kernel:  [<c0389a35>] schedule_timeout+0xb5/0xc0
Jan  1 18:05:12 core kernel:  [<c01536c4>] pipe_poll+0x34/0x80
Jan  1 18:05:12 core kernel:  [<c0159d1f>] do_pollfd+0x4f/0x90
Jan  1 18:05:12 core kernel:  [<c0159e0a>] do_poll+0xaa/0xd0
Jan  1 18:05:12 core kernel:  [<c0159f82>] sys_poll+0x152/0x210
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: mozilla-bin   S C04C0120     0  2859   2994          2860  2810 (NOTLB)
Jan  1 18:05:12 core kernel: db23bf10 00200086 c46cda20 c04c0120 000000d0 2d9fbab8 000000d0 f3ca81a0 
Jan  1 18:05:12 core kernel:        00000799 d2284a24 00028178 c46cda20 c46cdb7c 00000000 7fffffff db23bf68 
Jan  1 18:05:12 core kernel:        7fffffff c0389a35 c01536c4 f3ca81a0 cb1dbf20 db23bfa0 00000145 da0da768 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0389a35>] schedule_timeout+0xb5/0xc0
Jan  1 18:05:12 core kernel:  [<c01536c4>] pipe_poll+0x34/0x80
Jan  1 18:05:12 core kernel:  [<c0159d1f>] do_pollfd+0x4f/0x90
Jan  1 18:05:12 core kernel:  [<c0159e0a>] do_poll+0xaa/0xd0
Jan  1 18:05:12 core kernel:  [<c0159f82>] sys_poll+0x152/0x210
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: mozilla-bin   S C04C0150     0  2860   2994                2859 (NOTLB)
Jan  1 18:05:12 core kernel: c79ffe90 00200086 dde78a00 c04c0150 00028179 00000000 4105997b 00028179 
Jan  1 18:05:12 core kernel:        0000159c 4105b238 00028179 c46cd540 c46cd69c 2a0aec95 c79ffea4 fffffff5 
Jan  1 18:05:12 core kernel:        c79ffedc c03899e3 c79ffea4 2a0aec95 c79ffec8 c04c60c8 c04c60c8 2a0aec95 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c03899e3>] schedule_timeout+0x63/0xc0
Jan  1 18:05:12 core kernel:  [<c011aa90>] process_timeout+0x0/0x10
Jan  1 18:05:12 core kernel:  [<c012585f>] futex_wait+0x12f/0x170
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c01535d0>] pipe_write+0x0/0x40
Jan  1 18:05:12 core kernel:  [<c0125b18>] do_futex+0x48/0xa0
Jan  1 18:05:12 core kernel:  [<c0125c5e>] sys_futex+0xee/0x100
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: xchat         S C04C0120     0  7965      1          7966 26544 (NOTLB)
Jan  1 18:05:12 core kernel: e4e43f10 00200082 ce06d5a0 c04c0120 e4e43fa0 e93f8af8 c012ff73 d5703400 
Jan  1 18:05:12 core kernel:        00000852 5500c143 0002817b ce06d5a0 ce06d6fc 2a0adc55 e4e43f24 e4e43f68 
Jan  1 18:05:12 core kernel:        00000034 c03899e3 e4e43f24 2a0adc55 c86fd740 c04c5a10 c04c5a10 2a0adc55 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c012ff73>] __get_free_pages+0x33/0x40
Jan  1 18:05:12 core kernel:  [<c03899e3>] schedule_timeout+0x63/0xc0
Jan  1 18:05:12 core kernel:  [<c011aa90>] process_timeout+0x0/0x10
Jan  1 18:05:12 core kernel:  [<c0159e0a>] do_poll+0xaa/0xd0
Jan  1 18:05:12 core kernel:  [<c0159f82>] sys_poll+0x152/0x210
Jan  1 18:05:12 core kernel:  [<c01163fb>] sys_gettimeofday+0x3b/0x80
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: xchat         S C04C05C8     0  7966      1         12134  7965 (NOTLB)
Jan  1 18:05:12 core kernel: c30bfeb4 00200082 ce06d5a0 c04c05c8 0002817b 00000001 5500689d 0002817b 
Jan  1 18:05:12 core kernel:        000006de 55006e0a 0002817b dde78520 dde7867c 00000000 7fffffff 00000006 
Jan  1 18:05:12 core kernel:        00000006 c0389a35 c1346120 00000000 c01593f5 f64b54a4 00200246 f64b54a4 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0389a35>] schedule_timeout+0xb5/0xc0
Jan  1 18:05:12 core kernel:  [<c01593f5>] __pollwait+0x85/0xd0
Jan  1 18:05:12 core kernel:  [<c01536c4>] pipe_poll+0x34/0x80
Jan  1 18:05:12 core kernel:  [<c0159693>] do_select+0x173/0x2b0
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c0159abf>] sys_select+0x2bf/0x4d0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: smbd          D C04C0120     0  9058      1         12172 26379 (NOTLB)
Jan  1 18:05:12 core kernel: e1d95c24 00000082 ed6365a0 c04c0120 f7cbdce0 c03762de d0130400 f7cbdce0 
Jan  1 18:05:12 core kernel:        00011688 13c6a05f 000280e1 ed6365a0 ed6366fc f7cbdce0 e1d95c60 f7cbdd58 
Jan  1 18:05:12 core kernel:        e1d95c40 c037800d f7cbdce0 00000000 da6b95f8 e1d94000 00000000 00000000 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c03762de>] xprt_prepare_transmit+0x7e/0xc0
Jan  1 18:05:12 core kernel:  [<c037800d>] __rpc_execute+0x13d/0x3c0
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c03786b6>] rpc_new_task+0x36/0xb0
Jan  1 18:05:12 core kernel:  [<c0373a24>] rpc_call_sync+0x74/0xb0
Jan  1 18:05:12 core kernel:  [<c01dd6fa>] nfs3_rpc_wrapper+0x3a/0x80
Jan  1 18:05:12 core kernel:  [<c01ddc9a>] nfs3_proc_access+0xda/0x170
Jan  1 18:05:12 core kernel:  [<c01bdc75>] __journal_file_buffer+0x175/0x230
Jan  1 18:05:12 core kernel:  [<c01bd02f>] journal_dirty_metadata+0xef/0x170
Jan  1 18:05:12 core kernel:  [<c037973c>] rpcauth_lookup_credcache+0x1cc/0x210
Jan  1 18:05:12 core kernel:  [<c01d29c5>] nfs_do_access+0x65/0xb0
Jan  1 18:05:12 core kernel:  [<c01d2b00>] nfs_permission+0xf0/0x170
Jan  1 18:05:12 core kernel:  [<c0154241>] permission+0x51/0x60
Jan  1 18:05:12 core kernel:  [<c01551c2>] link_path_walk+0xa92/0xb60
Jan  1 18:05:12 core kernel:  [<c0160120>] update_atime+0xd0/0xe0
Jan  1 18:05:12 core kernel:  [<c0154e0d>] link_path_walk+0x6dd/0xb60
Jan  1 18:05:12 core kernel:  [<c01554e0>] path_lookup+0x70/0x110
Jan  1 18:05:12 core kernel:  [<c0155733>] __user_walk+0x33/0x60
Jan  1 18:05:12 core kernel:  [<c0150a7f>] vfs_stat+0x1f/0x60
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0103ee8>] math_state_restore+0x28/0x50
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: lsof          D C04C0120     0 12134      1         12137  7966 (NOTLB)
Jan  1 18:05:12 core kernel: c2341da8 00000086 c46cd060 c04c0120 c0389b45 d900a5f8 c0149040 c2341dd4 
Jan  1 18:05:12 core kernel:        0000196b 064b7168 000280e4 c46cd060 c46cd1bc df1cbe94 df1cbda0 c2341dcc 
Jan  1 18:05:12 core kernel:        df1cbeb8 c01d4a65 c0124cc0 c2341dd4 c2341dd4 00000000 d6aba200 00000002 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0389b45>] __wait_on_bit+0x45/0x60
Jan  1 18:05:12 core kernel:  [<c0149040>] sync_buffer+0x0/0x50
Jan  1 18:05:12 core kernel:  [<c01d4a65>] nfs_wait_on_inode+0xd5/0x1e0
Jan  1 18:05:12 core kernel:  [<c0124cc0>] wake_bit_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c01d4f64>] __nfs_revalidate_inode+0x74/0x360
Jan  1 18:05:12 core kernel:  [<c01545a7>] follow_mount+0x57/0xa0
Jan  1 18:05:12 core kernel:  [<c0155011>] link_path_walk+0x8e1/0xb60
Jan  1 18:05:12 core kernel:  [<c01d52ea>] nfs_revalidate_inode+0x4a/0x70
Jan  1 18:05:12 core kernel:  [<c01d4bd0>] nfs_getattr+0x60/0xa0
Jan  1 18:05:12 core kernel:  [<c01509f9>] vfs_getattr+0x39/0xa0
Jan  1 18:05:12 core kernel:  [<c0150aaf>] vfs_stat+0x4f/0x60
Jan  1 18:05:12 core kernel:  [<c01532d7>] pipe_read+0x37/0x40
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0147ea1>] vfs_read+0xd1/0x130
Jan  1 18:05:12 core kernel:  [<c038944e>] schedule+0x2ce/0x4d0
Jan  1 18:05:12 core kernel:  [<c0148171>] sys_read+0x51/0x80
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: lsof          D C04C0120     0 12137      1         12141 12134 (NOTLB)
Jan  1 18:05:12 core kernel: cbed5da8 00200086 f2996a40 c04c0120 c0139fa3 dbd4d900 d9115078 d221efe4 
Jan  1 18:05:12 core kernel:        00001704 00436954 000280e6 f2996a40 f2996b9c df1cbe94 df1cbda0 cbed5dcc 
Jan  1 18:05:12 core kernel:        df1cbeb8 c01d4a65 d9115078 c013a386 dbd4d900 00000000 d6aba200 00000001 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0139fa3>] do_no_page+0x63/0x250
Jan  1 18:05:12 core kernel:  [<c01d4a65>] nfs_wait_on_inode+0xd5/0x1e0
Jan  1 18:05:12 core kernel:  [<c013a386>] handle_mm_fault+0xf6/0x170
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c010d34c>] do_page_fault+0x18c/0x599
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c01d4f64>] __nfs_revalidate_inode+0x74/0x360
Jan  1 18:05:12 core kernel:  [<c01545a7>] follow_mount+0x57/0xa0
Jan  1 18:05:12 core kernel:  [<c0155011>] link_path_walk+0x8e1/0xb60
Jan  1 18:05:12 core kernel:  [<c01d52ea>] nfs_revalidate_inode+0x4a/0x70
Jan  1 18:05:12 core kernel:  [<c01d4bd0>] nfs_getattr+0x60/0xa0
Jan  1 18:05:12 core kernel:  [<c01509f9>] vfs_getattr+0x39/0xa0
Jan  1 18:05:12 core kernel:  [<c0150aaf>] vfs_stat+0x4f/0x60
Jan  1 18:05:12 core kernel:  [<c01532d7>] pipe_read+0x37/0x40
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0147ea1>] vfs_read+0xd1/0x130
Jan  1 18:05:12 core kernel:  [<c038944e>] schedule+0x2ce/0x4d0
Jan  1 18:05:12 core kernel:  [<c0148171>] sys_read+0x51/0x80
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: lsof          D C04C0120     0 12141      1         12169 12137 (NOTLB)
Jan  1 18:05:12 core kernel: d6507da8 00200082 f2996560 c04c0120 c0139fa3 dbd4db20 e57184bc d6224fe4 
Jan  1 18:05:12 core kernel:        00001ad7 d3d93fee 000280e9 f2996560 f29966bc df1cbe94 df1cbda0 d6507dcc 
Jan  1 18:05:12 core kernel:        df1cbeb8 c01d4a65 e57184bc c013a386 dbd4db20 00000000 d6aba200 00000001 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0139fa3>] do_no_page+0x63/0x250
Jan  1 18:05:12 core kernel:  [<c01d4a65>] nfs_wait_on_inode+0xd5/0x1e0
Jan  1 18:05:12 core kernel:  [<c013a386>] handle_mm_fault+0xf6/0x170
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c010d34c>] do_page_fault+0x18c/0x599
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c01d4f64>] __nfs_revalidate_inode+0x74/0x360
Jan  1 18:05:12 core kernel:  [<c01545a7>] follow_mount+0x57/0xa0
Jan  1 18:05:12 core kernel:  [<c0155011>] link_path_walk+0x8e1/0xb60
Jan  1 18:05:12 core kernel:  [<c01d52ea>] nfs_revalidate_inode+0x4a/0x70
Jan  1 18:05:12 core kernel:  [<c01d4bd0>] nfs_getattr+0x60/0xa0
Jan  1 18:05:12 core kernel:  [<c01509f9>] vfs_getattr+0x39/0xa0
Jan  1 18:05:12 core kernel:  [<c0150aaf>] vfs_stat+0x4f/0x60
Jan  1 18:05:12 core kernel:  [<c01532d7>] pipe_read+0x37/0x40
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0147ea1>] vfs_read+0xd1/0x130
Jan  1 18:05:12 core kernel:  [<c038944e>] schedule+0x2ce/0x4d0
Jan  1 18:05:12 core kernel:  [<c0148171>] sys_read+0x51/0x80
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: lsof          D C04C0120     0 12169      1         12171 12141 (NOTLB)
Jan  1 18:05:12 core kernel: de709da8 00200082 dfe36a80 c04c0120 c0139fa3 dbd4d6e0 d9115ee8 ef49dfe4 
Jan  1 18:05:12 core kernel:        000019ce 39260da8 000280f0 dfe36a80 dfe36bdc df1cbe94 df1cbda0 de709dcc 
Jan  1 18:05:12 core kernel:        df1cbeb8 c01d4a65 d9115ee8 c013a386 dbd4d6e0 00000000 d6aba200 00000001 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0139fa3>] do_no_page+0x63/0x250
Jan  1 18:05:12 core kernel:  [<c01d4a65>] nfs_wait_on_inode+0xd5/0x1e0
Jan  1 18:05:12 core kernel:  [<c013a386>] handle_mm_fault+0xf6/0x170
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c010d34c>] do_page_fault+0x18c/0x599
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c01d4f64>] __nfs_revalidate_inode+0x74/0x360
Jan  1 18:05:12 core kernel:  [<c01545a7>] follow_mount+0x57/0xa0
Jan  1 18:05:12 core kernel:  [<c0155011>] link_path_walk+0x8e1/0xb60
Jan  1 18:05:12 core kernel:  [<c01d52ea>] nfs_revalidate_inode+0x4a/0x70
Jan  1 18:05:12 core kernel:  [<c01d4bd0>] nfs_getattr+0x60/0xa0
Jan  1 18:05:12 core kernel:  [<c01509f9>] vfs_getattr+0x39/0xa0
Jan  1 18:05:12 core kernel:  [<c0150aaf>] vfs_stat+0x4f/0x60
Jan  1 18:05:12 core kernel:  [<c01532d7>] pipe_read+0x37/0x40
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0147ea1>] vfs_read+0xd1/0x130
Jan  1 18:05:12 core kernel:  [<c038944e>] schedule+0x2ce/0x4d0
Jan  1 18:05:12 core kernel:  [<c0148171>] sys_read+0x51/0x80
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: lsof          D C04C0120     0 12171      1         26379 12169 (NOTLB)
Jan  1 18:05:12 core kernel: ed273da8 00200082 dde78040 c04c0120 c0139fa3 eb16f740 d9115e40 de75efe4 
Jan  1 18:05:12 core kernel:        000019a1 bccd2773 000280f0 dde78040 dde7819c df1cbe94 df1cbda0 ed273dcc 
Jan  1 18:05:12 core kernel:        df1cbeb8 c01d4a65 d9115e40 c013a386 eb16f740 00000000 d6aba200 00000001 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c0139fa3>] do_no_page+0x63/0x250
Jan  1 18:05:12 core kernel:  [<c01d4a65>] nfs_wait_on_inode+0xd5/0x1e0
Jan  1 18:05:12 core kernel:  [<c013a386>] handle_mm_fault+0xf6/0x170
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c010d34c>] do_page_fault+0x18c/0x599
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c01d4f64>] __nfs_revalidate_inode+0x74/0x360
Jan  1 18:05:12 core kernel:  [<c01545a7>] follow_mount+0x57/0xa0
Jan  1 18:05:12 core kernel:  [<c0155011>] link_path_walk+0x8e1/0xb60
Jan  1 18:05:12 core kernel:  [<c01d52ea>] nfs_revalidate_inode+0x4a/0x70
Jan  1 18:05:12 core kernel:  [<c01d4bd0>] nfs_getattr+0x60/0xa0
Jan  1 18:05:12 core kernel:  [<c01509f9>] vfs_getattr+0x39/0xa0
Jan  1 18:05:12 core kernel:  [<c0150aaf>] vfs_stat+0x4f/0x60
Jan  1 18:05:12 core kernel:  [<c01532d7>] pipe_read+0x37/0x40
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0147ea1>] vfs_read+0xd1/0x130
Jan  1 18:05:12 core kernel:  [<c038944e>] schedule+0x2ce/0x4d0
Jan  1 18:05:12 core kernel:  [<c0148171>] sys_read+0x51/0x80
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: smbd          D C04C0120     0 12172      1         13162  9058 (NOTLB)
Jan  1 18:05:12 core kernel: e1763c24 00000082 d696da40 c04c0120 f7cbdec0 c03762de d0130400 f7cbdec0 
Jan  1 18:05:12 core kernel:        000d4246 e30867f8 000280ff d696da40 d696db9c f7cbdec0 e1763c60 f7cbdf38 
Jan  1 18:05:12 core kernel:        e1763c40 c037800d f7cbdec0 00000000 da6b95f8 e1762000 00000000 00000000 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c03762de>] xprt_prepare_transmit+0x7e/0xc0
Jan  1 18:05:12 core kernel:  [<c037800d>] __rpc_execute+0x13d/0x3c0
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c0124c60>] autoremove_wake_function+0x0/0x60
Jan  1 18:05:12 core kernel:  [<c03786b6>] rpc_new_task+0x36/0xb0
Jan  1 18:05:12 core kernel:  [<c0373a24>] rpc_call_sync+0x74/0xb0
Jan  1 18:05:12 core kernel:  [<c01dd6fa>] nfs3_rpc_wrapper+0x3a/0x80
Jan  1 18:05:12 core kernel:  [<c01ddc9a>] nfs3_proc_access+0xda/0x170
Jan  1 18:05:12 core kernel:  [<c01bdc75>] __journal_file_buffer+0x175/0x230
Jan  1 18:05:12 core kernel:  [<c01bd02f>] journal_dirty_metadata+0xef/0x170
Jan  1 18:05:12 core kernel:  [<c037973c>] rpcauth_lookup_credcache+0x1cc/0x210
Jan  1 18:05:12 core kernel:  [<c01d29c5>] nfs_do_access+0x65/0xb0
Jan  1 18:05:12 core kernel:  [<c01d2b00>] nfs_permission+0xf0/0x170
Jan  1 18:05:12 core kernel:  [<c0154241>] permission+0x51/0x60
Jan  1 18:05:12 core kernel:  [<c01551c2>] link_path_walk+0xa92/0xb60
Jan  1 18:05:12 core kernel:  [<c0160120>] update_atime+0xd0/0xe0
Jan  1 18:05:12 core kernel:  [<c0154e0d>] link_path_walk+0x6dd/0xb60
Jan  1 18:05:12 core kernel:  [<c01554e0>] path_lookup+0x70/0x110
Jan  1 18:05:12 core kernel:  [<c0155733>] __user_walk+0x33/0x60
Jan  1 18:05:12 core kernel:  [<c0150a7f>] vfs_stat+0x1f/0x60
Jan  1 18:05:12 core kernel:  [<c01511ab>] sys_stat64+0x1b/0x40
Jan  1 18:05:12 core kernel:  [<c0103ee8>] math_state_restore+0x28/0x50
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: nmbd          S C04C0120     0 13162      1         13438 12172 (NOTLB)
Jan  1 18:05:12 core kernel: f6ef1eb4 00200086 f7d585a0 c04c0120 c473dc18 c012ff73 c132a260 00000000 
Jan  1 18:05:12 core kernel:        000169ae 1415a4e0 0002817a f7d585a0 f7d586fc 2a0aee26 f6ef1ec8 0000000b 
Jan  1 18:05:12 core kernel:        0000000b c03899e3 f6ef1ec8 2a0aee26 f6ec60c0 c04079d8 d5349ec8 2a0aee26 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c012ff73>] __get_free_pages+0x33/0x40
Jan  1 18:05:12 core kernel:  [<c03899e3>] schedule_timeout+0x63/0xc0
Jan  1 18:05:12 core kernel:  [<c011aa90>] process_timeout+0x0/0x10
Jan  1 18:05:12 core kernel:  [<c0159693>] do_select+0x173/0x2b0
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c0159abf>] sys_select+0x2bf/0x4d0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: mutt          S C04C0150     0 13427  26545                     (NOTLB)
Jan  1 18:05:12 core kernel: e751ff10 00000086 d4370ae0 c04c0150 00028179 00000001 f32707b5 00028179 
Jan  1 18:05:12 core kernel:        00000f9a f3bf6150 00028179 f2996080 f29961dc 2a13ecb8 e751ff24 e751ff68 
Jan  1 18:05:12 core kernel:        000927c1 c03899e3 e751ff24 2a13ecb8 00000145 c04c61e0 c04c61e0 2a13ecb8 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c03899e3>] schedule_timeout+0x63/0xc0
Jan  1 18:05:12 core kernel:  [<c011aa90>] process_timeout+0x0/0x10
Jan  1 18:05:12 core kernel:  [<c0159e0a>] do_poll+0xaa/0xd0
Jan  1 18:05:12 core kernel:  [<c0159f82>] sys_poll+0x152/0x210
Jan  1 18:05:12 core kernel:  [<c01163fb>] sys_gettimeofday+0x3b/0x80
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: xterm         S C04C0120     0 13438      1 13440         13162 (NOTLB)
Jan  1 18:05:12 core kernel: ca025eb4 00000082 f6fb1140 c04c0120 00000010 00000000 00000096 ec92f240 
Jan  1 18:05:12 core kernel:        00000b61 43a9a434 0002817b f6fb1140 f6fb129c 2a0adee6 ca025ec8 00000006 
Jan  1 18:05:12 core kernel:        00000006 c03899e3 ca025ec8 2a0adee6 f6805000 f57a7ec8 f0cf1ec8 2a0adee6 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c03899e3>] schedule_timeout+0x63/0xc0
Jan  1 18:05:12 core kernel:  [<c011aa90>] process_timeout+0x0/0x10
Jan  1 18:05:12 core kernel:  [<c0159693>] do_select+0x173/0x2b0
Jan  1 18:05:12 core kernel:  [<c0159370>] __pollwait+0x0/0xd0
Jan  1 18:05:12 core kernel:  [<c0159abf>] sys_select+0x2bf/0x4d0
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: bash          S C04C0150     0 13440  13438 13443               (NOTLB)
Jan  1 18:05:12 core kernel: f4179f1c 00000086 d3a0db00 c04c0150 0002813e f30432a0 27a9401d 0002813e 
Jan  1 18:05:12 core kernel:        0002baaf 27a9401d 0002813e f6fb1620 f6fb177c fffffe00 f6fb1620 f6fb16c4 
Jan  1 18:05:12 core kernel:        f6fb16c4 c01159cd ffffffff 00000006 f1b15100 f4179f50 00000292 00030002 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c01159cd>] do_wait+0x18d/0x460
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c022cf2e>] copy_to_user+0x3e/0x50
Jan  1 18:05:12 core kernel:  [<c0115d6f>] sys_wait4+0x3f/0x50
Jan  1 18:05:12 core kernel:  [<c0115da7>] sys_waitpid+0x27/0x2b
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb
Jan  1 18:05:12 core kernel: bash          S C04C0150     0 13443  13440                     (NOTLB)
Jan  1 18:05:12 core kernel: f169fe70 00000086 f6fb1140 c04c0150 00028143 c038944e 600f9ce1 00028143 
Jan  1 18:05:12 core kernel:        000013b7 60106e9a 00028143 f1b15100 f1b1525c e000f000 7fffffff edbe8000 
Jan  1 18:05:12 core kernel:        c081dc40 c0389a35 00000002 e7593c0f c0272c48 f6805000 e7593c11 00000000 
Jan  1 18:05:12 core kernel: Call Trace:
Jan  1 18:05:12 core kernel:  [<c038944e>] schedule+0x2ce/0x4d0
Jan  1 18:05:12 core kernel:  [<c0389a35>] schedule_timeout+0xb5/0xc0
Jan  1 18:05:12 core kernel:  [<c0272c48>] pty_write+0x68/0x70
Jan  1 18:05:12 core kernel:  [<c0271a73>] read_chan+0x5e3/0x6f0
Jan  1 18:05:12 core kernel:  [<c0271cdd>] write_chan+0x15d/0x210
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c010f740>] default_wake_function+0x0/0x20
Jan  1 18:05:12 core kernel:  [<c026c94e>] tty_write+0x20e/0x260
Jan  1 18:05:12 core kernel:  [<c026c721>] tty_read+0xe1/0x100
Jan  1 18:05:12 core kernel:  [<c0147e88>] vfs_read+0xb8/0x130
Jan  1 18:05:12 core kernel:  [<c0148171>] sys_read+0x51/0x80
Jan  1 18:05:12 core kernel:  [<c010254b>] syscall_call+0x7/0xb

I was not able to reproduce as of now, because i have to use the system.

Christian Leber

Signature · 1 line
http://www.nosoftwarepatents.com
#5 lore raw
In reply to Gildas LE NADAN
Quoted 3 lines · Gildas LE NADAN

I experience hangs on samba processes on a filer using xfs over lvm2 as data partitions, when there is active snapshots of the xfs partitions.

Your problem probably lies between lvm2 and XFS. I got the same problems this summer while doing the exact same thing.

The server would just completely hang once I started doing lvm snapshots:
-> at the beginning, the snapshots would work OK, but XFS would hang
when accessing the filesystem afterwards
-> after a while (usually 2 or 3 snapshots, and I was taking a
snapshot every 2 hours), the snapshot would not complete, and then only a hard reboot would work

I was doing the snapshots from a crontab, the script used xfs_freeze to freeze the filesystem before doing the snapshot (and unfreeze it afterwards, of course). Sometimes xfs_freeze -u would hang too (but at this time, the server was in a pretty bad state already).

The server wasn't loaded at all, we were doing some reads/writes through samba to have some modified files lying around, but we were mainly prototyping the server, not stress-testing it.

LVM and XFS just don't play nice together when it comes to snapshots, I thought it had been fixed already, but it's not the case, as we both know...

(I can't remember the kernel version, it could have been a 2.4 kernel, but I was using LVM2 and the latest XFS code available)

Feel free to correct me if I did something wrong (but AFAIK I took care of everything, knowing there could have been bad interactions between LVM and XFS).

JB.

Signature · 2 lines
Julien BLACHE                                   <http://www.jblache.org> 
<[email protected]>                                  GPG KeyID 0xF5D65169
#6 lore raw
In reply to Gildas LE NADAN
Quoted 29 lines · Gildas LE NADAN

Hi,

I experience hangs on samba processes on a filer using xfs over lvm2 as data partitions, when there is active snapshots of the xfs partitions.

I have a clone of the production server (same software, same hardware) where the situation can be reproduced perfectly.

Testings showed that the result was the same, whether the snapshots were mounted or not : smbd processes are locked and unkillable while the machine is normaly working otherwise, except software reboot is impossible and hardware reset is needed.

I noticed Brad Fitzpatrick's case in kernel 2.6.10 changelog
(http://lkml.org/lkml/2004/11/14/98) and tested kernel 2.6.10 today without success.

Configuration is the following :
- supermicro m/b with dual Xeon 2,8Ghz (SMT is active)
- 1 GB ram,
- adaptec u320 raid controler
- kernel 2.6.10
- debian sarge
- samba 3
- LVM2
- XFS with quota turned on

All software are from debian sarge packages, except the kernel.

Quoted 7 lines

I'm not able to determine if the problem is more xfs, device mapper or samba related, and was not able to do extensive testings (using a different filesystem, testing with a different daemon, etc...), but SMT/SMP testings showed that this is not a SMP/SMT related problem.

I've compiled the kernel with the debugging options, so I might provide additional informations if needed as in Brad's case.

I'll try to reproduce your problems soon.