Skip to contentSkip to conversation

nfsd threads locked, 2.6.7 & ia64

DISCUSSION Garrick Staples 3 participants 7 messages Jun 27, 2004 – Jun 28, 2004

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Garrick Staples

author
#0 lore raw

Well here's something I've not seen yet. All 512 nfsd threads are stuck in IO wait (state D in ps). All clients are hung on that mount. The actual filesystem on the server seems fine.

It start at 8am with no messages at all.

Jun 26 08:06:58 hpc-master nagios: SERVICE ALERT: hpc-fs3;nfs;CRITICAL;SOFT;1;CRITICAL: RPC program nfs  version 3 udp is not running
Jun 26 08:07:21 hpc934-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Jun 26 08:09:12 hpc972-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Jun 26 08:09:13 hpc941-e0 kernel: nfs: server hpc-fs3 not responding, still trying

...

I can't find anything otherwise wrong with the machine at all, just that nfsd threads are stuck. rpcinfo and showmount still work fine. Nothing in dmesg or messages. I'm stuck too.

Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator
University of Southern California
#1 lore raw
Quoted 10 lines · Garrick Staples

Well here's something I've not seen yet. All 512 nfsd threads are stuck in IO wait (state D in ps). All clients are hung on that mount. The actual filesystem on the server seems fine.

It start at 8am with no messages at all.

Jun 26 08:06:58 hpc-master nagios: SERVICE ALERT: hpc-fs3;nfs;CRITICAL;SOFT;1;CRITICAL: RPC program nfs  version 3 udp is not running
Jun 26 08:07:21 hpc934-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Jun 26 08:09:12 hpc972-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Jun 26 08:09:13 hpc941-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Quoted 5 lines

...

I can't find anything otherwise wrong with the machine at all, just that nfsd threads are stuck. rpcinfo and showmount still work fine. Nothing in dmesg or messages. I'm stuck too.

Does Sysrq-T give you any idea where they're stuck?

--Bruce Fields

-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training. Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital self defense, top technical experts, no vendor pitches, unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#2 lore raw

On Sun, Jun 27, 2004 at 03:33:14PM -0400, J. Bruce Fields alleged:

Quoted 13 lines · Garrick Staples
Quoted 12 lines

Well here's something I've not seen yet. All 512 nfsd threads are stuck in IO wait (state D in ps). All clients are hung on that mount. The actual filesystem on the server seems fine.

It start at 8am with no messages at all.

Jun 26 08:06:58 hpc-master nagios: SERVICE ALERT: hpc-fs3;nfs;CRITICAL;SOFT;1;CRITICAL: RPC program nfs  version 3 udp is not running
Jun 26 08:07:21 hpc934-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Jun 26 08:09:12 hpc972-e0 kernel: nfs: server hpc-fs3 not responding, still trying
Jun 26 08:09:13 hpc941-e0 kernel: nfs: server hpc-fs3 not responding, still trying

...

Quoted 5 lines
Quoted 3 lines

I can't find anything otherwise wrong with the machine at all, just that nfsd threads are stuck. rpcinfo and showmount still work fine. Nothing in dmesg or messages. I'm stuck too.

Does Sysrq-T give you any idea where they're stuck?

I won't know until I get into the machine room tomorrow. Any info I can give you from a remote shell?

Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator
University of Southern California
#3 lore raw
Quoted 7 lines · On Sunday June 27, [email protected] wrote:

On Sun, Jun 27, 2004 at 03:33:14PM -0400, J. Bruce Fields alleged:

Quoted 2 lines

Does Sysrq-T give you any idea where they're stuck?

I won't know until I get into the machine room tomorrow. Any info I can give you from a remote shell?

# echo t > /proc/sysrq-trigger
# dmesg

NeilBrown

Quoted 3 lines
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator
University of Southern California

-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training. Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital self defense, top technical experts, no vendor pitches, unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#4 lore raw

On Mon, Jun 28, 2004 at 11:04:08AM +1000, Neil Brown alleged:

Quoted 11 lines · On Sunday June 27, [email protected] wrote:
Quoted 7 lines

On Sun, Jun 27, 2004 at 03:33:14PM -0400, J. Bruce Fields alleged:

Quoted 2 lines

Does Sysrq-T give you any idea where they're stuck?

I won't know until I get into the machine room tomorrow. Any info I can give you from a remote shell?

# echo t > /proc/sysrq-trigger
# dmesg

Learn something new every day :)

As you may remember, I have a pair of these ia64 machines. The one with the stuck threads is currently only serving 1 xfs filesystem. The other machine currently has 8 ext3 filesystems.

Below is 3 different traces of nfsd, all 512 threads seem to follow one of those 3 patterns.

nfsd D a000000100560810 0 2462 1 2406 2404 (L-TLB)

Call Trace:
 [<a000000100561a00>] schedule+0xd20/0x12a0
                                sp=e0000000059cfaa0 bsp=e0000000059c9290
 [<a000000100560810>] __down+0x210/0x320
                                sp=e0000000059cfab0 bsp=e0000000059c9230
 [<a000000200723ef0>] linvfs_writev+0x290/0x320 [xfs]
                                sp=e0000000059cfae0 bsp=e0000000059c91d8
 [<a000000100126110>] do_readv_writev+0x330/0x500
                                sp=e0000000059cfc10 bsp=e0000000059c9170
 [<a0000002003e8be0>] nfsd_write+0x1c0/0x7e0 [nfsd]
                                sp=e0000000059cfc90 bsp=e0000000059c90f8
 [<a0000002003fc360>] nfsd3_proc_write+0x180/0x260 [nfsd]
                                sp=e0000000059cfdf0 bsp=e0000000059c90a8
 [<a0000002003df030>] nfsd_dispatch+0x290/0x540 [nfsd]
                                sp=e0000000059cfdf0 bsp=e0000000059c9058
 [<a00000020039bde0>] svc_process+0x10a0/0x1380 [sunrpc]
                                sp=e0000000059cfdf0 bsp=e0000000059c8fe8
 [<a0000002003de8e0>] nfsd+0x500/0x9c0 [nfsd]
                                sp=e0000000059cfe00 bsp=e0000000059c8ee8
 [<a00000010001b380>] kernel_thread_helper+0xe0/0x100
                                sp=e0000000059cfe30 bsp=e0000000059c8ec0
 [<a000000100009080>] start_kernel_thread+0x20/0x40
                                sp=e0000000059cfe30 bsp=e0000000059c8ec0

nfsd D a000000100560810 0 2400 1 2458 2455 (L-TLB)

Call Trace:
 [<a000000100561a00>] schedule+0xd20/0x12a0
                                sp=e00000003e13fcb0 bsp=e00000003e1391b8
 [<a000000100560810>] __down+0x210/0x320
                                sp=e00000003e13fcc0 bsp=e00000003e139158
 [<a0000002003e7cc0>] nfsd_sync+0x240/0x280 [nfsd]
                                sp=e00000003e13fcf0 bsp=e00000003e139118
 [<a0000002003e9360>] nfsd_commit+0x160/0x180 [nfsd]
                                sp=e00000003e13fcf0 bsp=e00000003e1390e8
 [<a0000002003fe720>] nfsd3_proc_commit+0x180/0x220 [nfsd]
                                sp=e00000003e13fdf0 bsp=e00000003e1390a8
 [<a0000002003df030>] nfsd_dispatch+0x290/0x540 [nfsd]
                                sp=e00000003e13fdf0 bsp=e00000003e139058
 [<a00000020039bde0>] svc_process+0x10a0/0x1380 [sunrpc]
                                sp=e00000003e13fdf0 bsp=e00000003e138fe8
 [<a0000002003de8e0>] nfsd+0x500/0x9c0 [nfsd]
                                sp=e00000003e13fe00 bsp=e00000003e138ee8
 [<a00000010001b380>] kernel_thread_helper+0xe0/0x100
                                sp=e00000003e13fe30 bsp=e00000003e138ec0
 [<a000000100009080>] start_kernel_thread+0x20/0x40
                                sp=e00000003e13fe30 bsp=e00000003e138ec0

nfsd D a000000100562bd0 0 2455 1 2400 2399 (L-TLB)

Call Trace:
 [<a000000100561a00>] schedule+0xd20/0x12a0
                                sp=e000000005aa76c0 bsp=e000000005aa18b0
 [<a000000100562bd0>] io_schedule+0x70/0xa0
                                sp=e000000005aa76d0 bsp=e000000005aa1898
 [<a0000001000dbf00>] __lock_page+0x260/0x2e0
                                sp=e000000005aa76d0 bsp=e000000005aa1860
 [<a000000100178730>] mpage_writepages+0x290/0x700
                                sp=e000000005aa7750 bsp=e000000005aa1780
 [<a0000001000e8c00>] do_writepages+0xe0/0x100
                                sp=e000000005aa7800 bsp=e000000005aa1758
 [<a0000001000daf00>] __filemap_fdatawrite+0x160/0x180
                                sp=e000000005aa7800 bsp=e000000005aa1738
 [<a00000020072eb60>] xfs_flush_inode+0x40/0x60 [xfs]
                                sp=e000000005aa7880 bsp=e000000005aa1718
 [<a0000002006da680>] xfs_flush_space+0x1c0/0x200 [xfs]
                                sp=e000000005aa7880 bsp=e000000005aa16f0
 [<a0000002006db2a0>] xfs_iomap_write_delay+0x560/0x760 [xfs]
                                sp=e000000005aa7880 bsp=e000000005aa1618
 [<a0000002006da110>] xfs_iomap+0x450/0x800 [xfs]
                                sp=e000000005aa7930 bsp=e000000005aa15a0
 [<a00000020072e060>] xfs_bmap+0x40/0x60 [xfs]
                                sp=e000000005aa7970 bsp=e000000005aa1558
 [<a00000020071c160>] linvfs_get_block_core+0xe0/0x5c0 [xfs]
                                sp=e000000005aa7970 bsp=e000000005aa14f0
 [<a00000010012e790>] __block_prepare_write+0x5f0/0xa60
                                sp=e000000005aa79b0 bsp=e000000005aa1468
 [<a00000010012fc80>] block_prepare_write+0x40/0xa0
                                sp=e000000005aa79e0 bsp=e000000005aa1438
 [<a00000020071d1d0>] linvfs_prepare_write+0x90/0xc0 [xfs]
                                sp=e000000005aa79e0 bsp=e000000005aa1400
 [<a0000001000dfbf0>] generic_file_aio_write_nolock+0x810/0x12e0
                                sp=e000000005aa79e0 bsp=e000000005aa1308
 [<a00000020072d540>] xfs_write+0x3e0/0xda0 [xfs]
                                sp=e000000005aa7ab0 bsp=e000000005aa1230
 [<a000000200723e10>] linvfs_writev+0x1b0/0x320 [xfs]
                                sp=e000000005aa7ae0 bsp=e000000005aa11d8
 [<a000000100126110>] do_readv_writev+0x330/0x500
                                sp=e000000005aa7c10 bsp=e000000005aa1170
 [<a0000002003e8be0>] nfsd_write+0x1c0/0x7e0 [nfsd]
                                sp=e000000005aa7c90 bsp=e000000005aa10f8
 [<a0000002003fc360>] nfsd3_proc_write+0x180/0x260 [nfsd]
                                sp=e000000005aa7df0 bsp=e000000005aa10a8
 [<a0000002003df030>] nfsd_dispatch+0x290/0x540 [nfsd]
                                sp=e000000005aa7df0 bsp=e000000005aa1058
 [<a00000020039bde0>] svc_process+0x10a0/0x1380 [sunrpc]
                                sp=e000000005aa7df0 bsp=e000000005aa0fe8
 [<a0000002003de8e0>] nfsd+0x500/0x9c0 [nfsd]
                                sp=e000000005aa7e00 bsp=e000000005aa0ee8
 [<a00000010001b380>] kernel_thread_helper+0xe0/0x100
                                sp=e000000005aa7e30 bsp=e000000005aa0ec0
 [<a000000100009080>] start_kernel_thread+0x20/0x40
                                sp=e000000005aa7e30 bsp=e000000005aa0ec0
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator
University of Southern California
#5 lore raw
Quoted 4 lines · On Sunday June 27, [email protected] wrote:

As you may remember, I have a pair of these ia64 machines. The one with the stuck threads is currently only serving 1 xfs filesystem. The other machine currently has 8 ext3 filesystems.

Sounds like the finger is pointing at xfs ....

Quoted 7 lines

Below is 3 different traces of nfsd, all 512 threads seem to follow one of those 3 patterns.

nfsd D a000000100560810 0 2462 1 2406 2404 (L-TLB)

Quoted 7 lines
Call Trace:
 [<a000000100561a00>] schedule+0xd20/0x12a0
                                sp=e0000000059cfaa0 bsp=e0000000059c9290
 [<a000000100560810>] __down+0x210/0x320
                                sp=e0000000059cfab0 bsp=e0000000059c9230
 [<a000000200723ef0>] linvfs_writev+0x290/0x320 [xfs]
                                sp=e0000000059cfae0 bsp=e0000000059c91d8

So this is waiting to "down" inode->i_sem. Someone must be holding that semaphore already...

Quoted 10 lines

nfsd D a000000100560810 0 2400 1 2458 2455 (L-TLB)

Call Trace:
 [<a000000100561a00>] schedule+0xd20/0x12a0
                                sp=e00000003e13fcb0 bsp=e00000003e1391b8
 [<a000000100560810>] __down+0x210/0x320
                                sp=e00000003e13fcc0 bsp=e00000003e139158
 [<a0000002003e7cc0>] nfsd_sync+0x240/0x280 [nfsd]
                                sp=e00000003e13fcf0 bsp=e00000003e139118

This is waiting on the same semapore....

Quoted 4 lines

nfsd D a000000100562bd0 0 2455 1 2400 2399 (L-TLB)

This one is holding the semaphore (below)....

Quoted 7 lines
Call Trace:
 [<a000000100561a00>] schedule+0xd20/0x12a0
                                sp=e000000005aa76c0 bsp=e000000005aa18b0
 [<a000000100562bd0>] io_schedule+0x70/0xa0
                                sp=e000000005aa76d0 bsp=e000000005aa1898
 [<a0000001000dbf00>] __lock_page+0x260/0x2e0
                                sp=e000000005aa76d0 bsp=e000000005aa1860

... and it seems to be trying to lock a page. I wonder why it can't. Neither of the other threads should be holding a lock on this page.

Maybe if you put all the traces of all threads on a website somewhere, and send mail to [email protected] suggesting that you might have hit an XFS problem.

NeilBrown

Quoted 22 lines
 [<a000000100178730>] mpage_writepages+0x290/0x700
                                sp=e000000005aa7750 bsp=e000000005aa1780
 [<a0000001000e8c00>] do_writepages+0xe0/0x100
                                sp=e000000005aa7800 bsp=e000000005aa1758
 [<a0000001000daf00>] __filemap_fdatawrite+0x160/0x180
                                sp=e000000005aa7800 bsp=e000000005aa1738
 [<a00000020072eb60>] xfs_flush_inode+0x40/0x60 [xfs]
                                sp=e000000005aa7880 bsp=e000000005aa1718
 [<a0000002006da680>] xfs_flush_space+0x1c0/0x200 [xfs]
                                sp=e000000005aa7880 bsp=e000000005aa16f0
 [<a0000002006db2a0>] xfs_iomap_write_delay+0x560/0x760 [xfs]
                                sp=e000000005aa7880 bsp=e000000005aa1618
 [<a0000002006da110>] xfs_iomap+0x450/0x800 [xfs]
                                sp=e000000005aa7930 bsp=e000000005aa15a0
 [<a00000020072e060>] xfs_bmap+0x40/0x60 [xfs]
                                sp=e000000005aa7970 bsp=e000000005aa1558
 [<a00000020071c160>] linvfs_get_block_core+0xe0/0x5c0 [xfs]
                                sp=e000000005aa7970 bsp=e000000005aa14f0
 [<a00000010012e790>] __block_prepare_write+0x5f0/0xa60
                                sp=e000000005aa79b0 bsp=e000000005aa1468
 [<a00000010012fc80>] block_prepare_write+0x40/0xa0
                                sp=e000000005aa79e0 bsp=e000000005aa1438
Quoted 8 lines
 [<a00000020071d1d0>] linvfs_prepare_write+0x90/0xc0 [xfs]
                                sp=e000000005aa79e0 bsp=e000000005aa1400
 [<a0000001000dfbf0>] generic_file_aio_write_nolock+0x810/0x12e0
                                sp=e000000005aa79e0 bsp=e000000005aa1308
 [<a00000020072d540>] xfs_write+0x3e0/0xda0 [xfs]
                                sp=e000000005aa7ab0 bsp=e000000005aa1230
 [<a000000200723e10>] linvfs_writev+0x1b0/0x320 [xfs]
                                sp=e000000005aa7ae0 bsp=e000000005aa11d8

***This is where the sem is being held****

Quoted 8 lines
 [<a000000100126110>] do_readv_writev+0x330/0x500
                                sp=e000000005aa7c10 bsp=e000000005aa1170
 [<a0000002003e8be0>] nfsd_write+0x1c0/0x7e0 [nfsd]
                                sp=e000000005aa7c90 bsp=e000000005aa10f8
 [<a0000002003fc360>] nfsd3_proc_write+0x180/0x260 [nfsd]
                                sp=e000000005aa7df0 bsp=e000000005aa10a8
 [<a0000002003df030>] nfsd_dispatch+0x290/0x540 [nfsd]
                                sp=e000000005aa7df0 bsp=e000000005aa1058
Quoted 12 lines
 [<a00000020039bde0>] svc_process+0x10a0/0x1380 [sunrpc]
                                sp=e000000005aa7df0 bsp=e000000005aa0fe8
 [<a0000002003de8e0>] nfsd+0x500/0x9c0 [nfsd]
                                sp=e000000005aa7e00 bsp=e000000005aa0ee8
 [<a00000010001b380>] kernel_thread_helper+0xe0/0x100
                                sp=e000000005aa7e30 bsp=e000000005aa0ec0
 [<a000000100009080>] start_kernel_thread+0x20/0x40
                                sp=e000000005aa7e30 bsp=e000000005aa0ec0
Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator
University of Southern California

-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training. Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital self defense, top technical experts, no vendor pitches, unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#6 lore raw
In reply to Garrick Staples

On Sat, Jun 26, 2004 at 10:11:29PM -0700, Garrick Staples alleged:

Quoted 3 lines

Well here's something I've not seen yet. All 512 nfsd threads are stuck in IO wait (state D in ps). All clients are hung on that mount. The actual filesystem on the server seems fine.

I just ran a few 'find's through the 4TB filesystem and 1 of them got stuck. I guess the filesystem hosed itself. Guess I'll go back to reiser.

*sigh* I just found this patch for random data corruption over NFS.
http://linus.bkbits.net:8080/linux-2.5/cset@40d263bcB5wLHA2ALqKxBm1mzuGiFw?nav=index.html|tags|[email protected]..

But it doesn't mention anything about getting stuck in IO wait.

Signature · 2 lines
Garrick Staples, Linux/HPCC Administrator
University of Southern California