Skip to contentSkip to conversation

processes in D state too long too often

DISCUSSION Gary L. Grobe 2 participants 2 messages Feb 10, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw
Quoted 12 lines

Presumably that's the msleep(10) in nfsd_vfs_write().

That wouldn't explain the same nfsd thread waiting for several seconds, though. Or was it just that that several seconds during which different nfsd threads were stuck in D, not necessarily the same ones?

What does your /etc/exports file look like on the server, and what are the mount options on the client?

You could try turning off that msleep() with no_wdelay, but it may not help.

Quoted 8 lines

The more likely explanation is that you just switched to a more recent distro where "sync" (as opposed to "async") is the option. Depending on workload, "async" may improve performance a great deal, at the expense of possible data corruption on server reboot!

If you're doing a lot of writing and using NFSv2, then switching to NFSv3 may give you performance close to the "async" performance without the corruption worries.

Apologies for the unintentional separate thread.

I really think I'm seeing the same nfsd threads going into D for a very short time. Here's what's in /etc/exports ...

/diskless/10.0.1.1 10.0.1.1(sync,rw,no_root_squash,no_all_squash,no_subtree_check) /diskless/10.0.1.2 10.0.1.2(sync,rw,no_root_squash,no_all_squash,no_subtree_check) /diskless/10.0.1.3 10.0.1.3(sync,rw,no_root_squash,no_all_squash,no_subtree_check)

# ... same lines above for another 80+ nodes

# Common to all slave nodes.

/usr    10.0.0.0/16(sync,ro,subtree_check,no_root_squash,no_all_squash)
/opt    10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)
/home   10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)

#/var/log 10.0.0.0/16(sync,rw,subtree_check,no_root_squash,no_all_squash)

Mount options on each client are as follows ...

10.0.0.10:/diskless/10.0.1.1    /   nfs     sync,hard,intr,rw,rsize=8192,wsize=8192 0   0
10.0.0.10:/opt          /opt    nfs sync,hard,intr,rw,rsize=8192,wsize=8192 0   0
10.0.0.10:/usr          /usr    nfs sync,hard,intr,ro,rsize=8192,wsize=8192 0   0
10.0.0.10:/home         /home   nfs sync,hard,intr,rw,rsize=8192,wsize=8192 0   0
none                    /proc   proc    defaults    0 0

#10.0.0.10:/var/log /var/log nfs sync,hard,intr,rw 0 0

I'm not following turning off the msleep() option. Where are you referring to this from?

I've got NFSv3 enabled and have used this in a previous installation (using the same distro, gentoo) on this same hardware with no issues, and 'sync', and the performance was much better. Something worth noting, I've rolled back my kernel several times now and each time I go back (w/ same vers on master and slave node), the D state time in simulation processes keeps getting better (cut down). I went from 2.6.27-r7 to 2.6.24-r8 and now I'm running 2.6.20-r10, and each one better than the previous (and later) kernel. I was running 2.6.18-r2 in the past, which I'm having difficulties getting at the moment.

#1 lore raw
Quoted 38 lines · Gary L. Grobe
Quoted 12 lines

Presumably that's the msleep(10) in nfsd_vfs_write().

That wouldn't explain the same nfsd thread waiting for several seconds, though. Or was it just that that several seconds during which different nfsd threads were stuck in D, not necessarily the same ones?

What does your /etc/exports file look like on the server, and what are the mount options on the client?

You could try turning off that msleep() with no_wdelay, but it may not help.

Quoted 8 lines

The more likely explanation is that you just switched to a more recent distro where "sync" (as opposed to "async") is the option. Depending on workload, "async" may improve performance a great deal, at the expense of possible data corruption on server reboot!

If you're doing a lot of writing and using NFSv2, then switching to NFSv3 may give you performance close to the "async" performance without the corruption worries.

Apologies for the unintentional separate thread.

I really think I'm seeing the same nfsd threads going into D for a very short time. Here's what's in /etc/exports ...

/diskless/10.0.1.1 10.0.1.1(sync,rw,no_root_squash,no_all_squash,no_subtree_check) /diskless/10.0.1.2 10.0.1.2(sync,rw,no_root_squash,no_all_squash,no_subtree_check) /diskless/10.0.1.3 10.0.1.3(sync,rw,no_root_squash,no_all_squash,no_subtree_check)

# ... same lines above for another 80+ nodes

# Common to all slave nodes.

/usr    10.0.0.0/16(sync,ro,subtree_check,no_root_squash,no_all_squash)
/opt    10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)
/home   10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)

#/var/log 10.0.0.0/16(sync,rw,subtree_check,no_root_squash,no_all_squash)

Mount options on each client are as follows ...

Quoted 8 lines
10.0.0.10:/diskless/10.0.1.1    /   nfs     sync,hard,intr,rw,rsize=8192,wsize=8192 0   0
10.0.0.10:/opt          /opt    nfs sync,hard,intr,rw,rsize=8192,wsize=8192 0   0
10.0.0.10:/usr          /usr    nfs sync,hard,intr,ro,rsize=8192,wsize=8192 0   0
10.0.0.10:/home         /home   nfs sync,hard,intr,rw,rsize=8192,wsize=8192 0   0
none                    /proc   proc    defaults    0 0

#10.0.0.10:/var/log /var/log nfs sync,hard,intr,rw 0 0

I'm not following turning off the msleep() option. Where are you referring to this from?

If you add no_wdelay to the export options, then we won't see the mdelay() calls in the sysrq-w output anymore. I doubt that'll solve the problem, but it may be worth a try just to see what changes.

Quoted 3 lines

I've got NFSv3 enabled and have used this in a previous installation (using the same distro, gentoo) on this same hardware with no issues, and 'sync', and the performance was much better.

OK, then I'm out of theories for now....

--b.

Quoted 15 lines

Something worth
noting, I've rolled back my kernel several times now and each time I go back (w/ same vers on master and slave node), the D state time in simulation processes keeps getting better (cut down). I went from 2.6.27-r7 to 2.6.24-r8 and now I'm running 2.6.20-r10, and each one better than the previous (and later) kernel. I was running 2.6.18-r2 in the past, which I'm having difficulties getting at the moment.

-- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html