Skip to contentSkip to conversation

nfs: don't call blocking operations while !TASK_RUNNING

PATCH Superseded Jeff Layton 5 participants 9 messages Jan 14, 2015 – Mar 20, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nfs: don't call blocking operations while !TASK_RUNNING

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

---
 fs/nfs/callback.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/nfs/callback.c +3 -2
index b8fb3a4ef649..ebba958e084c 100644
@@ -128,13 +128,14 @@ nfs41_callback_svc(void *vrqstp)
 		if (try_to_freeze())
 			continue;
 
-		prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_INTERRUPTIBLE);
+		prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_UNINTERRUPTIBLE);
 		spin_lock_bh(&serv->sv_cb_lock);
 		if (!list_empty(&serv->sv_cb_list)) {
 			req = list_first_entry(&serv->sv_cb_list,
 					struct rpc_rqst, rq_bc_list);
 			list_del(&req->rq_bc_list);
 			spin_unlock_bh(&serv->sv_cb_lock);
+			finish_wait(&serv->sv_cb_waitq, &wq);
 			dprintk("Invoking bc_svc_process()\n");
 			error = bc_svc_process(serv, req, rqstp);
 			dprintk("bc_svc_process() returned w/ error code= %d\n",
@@ -142,8 +143,8 @@ nfs41_callback_svc(void *vrqstp)
 		} else {
 			spin_unlock_bh(&serv->sv_cb_lock);
 			schedule();
+			finish_wait(&serv->sv_cb_waitq, &wq);
 		}
-		finish_wait(&serv->sv_cb_waitq, &wq);
 	}
 	return 0;
 }
Signature · 1 line
2.1.0
#1 lore raw
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---
Quoted 3 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

ACK.

Quoted 2 lines

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

Umm... Won't that end up triggering the hung task watchdog for every 120seconds with no callback activity?

Quoted 39 lines
---
 fs/nfs/callback.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/nfs/callback.c +3 -3
index b8fb3a4ef649..ebba958e084c 100644
@@ -128,13 +128,14 @@ nfs41_callback_svc(void *vrqstp)
                if (try_to_freeze())
                        continue;
 
-               prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_INTERRUPTIBLE);
+               prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_UNINTERRUPTIBLE);
                spin_lock_bh(&serv->sv_cb_lock);
                if (!list_empty(&serv->sv_cb_list)) {
                        req = list_first_entry(&serv->sv_cb_list,
                                        struct rpc_rqst, rq_bc_list);
                        list_del(&req->rq_bc_list);
                        spin_unlock_bh(&serv->sv_cb_lock);
+                       finish_wait(&serv->sv_cb_waitq, &wq);
                        dprintk("Invoking bc_svc_process()\n");
                        error = bc_svc_process(serv, req, rqstp);
                        dprintk("bc_svc_process() returned w/ error code= %d\n",
@@ -142,8 +143,8 @@ nfs41_callback_svc(void *vrqstp)
                } else {
                        spin_unlock_bh(&serv->sv_cb_lock);
                        schedule();
+                       finish_wait(&serv->sv_cb_waitq, &wq);
                }
-               finish_wait(&serv->sv_cb_waitq, &wq);
        }
        return 0;
 }
--

2.1.0

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
#2 lore raw
Quoted 40 lines · On Wed, 14 Jan 2015 12:13:43 -0500 Trond Myklebust…
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---
Quoted 3 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Quoted 8 lines

ACK.

Quoted 2 lines

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

Umm... Won't that end up triggering the hung task watchdog for every 120seconds with no callback activity?

Doh! You're correct.

What's the right way to do this then? Do we need to use schedule_timeout and wake up every 100s or so? Using TASK_INTERRUPTIBLE to work around the watchdog seems wrong but I guess we can live with that in the short term if it's the only way.

Long-term, it would probably make sense to convert this (and other service threads like this) over to workqueue-based services. I still need to chase down where the latency comes from before that patchset is ready for merge though.

Quoted 43 lines
Quoted 40 lines
---
 fs/nfs/callback.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/nfs/callback.c +2 -1
index b8fb3a4ef649..ebba958e084c 100644
@@ -128,13 +128,14 @@ nfs41_callback_svc(void *vrqstp)
                if (try_to_freeze())
                        continue;
 
-               prepare_to_wait(&serv->sv_cb_waitq, &wq,
 TASK_INTERRUPTIBLE);
+               prepare_to_wait(&serv->sv_cb_waitq, &wq,
 TASK_UNINTERRUPTIBLE); spin_lock_bh(&serv->sv_cb_lock);
                if (!list_empty(&serv->sv_cb_list)) {
                        req = list_first_entry(&serv->sv_cb_list,
                                        struct rpc_rqst,
 rq_bc_list); list_del(&req->rq_bc_list);
                        spin_unlock_bh(&serv->sv_cb_lock);
+                       finish_wait(&serv->sv_cb_waitq, &wq);
                        dprintk("Invoking bc_svc_process()\n");
                        error = bc_svc_process(serv, req, rqstp);

dprintk("bc_svc_process() returned w/ error
code= %d\n", @@ -142,8 +143,8 @@ nfs41_callback_svc(void *vrqstp)

                } else {
                        spin_unlock_bh(&serv->sv_cb_lock);
                        schedule();
fs/nfs/callback.c +1 -2
+                       finish_wait(&serv->sv_cb_waitq, &wq);
                }
-               finish_wait(&serv->sv_cb_waitq, &wq);
        }
        return 0;
 }
--

2.1.0

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 52 lines · Jeff Layton
Quoted 40 lines · On Wed, 14 Jan 2015 12:13:43 -0500 Trond Myklebust…
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---
Quoted 3 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Quoted 8 lines

ACK.

Quoted 2 lines

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

Umm... Won't that end up triggering the hung task watchdog for every 120seconds with no callback activity?

Quoted 6 lines

Doh! You're correct.

What's the right way to do this then? Do we need to use schedule_timeout and wake up every 100s or so? Using TASK_INTERRUPTIBLE to work around the watchdog seems wrong but I guess we can live with that in the short term if it's the only way.

The alternative is to use TASK_KILLABLE. That's a little more restrictive, but still a PITA. Note the both TASK_INTERRUPTIBLE and TASK_KILLABLE will need something to handle that signalled() case, otherwise we end up with a permanent busy-waiting loop.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]
#4 lore raw
Quoted 61 lines · On Wed, 14 Jan 2015 13:06:17 -0500 Trond Myklebust…
Quoted 52 lines · Jeff Layton
Quoted 40 lines · On Wed, 14 Jan 2015 12:13:43 -0500 Trond Myklebust…
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---
Quoted 3 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Quoted 8 lines

ACK.

Quoted 2 lines

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

Umm... Won't that end up triggering the hung task watchdog for every 120seconds with no callback activity?

Quoted 6 lines

Doh! You're correct.

What's the right way to do this then? Do we need to use schedule_timeout and wake up every 100s or so? Using TASK_INTERRUPTIBLE to work around the watchdog seems wrong but I guess we can live with that in the short term if it's the only way.

Quoted 5 lines

The alternative is to use TASK_KILLABLE. That's a little more restrictive, but still a PITA. Note the both TASK_INTERRUPTIBLE and TASK_KILLABLE will need something to handle that signalled() case, otherwise we end up with a permanent busy-waiting loop.

Heh, our emails appear to have crossed. I went with a schedule_timeout for v2 so we can sidestep the whole issue of signals.

It's simple enough to add a flush_signals() call at the end of the loop, but I'm a little leery of what effects we might see if the thread catches a signal while in the middle of doing real work.

If you really think that's the better approach, I can do a v3 patch that does that however.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 73 lines · Jeff Layton
Quoted 61 lines · On Wed, 14 Jan 2015 13:06:17 -0500 Trond Myklebust…
Quoted 52 lines · Jeff Layton
Quoted 40 lines · On Wed, 14 Jan 2015 12:13:43 -0500 Trond Myklebust…
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

    ------------[ cut here ]------------
    WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
   do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
   Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
   CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
    0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
    0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
    ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
   Call Trace:
    [<ffffffff8186ac78>] dump_stack+0x4c/0x65
    [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
    [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
    [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
    [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
    [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
    [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
    [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
    [<ffffffff810d941e>] groups_alloc+0x3e/0x130
    [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
    [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
    [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
    [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
    [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
    [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
    [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
    [<ffffffff810d45bf>] kthread+0x11f/0x140
    [<ffffffff810ea815>] ? local_clock+0x15/0x30
    [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
    [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
   ---[ end trace 675220a11e30f4f2 ]---
Quoted 3 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Quoted 8 lines

ACK.

Quoted 2 lines

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

Umm... Won't that end up triggering the hung task watchdog for every 120seconds with no callback activity?

Quoted 6 lines

Doh! You're correct.

What's the right way to do this then? Do we need to use schedule_timeout and wake up every 100s or so? Using TASK_INTERRUPTIBLE to work around the watchdog seems wrong but I guess we can live with that in the short term if it's the only way.

Quoted 5 lines

The alternative is to use TASK_KILLABLE. That's a little more restrictive, but still a PITA. Note the both TASK_INTERRUPTIBLE and TASK_KILLABLE will need something to handle that signalled() case, otherwise we end up with a permanent busy-waiting loop.

Heh, our emails appear to have crossed. I went with a schedule_timeout for v2 so we can sidestep the whole issue of signals.

Quoted 6 lines

It's simple enough to add a flush_signals() call at the end of the loop, but I'm a little leery of what effects we might see if the thread catches a signal while in the middle of doing real work.

If you really think that's the better approach, I can do a v3 patch that does that however.

After applying this patch to a 3.19.0 test system, the WARNINGS are gone, but the system load average is pinned at 1.00.

root 12105 0.1 0.0 0 0 ? D Mar18 1:19 [nfsv4.1-svc]

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#6 lore raw
Quoted 80 lines · On Thu, 19 Mar 2015 17:59:15 -0700 Chuck Lever <[email protected]> wrote:
Quoted 77 lines · Jeff Layton
Quoted 61 lines · On Wed, 14 Jan 2015 13:06:17 -0500 Trond Myklebust…
Quoted 52 lines · Jeff Layton
Quoted 40 lines · On Wed, 14 Jan 2015 12:13:43 -0500 Trond Myklebust…
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

    ------------[ cut here ]------------
    WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
   do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
   Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
   CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
    0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
    0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
    ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
   Call Trace:
    [<ffffffff8186ac78>] dump_stack+0x4c/0x65
    [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
    [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
    [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
    [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
    [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
    [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
    [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
    [<ffffffff810d941e>] groups_alloc+0x3e/0x130
    [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
    [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
    [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
    [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
    [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
    [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
    [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
    [<ffffffff810d45bf>] kthread+0x11f/0x140
    [<ffffffff810ea815>] ? local_clock+0x15/0x30
    [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
    [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
   ---[ end trace 675220a11e30f4f2 ]---
Quoted 3 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Quoted 8 lines

ACK.

Quoted 2 lines

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead.

Umm... Won't that end up triggering the hung task watchdog for every 120seconds with no callback activity?

Quoted 6 lines

Doh! You're correct.

What's the right way to do this then? Do we need to use schedule_timeout and wake up every 100s or so? Using TASK_INTERRUPTIBLE to work around the watchdog seems wrong but I guess we can live with that in the short term if it's the only way.

Quoted 5 lines

The alternative is to use TASK_KILLABLE. That's a little more restrictive, but still a PITA. Note the both TASK_INTERRUPTIBLE and TASK_KILLABLE will need something to handle that signalled() case, otherwise we end up with a permanent busy-waiting loop.

Heh, our emails appear to have crossed. I went with a schedule_timeout for v2 so we can sidestep the whole issue of signals.

It's simple enough to add a flush_signals() call at the end of the loop, but I'm a little leery of what effects we might see if the thread catches a signal while in the middle of doing real work.

Quoted 8 lines
Quoted 2 lines

If you really think that's the better approach, I can do a v3 patch that does that however.

After applying this patch to a 3.19.0 test system, the WARNINGS are gone, but the system load average is pinned at 1.00.

root 12105 0.1 0.0 0 0 ? D Mar18 1:19 [nfsv4.1-svc]

Darn, good point...

I forgot about the stupid load avg on Linux counting D state tasks toward the load avg (why, oh why...).

I guess we will have to use a different task state after all -- probably TASK_KILLABLE and just try to ignore fatal signals as best we can. I'll plan to do that when I get a minute. Suggestions of better ways to do this would be appreciated though.

Cheers,

Signature · 1 line
Jeff Layton <[email protected]>
#7 lore raw

PATCH v2 1/1 nfs: don't call blocking operations while !TASK_RUNNING

In reply to Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead. That however, opens us up hung task warnings from the watchdog, so have the schedule_timeout wake up every 60s if there's no callback activity.

---
 fs/nfs/callback.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
fs/nfs/callback.c +5 -3
index b8fb3a4ef649..351be9205bf8 100644
@@ -128,22 +128,24 @@ nfs41_callback_svc(void *vrqstp)
 		if (try_to_freeze())
 			continue;
 
-		prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_INTERRUPTIBLE);
+		prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_UNINTERRUPTIBLE);
 		spin_lock_bh(&serv->sv_cb_lock);
 		if (!list_empty(&serv->sv_cb_list)) {
 			req = list_first_entry(&serv->sv_cb_list,
 					struct rpc_rqst, rq_bc_list);
 			list_del(&req->rq_bc_list);
 			spin_unlock_bh(&serv->sv_cb_lock);
+			finish_wait(&serv->sv_cb_waitq, &wq);
 			dprintk("Invoking bc_svc_process()\n");
 			error = bc_svc_process(serv, req, rqstp);
 			dprintk("bc_svc_process() returned w/ error code= %d\n",
 				error);
 		} else {
 			spin_unlock_bh(&serv->sv_cb_lock);
-			schedule();
+			/* schedule_timeout to game the hung task watchdog */
+			schedule_timeout(60 * HZ);
+			finish_wait(&serv->sv_cb_waitq, &wq);
 		}
-		finish_wait(&serv->sv_cb_waitq, &wq);
 	}
 	return 0;
 }
Signature · 1 line
2.1.0
#8 lore raw
Quoted 35 lines · Jeff Layton

Bruce reported seeing this warning pop when mounting using v4.1:

     ------------[ cut here ]------------
     WARNING: CPU: 1 PID: 1121 at kernel/sched/core.c:7300 __might_sleep+0xbd/0xd0()
    do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff810ff58f>] prepare_to_wait+0x2f/0x90
    Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ppdev joydev snd virtio_console virtio_balloon pcspkr serio_raw parport_pc parport pvpanic floppy soundcore i2c_piix4 virtio_blk virtio_net qxl drm_kms_helper ttm drm virtio_pci virtio_ring ata_generic virtio pata_acpi
    CPU: 1 PID: 1121 Comm: nfsv4.1-svc Not tainted 3.19.0-rc4+ #25
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
     0000000000000000 000000004e5e3f73 ffff8800b998fb48 ffffffff8186ac78
     0000000000000000 ffff8800b998fba0 ffff8800b998fb88 ffffffff810ac9da
     ffff8800b998fb68 ffffffff81c923e7 00000000000004d9 0000000000000000
    Call Trace:
     [<ffffffff8186ac78>] dump_stack+0x4c/0x65
     [<ffffffff810ac9da>] warn_slowpath_common+0x8a/0xc0
     [<ffffffff810aca65>] warn_slowpath_fmt+0x55/0x70
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810ff58f>] ? prepare_to_wait+0x2f/0x90
     [<ffffffff810dd2ad>] __might_sleep+0xbd/0xd0
     [<ffffffff8124c973>] kmem_cache_alloc_trace+0x243/0x430
     [<ffffffff810d941e>] ? groups_alloc+0x3e/0x130
     [<ffffffff810d941e>] groups_alloc+0x3e/0x130
     [<ffffffffa0301b1e>] svcauth_unix_accept+0x16e/0x290 [sunrpc]
     [<ffffffffa0300571>] svc_authenticate+0xe1/0xf0 [sunrpc]
     [<ffffffffa02fc564>] svc_process_common+0x244/0x6a0 [sunrpc]
     [<ffffffffa02fd044>] bc_svc_process+0x1c4/0x260 [sunrpc]
     [<ffffffffa03d5478>] nfs41_callback_svc+0x128/0x1f0 [nfsv4]
     [<ffffffff810ff970>] ? wait_woken+0xc0/0xc0
     [<ffffffffa03d5350>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
     [<ffffffff810d45bf>] kthread+0x11f/0x140
     [<ffffffff810ea815>] ? local_clock+0x15/0x30
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
     [<ffffffff81874bfc>] ret_from_fork+0x7c/0xb0
     [<ffffffff810d44a0>] ? kthread_create_on_node+0x250/0x250
    ---[ end trace 675220a11e30f4f2 ]---
Quoted 8 lines

nfs41_callback_svc does most of its work while in TASK_INTERRUPTIBLE, which is just wrong. Fix that by finishing the wait immediately if we've found that the list has something on it.

Also, we don't expect this kthread to accept signals, so we should be using a TASK_UNINTERRUPTIBLE sleep instead. That however, opens us up hung task warnings from the watchdog, so have the schedule_timeout wake up every 60s if there's no callback activity.

Works for me, for what it's worth.

Quoted 42 lines
---
 fs/nfs/callback.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
fs/nfs/callback.c +5 -3
index b8fb3a4ef649..351be9205bf8 100644
@@ -128,22 +128,24 @@ nfs41_callback_svc(void *vrqstp)
 		if (try_to_freeze())
 			continue;
 
-		prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_INTERRUPTIBLE);
+		prepare_to_wait(&serv->sv_cb_waitq, &wq, TASK_UNINTERRUPTIBLE);
 		spin_lock_bh(&serv->sv_cb_lock);
 		if (!list_empty(&serv->sv_cb_list)) {
 			req = list_first_entry(&serv->sv_cb_list,
 					struct rpc_rqst, rq_bc_list);
 			list_del(&req->rq_bc_list);
 			spin_unlock_bh(&serv->sv_cb_lock);
+			finish_wait(&serv->sv_cb_waitq, &wq);
 			dprintk("Invoking bc_svc_process()\n");
 			error = bc_svc_process(serv, req, rqstp);
 			dprintk("bc_svc_process() returned w/ error code= %d\n",
 				error);
 		} else {
 			spin_unlock_bh(&serv->sv_cb_lock);
-			schedule();
+			/* schedule_timeout to game the hung task watchdog */
+			schedule_timeout(60 * HZ);
+			finish_wait(&serv->sv_cb_waitq, &wq);
 		}
-		finish_wait(&serv->sv_cb_waitq, &wq);
 	}
 	return 0;
 }
Signature · 1 line
2.1.0