ps stuck on cmdline reading

Hello,

I am sometines triggering an issue on a v3.14.x (v3.14.18 here) where the ps auxwwf is stuck.
The setup is several containers which own several process and with memory limit on each cgroup.
A strace reveals the ps command is stuck on a read of a cmdline file. The concerned process itself is on a non-interruptible IO state. The ps command is executed in the global cgroup.

I also had a similar issue on a 3.10.x some months ago where ps was stuck; the reason was the memory limit of the cgroup was reached and I only had to add some pages available to the conatiner in order to unlock the ps command.
But in my case the cgroups which own the pid has still lots of memory available and I did not found a way to unlock the process.

I don't know how to reproduce the issue but I am sometimes triggering it.

Does someone has some hint? How can I get more debug info about it?

Thanks,

William

What does /proc/<ps pid>/stack say?

Michal Hocko
SUSE Labs

Thanks for the hint.
forgot to have a look; seems to be a nfs issue:

[<ffffffff810eda29>] sleep_on_page+0x9/0x20
[<ffffffff810ee8d4>] __lock_page+0xa4/0xb0
[<ffffffff810fe38f>] truncate_inode_pages_range+0x3ef/0x6a0
[<ffffffff810fe650>] truncate_inode_pages+0x10/0x20
[<ffffffff812474b6>] nfs4_evict_inode+0x16/0x40
[<ffffffff8117359f>] evict+0xaf/0x1c0
[<ffffffff811744c2>] iput+0x102/0x1a0
[<ffffffff812178d5>] nfs_dentry_iput+0x35/0x50
[<ffffffff8116f36c>] dentry_kill+0x16c/0x290
[<ffffffff8116f9d8>] dput+0xa8/0x160
[<ffffffff8121c96b>] __put_nfs_open_context+0xbb/0x100
[<ffffffff8121d1bb>] put_nfs_open_context+0xb/0x20
[<ffffffff81229ea4>] nfs_commitdata_release+0x14/0x30
[<ffffffff81229ee7>] nfs_commit_release+0x27/0x30
[<ffffffff815d23d2>] rpc_free_task+0x32/0x80
[<ffffffff815d24a5>] rpc_final_put_task+0x85/0x90
[<ffffffff815d2505>] rpc_do_put_task+0x35/0x40
[<ffffffff815d320b>] rpc_put_task+0xb/0x20
[<ffffffff81229fd0>] nfs_initiate_commit+0xe0/0x120
[<ffffffff8122a269>] nfs_commit_list+0x69/0xb0
[<ffffffff8122a36b>] nfs_commit_inode+0x9b/0x160
[<ffffffff8121b02b>] nfs_release_page+0x7b/0xa0
[<ffffffff810f18dd>] try_to_release_page+0x3d/0x60
[<ffffffff81100b06>] shrink_page_list+0x8a6/0x9e0
[<ffffffff81101233>] shrink_inactive_list+0x183/0x420
[<ffffffff81101b45>] shrink_lruvec+0x335/0x6f0
[<ffffffff81101f66>] shrink_zone+0x66/0x1a0
[<ffffffff8110217b>] do_try_to_free_pages+0xdb/0x550
[<ffffffff81102764>] try_to_free_mem_cgroup_pages+0xa4/0xb0
[<ffffffff8114851e>] mem_cgroup_reclaim+0x4e/0xd0
[<ffffffff81148e29>] __mem_cgroup_try_charge+0x409/0xbf0
[<ffffffff81149d85>] mem_cgroup_charge_common+0x45/0xa0
[<ffffffff8114c1d6>] mem_cgroup_newpage_charge+0x26/0x30
[<ffffffff811181a8>] handle_mm_fault+0x8c8/0xcf0
[<ffffffff810342a3>] __do_page_fault+0x1b3/0x600
[<ffffffff8103472c>] do_page_fault+0xc/0x20
[<ffffffff81609572>] page_fault+0x22/0x30

[<ffffffffffffffff>] 0xffffffffffffffff

William

OK, so the memcg reclaim gets down to nfs and it wants to evict its inode and that is waiting for the page to get unlocked. It would be interesting to find out who is keeping the page locked. What is the nfsd doing?

Michal Hocko
SUSE Labs

Trond points me the commit
9590544 NFS: avoid deadlocks with loop-back mounted NFS filesystems.

but it relies on sched commit as well, it seems hard to be backported in 3.14.x

William