kmem_alloc hang in xlog_cil_insert_format_items

Hi,All.

Recently, I test xfs to verify kmem_alloc hang in xfs_dir2_block_to_sf in kernel-3.14, for patch b3f03bac8132207a20286d5602eda64500c19724.

My test case:
I use three ssd disks to build three xfs filesystems.

My mount options:
rw,noexec,nodev,noatime,nodiratime,barrier=0,discard,inode64,logbsize=256k,delaylog

Each xfs filesystem run a postmark and two fio.

postmark parameter:
set size 100000 100000000
set location /var/xfs-0/
set seed 900
set number 100000
set subdirectories 5000
set read 40960 65536000
set write 40960 65536000
set transactions 10000
set bias create 40
set bias read 60
run xfs-0-result.txt
show

fio parameter:
fio -directory /var/${1} -rw=randrw -ioengine=libaio -iodepth=128 -size=128M -name=stress_test -numjobs=256

After running for five hours , the output of warning all the time:

2015-07-07T23:32:18.230120+00:00 server-69 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250 size:32832)
2015-07-07T23:32:18.230131+00:00 server-69 kernel: CPU: 1 PID: 15954 Comm: postmark Tainted: G O 3.12.21-1.el6.x86_64 #1
2015-07-07T23:32:18.230134+00:00 server-69 kernel: Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.2.10 03/09/2015
2015-07-07T23:32:18.230147+00:00 server-69 kernel: 0000000000000000 ffff880481c99bb8 ffffffff8162562a 0000000000008040
2015-07-07T23:32:18.230148+00:00 server-69 kernel: 000000000000044c ffff880481c99bf8 ffffffffa06c457c ffff880481c99bf8
2015-07-07T23:32:18.230149+00:00 server-69 kernel: ffff88086936bcf0 ffff8804a17ecd00 ffff880066a48000 ffff880044158000
2015-07-07T23:32:18.230150+00:00 server-69 kernel: Call Trace:
2015-07-07T23:32:18.230152+00:00 server-69 kernel: [<ffffffff8162562a>] dump_stack+0x49/0x5f
2015-07-07T23:32:18.230153+00:00 server-69 kernel: [<ffffffffa06c457c>] kmem_alloc+0xec/0x100 [xfs]
2015-07-07T23:32:18.230186+00:00 server-69 kernel: [<ffffffffa0709b6f>] xlog_cil_insert_format_items+0x11f/0x1f0 [xfs]
2015-07-07T23:32:18.230205+00:00 server-69 kernel: [<ffffffffa06d0d00>] ? xfs_bmap_last_offset+0x30/0xc0 [xfs]
2015-07-07T23:32:18.230211+00:00 server-69 kernel: [<ffffffffa0709d8d>] xlog_cil_insert_items+0x3d/0x1b0 [xfs]
2015-07-07T23:32:18.230212+00:00 server-69 kernel: [<ffffffffa070a484>] xfs_log_commit_cil+0x54/0x150 [xfs]
2015-07-07T23:32:18.230228+00:00 server-69 kernel: [<ffffffffa06c3cb9>] xfs_trans_commit+0x79/0x270 [xfs]
2015-07-07T23:32:18.230230+00:00 server-69 kernel: [<ffffffffa06f78c2>] xfs_remove+0x2d2/0x350 [xfs]
2015-07-07T23:32:18.230235+00:00 server-69 kernel: [<ffffffff811c882f>] ? d_walk+0x5f/0x260
2015-07-07T23:32:18.230236+00:00 server-69 kernel: [<ffffffffa06b9782>] xfs_vn_unlink+0x52/0xa0 [xfs]
2015-07-07T23:32:18.230237+00:00 server-69 kernel: [<ffffffff811be22b>] vfs_rmdir+0xbb/0x110
2015-07-07T23:32:18.230238+00:00 server-69 kernel: [<ffffffff811c14a3>] do_rmdir+0x203/0x220
2015-07-07T23:32:18.230239+00:00 server-69 kernel: [<ffffffff811c1516>] SyS_rmdir+0x16/0x20
2015-07-07T23:32:18.230240+00:00 server-69 kernel: [<ffffffff81632729>] system_call_fastpath+0x16/0x1b

In function kmem_alloc, I modify code to get more info:
57c57
< if (!(++retries % 100))
---

59,60c59,62
< "possible memory allocation deadlock in %s (mode:0x%x)",
< __func__, lflags);
---

size:%zu)",

The patch b3f03bac8132207a20286d5602eda64500c19724 solves the problem of the big directroy size, more than 64k.
Now, the 'struct txfs_log_vec' need 32k, but fail when the physical memory fragmentation is very serious; In the later, kmalloc may fail when process 16k.

So, I think can we provide a unified solution? And I think why can't we use vmalloc when kmalloc fails?

Thanks
----------------
juncheng bai

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

You do not need to spam the list if you do not get an immediate response to your initial question.

As it is, you commented on the bug report a couple of days again where I explained exactly what the state of this problem was (i.e. you asked "why can't we just use vmalloc?")

And that's pretty clear what the trigger here is - the allocation is just over the PAGE_ALLOC_COSTLY_ORDER threshold. We see this retry come up at this point because that threshold changes the behaviour of memory reclaim and makes allocations more likely to fail...

Right now we have to choose between:
- running small machines out of vmalloc space
- triggering the OOM killer and killing random processes
- failing the allocation whcih corrupts the filesystem in memory, so we need to shut down the filesystem
- retrying until we succeed.

We currently default to the least worst situation which is "retry until we succeed" because all the other situations are much worse (i.e. take the filesystem and/or machine down, rather than recover when memory becomes available again).

As I said earlier, there is no quick fix for this problem and shouting at us about it isn't going to get it solved any sooner.

I told you - there is limited vmalloc space on 32 bit systems, so vmalloc can also fail due to vmalloc space being unavaiable, even when there is plenty of memory available. i.e. on 32 bit systems vmalloc is a lot more unreliable than kmalloc(). IOws, while vmalloc might fix your specific problem on your machine, it is not a general solution to the problem of kmalloc failing large allocations when you run your system to OOM conditions.

And, FWIW, you can reduce the impact of the problem simply by making your directory block size smaller.....

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs