2.6.37: mount with quota eats all available memory
I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now I'm no longer able to mount this filesystem with usrquota and grpquota.
xfs eats all ram, from slabtop:
3756380 3756380 100% 1.00K 244655 16 3914480K xfs_inode 251712 251711 99% 0.06K 3933 64 15732K kmalloc-64 118482 114287 96% 0.55K 4233 28 67728K radix_tree_node 88768 88711 99% 0.12K 2774 32 11096K kmalloc-128 28713 28713 100% 0.08K 563 51 2252K sysfs_dir_cache 22974 22973 99% 0.09K 547 42 2188K kmalloc-96 9776 9776 100% 0.25K 611 16 2444K kmalloc-256 7791 7791 100% 0.19K 371 21 1484K kmalloc-192 7168 7097 99% 0.01K 14 512 56K kmalloc-8
machine is not responsible anymore, xfs ate all ram
If I don't use usr/grp quota then filesystem mounts without a problem.
What are my options now? I need this system running again with quota but loosing all quota information is not a problem (I can set it again).
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · Arkadiusz Miskiewicz
I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
Recently after using
xfs_fsr(and getting a oops) had to doxfs_repairand now I'm no longer able to mount this filesystem with usrquota and grpquota.
Some data about fs:
# LC_ALL=C xfs_info /home
meta-data=/dev/md3 isize=256 agcount=32, agsize=6564128 blks
= sectsz=4096 attr=0
data = bsize=4096 blocks=210051776, imaxpct=25
= sunit=16 swidth=48 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=4096 sunit=1 blks, lazy-count=0
realtime =none extsz=196608 blocks=0, rtextents=0
# cat /proc/mdstat
Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 2/3 pages [8KB], 65536KB chunkSignature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 12 lines · Arkadiusz Miskiewicz
Quoted 6 lines · Arkadiusz Miskiewicz
I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
Recently after using
xfs_fsr(and getting a oops) had to doxfs_repairand now I'm no longer able to mount this filesystem with usrquota and grpquota.Some data about fs:
# LC_ALL=Cxfs_info/home
meta-data=/dev/md3 isize=256 agcount=32, agsize=6564128
blks = sectsz=4096 attr=0
Quoted 11 lines
data = bsize=4096 blocks=210051776, imaxpct=25
= sunit=16 swidth=48 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=4096 sunit=1 blks, lazy-count=0
realtime =none extsz=196608 blocks=0, rtextents=0
# cat /proc/mdstat
Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 2/3 pages [8KB], 65536KB chunkMore info, tested current linus 2.6.38git including "xfs: Reduce OOM kill problems under heavy load V2" patchset but the result wasn't much better:
3549921 3549921 100% 1.06K 132791 30 4249312K xfs_inode
(from slabtop), machine became unresponsible and rebooted.
I wonder if 8GB of ram would be enough for this 800GB fs to mount.
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 17 lines · Arkadiusz Miskiewicz
I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.
Recently after using
xfs_fsr(and getting a oops) had to doxfs_repairand now I'm no longer able to mount this filesystem with usrquota and grpquota.xfs eats all ram, from slabtop:
3756380 3756380 100% 1.00K 244655 16 3914480K xfs_inode 251712 251711 99% 0.06K 3933 64 15732K kmalloc-64 118482 114287 96% 0.55K 4233 28 67728K radix_tree_node 88768 88711 99% 0.12K 2774 32 11096K kmalloc-128 28713 28713 100% 0.08K 563 51 2252K sysfs_dir_cache 22974 22973 99% 0.09K 547 42 2188K kmalloc-96 9776 9776 100% 0.25K 611 16 2444K kmalloc-256 7791 7791 100% 0.19K 371 21 1484K kmalloc-192 7168 7097 99% 0.01K 14 512 56K kmalloc-8
Quoted 6 lines
machine is not responsible anymore, xfs ate all ram
If I don't use usr/grp quota then filesystem mounts without a problem.
What are my options now? I need this system running again with quota but loosing all quota information is not a problem (I can set it again).
I can reproduce it. There's some difference between the way userspace operates with bulkstat vs the way the kernel quotacheck operates with it.
Essentially the problem appears to be related to the inodes not being reclaimed and the kernel declaring OOM with about 50 pages in the page cache but still having millions of reclaimable inodes. Whether it is a result of the kernel not attempting to reclaim them or something else, I don't know yet.
However, there's definitely something fishy going on because quotacheck inodes aren't showing up as VFS inodes even though they probably should because they are fully instantiated. I'll look at it further tomorrow.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs