Skip to contentSkip to conversation

2.6.37: mount with quota eats all available memory

DISCUSSION Arkadiusz Miskiewicz 2 participants 4 messages Mar 11, 2011 – Mar 13, 2011

lore
Share & tools
lore
#0 lore raw

I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.

Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now I'm no longer able to mount this filesystem with usrquota and grpquota.

xfs eats all ram, from slabtop:

3756380 3756380 100%    1.00K 244655       16   3914480K xfs_inode
251712 251711  99%    0.06K   3933       64     15732K kmalloc-64
118482 114287  96%    0.55K   4233       28     67728K radix_tree_node
 88768  88711  99%    0.12K   2774       32     11096K kmalloc-128
 28713  28713 100%    0.08K    563       51      2252K sysfs_dir_cache
 22974  22973  99%    0.09K    547       42      2188K kmalloc-96
  9776   9776 100%    0.25K    611       16      2444K kmalloc-256
  7791   7791 100%    0.19K    371       21      1484K kmalloc-192
  7168   7097  99%    0.01K     14      512        56K kmalloc-8

machine is not responsible anymore, xfs ate all ram

If I don't use usr/grp quota then filesystem mounts without a problem.

What are my options now? I need this system running again with quota but loosing all quota information is not a problem (I can set it again).

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 5 lines · Arkadiusz Miskiewicz

I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.

Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now I'm no longer able to mount this filesystem with usrquota and grpquota.

Some data about fs:
# LC_ALL=C xfs_info /home

meta-data=/dev/md3               isize=256    agcount=32, agsize=6564128 blks
         =                       sectsz=4096  attr=0
data     =                       bsize=4096   blocks=210051776, imaxpct=25
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=0
realtime =none                   extsz=196608 blocks=0, rtextents=0
# cat /proc/mdstat
Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
      840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/3 pages [8KB], 65536KB chunk
Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw
Quoted 12 lines · Arkadiusz Miskiewicz
Quoted 6 lines · Arkadiusz Miskiewicz

I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.

Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now I'm no longer able to mount this filesystem with usrquota and grpquota.

Some data about fs:
# LC_ALL=C xfs_info /home
meta-data=/dev/md3 isize=256 agcount=32, agsize=6564128
blks = sectsz=4096 attr=0

Quoted 11 lines
data     =                       bsize=4096   blocks=210051776, imaxpct=25
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=0
realtime =none                   extsz=196608 blocks=0, rtextents=0
# cat /proc/mdstat
Personalities : [raid10] [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sda4[0] sdd4[3] sdc4[2] sdb4[1]
      840207168 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/3 pages [8KB], 65536KB chunk

More info, tested current linus 2.6.38git including "xfs: Reduce OOM kill problems under heavy load V2" patchset but the result wasn't much better:

3549921 3549921 100% 1.06K 132791 30 4249312K xfs_inode
(from slabtop), machine became unresponsible and rebooted.

I wonder if 8GB of ram would be enough for this 800GB fs to mount.

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
In reply to Arkadiusz Miskiewicz
Quoted 17 lines · Arkadiusz Miskiewicz

I have a machine with 4GB, running 64bit 2.6.37, xfs on top of soft raid5.

Recently after using xfs_fsr (and getting a oops) had to do xfs_repair and now I'm no longer able to mount this filesystem with usrquota and grpquota.

xfs eats all ram, from slabtop:

3756380 3756380 100%    1.00K 244655       16   3914480K xfs_inode
251712 251711  99%    0.06K   3933       64     15732K kmalloc-64
118482 114287  96%    0.55K   4233       28     67728K radix_tree_node
 88768  88711  99%    0.12K   2774       32     11096K kmalloc-128
 28713  28713 100%    0.08K    563       51      2252K sysfs_dir_cache
 22974  22973  99%    0.09K    547       42      2188K kmalloc-96
  9776   9776 100%    0.25K    611       16      2444K kmalloc-256
  7791   7791 100%    0.19K    371       21      1484K kmalloc-192
  7168   7097  99%    0.01K     14      512        56K kmalloc-8
Quoted 6 lines

machine is not responsible anymore, xfs ate all ram

If I don't use usr/grp quota then filesystem mounts without a problem.

What are my options now? I need this system running again with quota but loosing all quota information is not a problem (I can set it again).

I can reproduce it. There's some difference between the way userspace operates with bulkstat vs the way the kernel quotacheck operates with it.

Essentially the problem appears to be related to the inodes not being reclaimed and the kernel declaring OOM with about 50 pages in the page cache but still having millions of reclaimable inodes. Whether it is a result of the kernel not attempting to reclaim them or something else, I don't know yet.

However, there's definitely something fishy going on because quotacheck inodes aren't showing up as VFS inodes even though they probably should because they are fully instantiated. I'll look at it further tomorrow.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs