"No space left on device" from hell
Hello.
3.10.40 kernel and creating files causes "No space left on device". After deleting some files I can create new files again until the problem happens again.
Any ideas what is going on and, what is more important, how to fix the problem?
# df -h /dev/mapper/vgsys-home 600G 368G 232G 62% /home
# df -i /dev/mapper/vgsys-home 629145600 7982895 621162705 2% /home
# xfs_info /home
meta-data=/dev/mapper/vgsys-home isize=256 agcount=8, agsize=19660800 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=157286400, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=38400, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=imaxpct=25% but current inodes take only about 2GB, much less that 25% of 600G, so that shouldn't be problem
/dev/mapper/vgsys-home /home xfs rw,nosuid,nodev,relatime,attr2,inode64,usrquota,prjquota 0 0
so inode64 is not a problem
# xfs_quota -c "quota -p 0" /home
Disk quotas for Project #0 (0)
Filesystem Blocks Quota Limit Warn/Time Mounted
on
/dev/mapper/vgsys-home
27848804 0 0 00 [--------] /home
# xfs_quota -c "quota -u 0" /home
Disk quotas for User root (0)
Filesystem Blocks Quota Limit Warn/Time Mounted
on
/dev/mapper/vgsys-home
103484 0 0 00 [--------] /home
# xfs_quota -c "quota -i 0" /home
Disk quotas for User root (0)
Filesystem Files Quota Limit Warn/Time Mounted
on
/dev/mapper/vgsys-home
8336 0 0 00 [--------] /homeso quota is not a problem, too.
# xfs_db -r "-c freesp -s" /dev/mapper/vgsys-home
from to extents blocks pct
1 1 3426 3426 0.01
2 3 3319 8026 0.01
4 7 3694 19027 0.03
8 15 23225 330378 0.54
16 31 396 8180 0.01
32 63 75 3256 0.01
64 127 44 3443 0.01
128 255 26 4892 0.01
256 511 32 10737 0.02
512 1023 5 3867 0.01
1024 2047 8 13576 0.02
2048 4095 5 13893 0.02
4096 8191 3 16896 0.03
8192 16383 1 15532 0.03
16384 32767 1 17731 0.03
1048576 2097151 1 1414952 2.33
16777216 19660800 3 58913965 96.90
total free extents 34264
total free blocks 60801777
average free extent size 1774.51unfortunately no idea how to interpret this. Man page isn't helpful.
http://sprunge.us/IVjE
(trace-cmd of touch /home/x ... but looks to be partial only)
Signature · 6 lines
Arkadiusz Miśkiewicz, arekm / maven.pl _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
2014-09-24 9:03 GMT+02:00 Arkadiusz Miśkiewicz <[email protected]>:
Quoted 6 lines
Hello.
3.10.40 kernel and creating files causes "No space left on device". After deleting some files I can create new files again until the problem happens again.
Are you sure the problem is the filesystem device? Maybe you are running a process that fills up another device i.e. shm.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Fran Tsao Santín
2014-09-24 9:03 GMT+02:00 Arkadiusz Miśkiewicz <[email protected]>:
Quoted 5 lines
Hello.
3.10.40 kernel and creating files causes "No space left on device". After deleting some files I can create new files again until the problem happens again.
Are you sure the problem is the filesystem device? Maybe you are running a process that fills up another device i.e. shm.
Dave Chinner figured out the problem already (on irc).
Basically this fs was grown from 300GB to 600GB but due to bug (fixed by commit 9de67c3 ("xfs: allow inode allocations in post-growfs disk space")) that new space was never used for new inodes.
So ended up with full AGs 0-3 (xfs_db -r -c "freesp -s -a XX" /dev/xyz shows that for each AG XX).
Fs was never unmounted after grows but fortunately mount -o remount,inode32 ... followed by mount -o remount,inode64 cured it without a need to reboot, so now new inodes are distributed to all AGs.
I still need to free some space in AGs 0-3 though (by moving data to new dirs allocated in new AGs; xfs_db -r -c "convert inode DIR_INODE_NR agno" /dev/xyz shows in which AG new dir got allocated).
There are more bugs hiding (like if ENOSPC on some AG happens it should try next AG instead of failing). Dave knows all that magic.
Big thanks to Dave!
Signature · 6 lines
Arkadiusz Miśkiewicz, arekm / maven.pl _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs