xfs wrongly report ENOSPACE
Hi DJ and xfs developers,
i am ceph developer, build the OSD on top of the xfs mount point. the kernel xfs module i used was build from the 4.9.8 kernel release from kernel.org, that includes your recent changes in Dec, 2016.
recently, i found our cluster will get faulted due to the xfs mount wrongly report there is no space to create the new file as
" No space left on device ", but apparently the xfs mount point only
be around 50% full.
i used the systemtap to narrow it down the the function xfs_ialloc_ag_select which reports no free ag selected, the problem seems still exist, not sure your recent patches will fix it or not.
i am very glad to provide more info if your like to assist you to identify the problem.
Please use the XFS list, updated cc.
Quoted 11 lines · peng.hse
Hi DJ and xfs developers,
i am ceph developer, build the OSD on top of the xfs mount point. the kernel xfs module i used was build from the 4.9.8 kernel release from kernel.org, that includes your recent changes in Dec, 2016.
recently, i found our cluster will get faulted due to the xfs mount wrongly report there is no space to create the new file as
" No space left on device ", but apparently the xfs mount point only be
around 50% full.
Quoted 6 lines
i used the systemtap to narrow it down the the function
xfs_ialloc_ag_selectwhich reports no free ag selected, the problem seems
still exist, not sure your recent patches will fix it or not.i am very glad to provide more info if your like to assist you to identify the problem.
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
In this case, please at least provide the output from 'df -i <mnt>,' 'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
Chances are that this is caused by free space fragmentation. This means that while there might be plenty of free space, there isn't a contiguous free extent large enough to allocate an inode chunk. We've seen this kind of thing before with Ceph and it can be exacerbated by the use of large inodes. Potential workarounds may be to use the default inode size and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
Another possible cause is that you've simply hit the maximum inode allocation limit (see the xfs_growfs manpage and the '-m' option)...
Brian
Quoted 32 lines · Brian Foster
Please use the XFS list, updated cc.
Quoted 11 lines · peng.hse
Hi DJ and xfs developers,
i am ceph developer, build the OSD on top of the xfs mount point. the kernel xfs module i used was build from the 4.9.8 kernel release from kernel.org, that includes your recent changes in Dec, 2016.
recently, i found our cluster will get faulted due to the xfs mount wrongly report there is no space to create the new file as
" No space left on device ", but apparently the xfs mount point only be
around 50% full.Quoted 6 lines
i used the systemtap to narrow it down the the function
xfs_ialloc_ag_selectwhich reports no free ag selected, the problem seems
still exist, not sure your recent patches will fix it or not.i am very glad to provide more info if your like to assist you to identify the problem.
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
In this case, please at least provide the output from 'df -i <mnt>,' '
xfs_info<mnt>' and 'xfs_db-c "freesp -s" <dev>.'Chances are that this is caused by free space fragmentation. This means that while there might be plenty of free space, there isn't a contiguous free extent large enough to allocate an inode chunk. We've seen this kind of thing before with Ceph and it can be exacerbated by the use of large inodes. Potential workarounds may be to use the default inode size and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
Quoted 5 lines
Another possible cause is that you've simply hit the maximum inode allocation limit (see the
xfs_growfsmanpage and the '-m' option)...Brian
the problematic mnt point is /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
[root@xt1 ~]# touch /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No space left on device
[root@xt1 ~]# df -i /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
Filesystem Inodes IUsed IFree IUse% Mounted on/dev/sdc2 982848 47296 935552 5% /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
[root@xt1 ~]# xfs_info /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
meta-data=/dev/sdc2 isize=2048 agcount=4, agsize=491455 blks
= sectsz=512 attr=2,
projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1965819, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0
blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@xt1 ~]# xfs_db -r '-c freesp -s' /dev/sdc2
from to extents blocks pct
1 1 80 80 0.01
2 3 61 159 0.02
4 7 88 483 0.06
8 15 146 1680 0.21
16 31 34312 788594 99.70total free extents 34687
total free blocks 790996
average free extent size 22.8038
the total number of files underneath the mnt is [root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l 47294
Quoted 29 lines
Quoted 10 lines
In this case, please at least provide the output from 'df -i <mnt>,' '
xfs_info<mnt>' and 'xfs_db-c "freesp -s" <dev>.'Chances are that this is caused by free space fragmentation. This means that while there might be plenty of free space, there isn't a contiguous free extent large enough to allocate an inode chunk. We've seen this kind of thing before with Ceph and it can be exacerbated by the use of large inodes. Potential workarounds may be to use the default inode size and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
Quoted 5 lines
Another possible cause is that you've simply hit the maximum inode allocation limit (see the
xfs_growfsmanpage and the '-m' option)...Brian
the problematic mnt point is
/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/[root@xt1 ~]# touch
/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No space left on device[root@xt1 ~]# df -i
/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
Filesystem Inodes IUsed IFree IUse% Mounted on/dev/sdc2982848 47296 935552 5%/mnt/819f7954-bf88-430d-a637-a08bad56cf6e
Quoted 3 lines
[root@xt1 ~]#
xfs_info/mnt/819f7954-bf88-430d-a637-a08bad56cf6e
meta-data=/dev/sdc2 isize=2048 agcount=4, agsize=491455 blks
= sectsz=512 attr=2,
2048 bytes inode, I'd bet your filesystem is fragmented enough to not have any more room to create inodes into it.
Quoted 14 lines
projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1965819,
imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2560,
version=2
= sectsz=512 sunit=0 blks,
lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Quoted 7 lines
[root@xt1 ~]#
xfs_db-r '-c freesp -s'/dev/sdc2
from to extents blocks pct1 1 80 80 0.01 2 3 61 159 0.02 4 7 88 483 0.06 8 15 146 1680 0.21 16 31 34312 788594 99.70
for creating new inodes you will need a contiguous space of 131072 bytes, which you don't have anymore, so you won't be able to create any more files into this filesystem, unless you free up some space to allow room for the creation of new inodes.
XFS create inodes in chunks of 64 inodes, if you don't have enough space for that, then you get a -ENOSPC as you are getting.
Cheers.
Quoted 14 lines
total free extents 34687
total free blocks 790996
average free extent size 22.8038the total number of files underneath the mnt is [root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l 47294
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Carlos