Issue with "no space left on device"
Hello,
I have a 35 GB XFS partition with both plenty of free space and inodes, but I'm still getting the error "No space left on device" when trying to create new files.
Mount options:
# mount | grep vdc1 /dev/vdc1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
Used disk space:
# df /dev/vdc1 Filesystem 1K-blocks Used Available Use% Mounted on /dev/vdc1 36681200 16066576 20614624 44% /mnt
Used inodes:
# df -i /dev/vdc1 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vdc1 917440 199616 717824 22% /mnt
xfs info:
# xfs_info /dev/vdc1
meta-data=/dev/vdc1 isize=2048 agcount=4, agsize=2293695 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0
data = bsize=4096 blocks=9174779, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=4479, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0I've already tried repairing and rebooting, which unfortunately didn't help.
Any idea what's wrong?
Thanks!
- Sander
Quoted 17 lines · Sander van Schie / True
Hello,
I have a 35 GB XFS partition with both plenty of free space and inodes, but I'm still getting the error "No space left on device" when trying to create new files.
Mount options:
# mount | grep vdc1 /dev/vdc1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)Used disk space:
# df /dev/vdc1 Filesystem 1K-blocks Used Available Use% Mounted on /dev/vdc1 36681200 16066576 20614624 44% /mntUsed inodes:
Quoted 8 lines
# df -i /dev/vdc1 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vdc1 917440 199616 717824 22% /mntxfs info:
# xfs_info /dev/vdc1 meta-data=/dev/vdc1 isize=2048 agcount=4, agsize=2293695 blks
With 2k sized inodes, the most likely cause is free space fragmentation. What does 'xfs_db -c "freesp -s" <dev>' print for this fs?
Brian
Quoted 20 lines
= sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1 spinodes=0 data = bsize=4096 blocks=9174779, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=4479, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0I've already tried repairing and rebooting, which unfortunately didn't help.
Any idea what's wrong?
Thanks!
- Sander--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 24 lines · Brian Foster
Quoted 23 lines · Sander van Schie / True
Hello,
I have a 35 GB XFS partition with both plenty of free space and inodes, but I'm still getting the error "No space left on device" when trying to create new files.
Mount options:
# mount | grep vdc1 /dev/vdc1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)Used disk space:
# df /dev/vdc1 Filesystem 1K-blocks Used Available Use% Mounted on /dev/vdc1 36681200 16066576 20614624 44% /mntUsed inodes:
# df -i /dev/vdc1 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vdc1 917440 199616 717824 22% /mntxfs info:
Quoted 8 lines
Quoted 2 lines
# xfs_info /dev/vdc1 meta-data=/dev/vdc1 isize=2048 agcount=4, agsize=2293695 blksWith 2k sized inodes, the most likely cause is free space fragmentation. What does '
xfs_db-c "freesp -s" <dev>' print for this fs?Brian
The output of the command is as follows:
# xfs_db -c "freesp -s" /dev/vdc1
from to extents blocks pct
1 1 28 28 0,00
2 3 50 149 0,00
4 7 907 3799 0,07
8 15 1369 12990 0,25
16 31 183206 5126962 99,34
32 63 492 17197 0,33
total free extents 186052
total free blocks 5161125
average free extent size 27,7402(not sure what I'm looking at myself)
Quoted 20 lines
Quoted 20 lines
= sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1 spinodes=0 data = bsize=4096 blocks=9174779, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=4479, version=2= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0I've already tried repairing and rebooting, which unfortunately didn't help.
Any idea what's wrong?
Thanks!
- Sander--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Sander van Schie
On 17-08-17 18:47, Brian Foster wrote:
...
Quoted 20 lines
Quoted 5 lines
With 2k sized inodes, the most likely cause is free space fragmentation. What does '
xfs_db-c "freesp -s" <dev>' print for this fs?Brian
The output of the command is as follows:
# xfs_db -c "freesp -s" /dev/vdc1 from to extents blocks pct 1 1 28 28 0,00 2 3 50 149 0,00 4 7 907 3799 0,07 8 15 1369 12990 0,25 16 31 183206 5126962 99,34 32 63 492 17197 0,33 total free extents 186052 total free blocks 5161125 average free extent size 27,7402
Quoted 1 line
(not sure what I'm looking at myself)
It shows you where the freespace is in the filesystem, and how large the various freespace extents are; i.e. you have 28 single blocks free, and 50 free extents containing two to three free filesystem blocks, etc.
We allocate inodes in contiguous chunks of 64; with 2k inodes and 4k blocks, that means we allocate inodes in chunks of 32 blocks.
So you do have 492 extents with 32 free blocks or more, but allocation also has alignment requirements - these allocated chunks must also be aligned, in this case aligned to a 32-block boundary. From the output above, we don't know the alignment of that freespace
recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters, i.e.
xfs_db> freesp -A 32
will show only 32-block aligned free extents.
You may not have that recent of xfsprogs, but you could check out the git tree from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build it, and run xfs_db from within the tree, i.e.
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
Can you provide that output?
-Eric
Quoted 52 lines · Eric Sandeen
Quoted 2 lines · Sander van Schie
On 17-08-17 18:47, Brian Foster wrote:
...
Quoted 20 lines
Quoted 5 lines
With 2k sized inodes, the most likely cause is free space fragmentation. What does '
xfs_db-c "freesp -s" <dev>' print for this fs?Brian
The output of the command is as follows:
# xfs_db -c "freesp -s" /dev/vdc1 from to extents blocks pct 1 1 28 28 0,00 2 3 50 149 0,00 4 7 907 3799 0,07 8 15 1369 12990 0,25 16 31 183206 5126962 99,34 32 63 492 17197 0,33 total free extents 186052 total free blocks 5161125 average free extent size 27,7402Quoted 1 line
(not sure what I'm looking at myself)
It shows you where the freespace is in the filesystem, and how large the various freespace extents are; i.e. you have 28 single blocks free, and 50 free extents containing two to three free filesystem blocks, etc.
We allocate inodes in contiguous chunks of 64; with 2k inodes and 4k blocks, that means we allocate inodes in chunks of 32 blocks.
So you do have 492 extents with 32 free blocks or more, but allocation also has alignment requirements - these allocated chunks must also be aligned, in this case aligned to a 32-block boundary. From the output above, we don't know the alignment of that freespace
recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters, i.e.
xfs_db> freesp -A 32will show only 32-block aligned free extents.
You may not have that recent of xfsprogs, but you could check out the git tree from
git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build it, and runxfs_dbfrom within the tree, i.e.
Quoted 6 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1Can you provide that output?
-Eric
Thank you for the explanation!
The output of the command is as follows:
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
from to extents blocks pct
1 1 1 1 0,00
2 3 7 21 0,01
4 7 235 970 0,24
8 15 130 1313 0,33
16 31 14214 397375 99,42
total free extents 14587
total free blocks 399680
average free extent size 27,3997For what it's worth, practically all files (194024 out of 196385) on the filesystem are 80102 bytes in size.
- Sander
Quoted 2 lines · Sander van Schie
On 17-08-17 19:45, Eric Sandeen wrote:
...
Quoted 21 lines
Quoted 10 lines
recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters, i.e.
xfs_db> freesp -A 32
will show only 32-block aligned free extents.You may not have that recent of xfsprogs, but you could check out the git tree from
git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build it, and runxfs_dbfrom within the tree, i.e.Quoted 6 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1Can you provide that output?
-Eric
Thank you for the explanation!
The output of the command is as follows:
Quoted 7 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
from to extents blocks pct
1 1 1 1 0,00
2 3 7 21 0,01
4 7 235 970 0,24
8 15 130 1313 0,33
16 31 14214 397375 99,42So, I think that's the problem: There are no 32-block aligned free regions of 32 blocks length or greater.
(now that I think about it, the -A filter filters on free extents /starting/ on that alignment; I don't know if the inode allocator can make use of, say, a 64 block free extent which /overlaps/ an aligned 32-block range... hm)
Out of curiosity, what was the reason for 2k inodes?
-Eric
Quoted 38 lines · Eric Sandeen
Quoted 2 lines · Sander van Schie
On 17-08-17 19:45, Eric Sandeen wrote:
...
Quoted 21 lines
Quoted 10 lines
recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters, i.e.
xfs_db> freesp -A 32
will show only 32-block aligned free extents.You may not have that recent of xfsprogs, but you could check out the git tree from
git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build it, and runxfs_dbfrom within the tree, i.e.Quoted 6 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1Can you provide that output?
-Eric
Thank you for the explanation!
The output of the command is as follows:
Quoted 7 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1 from to extents blocks pct 1 1 1 1 0,00 2 3 7 21 0,01 4 7 235 970 0,24 8 15 130 1313 0,33 16 31 14214 397375 99,42So, I think that's the problem: There are no 32-block aligned free regions of 32 blocks length or greater.
Quoted 6 lines
(now that I think about it, the -A filter filters on free extents /starting/ on that alignment; I don't know if the inode allocator can make use of, say, a 64 block free extent which /overlaps/ an aligned 32-block range... hm)
Out of curiosity, what was the reason for 2k inodes?
Also for what it's worth - the sparse inodes feature, which is default on newer filesystems, alleviates this problem. When mounted, what does xfs_info /mount/point say, does it contain output for "spinodes?"
If you need 2k inodes, you probaboy want to get userspace+kernel that can support sparse inode allocation.
-Eric
Quoted 39 lines · Eric Sandeen
Quoted 38 lines · Eric Sandeen
Quoted 2 lines · Sander van Schie
On 17-08-17 19:45, Eric Sandeen wrote:
...
Quoted 21 lines
Quoted 10 lines
recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters, i.e.
xfs_db> freesp -A 32
will show only 32-block aligned free extents.You may not have that recent of xfsprogs, but you could check out the git tree from
git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build it, and runxfs_dbfrom within the tree, i.e.Quoted 6 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1Can you provide that output?
-Eric
Thank you for the explanation!
The output of the command is as follows:
Quoted 7 lines
# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1 from to extents blocks pct 1 1 1 1 0,00 2 3 7 21 0,01 4 7 235 970 0,24 8 15 130 1313 0,33 16 31 14214 397375 99,42So, I think that's the problem: There are no 32-block aligned free regions of 32 blocks length or greater.
Quoted 6 lines
Quoted 6 lines
(now that I think about it, the -A filter filters on free extents /starting/ on that alignment; I don't know if the inode allocator can make use of, say, a 64 block free extent which /overlaps/ an aligned 32-block range... hm)
Out of curiosity, what was the reason for 2k inodes?
The filesystem was created by a default Ceph deployement, so no particular reason. An inode size of 2k seems to be the default of Ceph(-deploy) to prevent performance issues due to, as I understand it, metadata otherwise possibly not fitting in a single extent.
Is this currently primarily an issue due to the fairly small partition size and big inode size? Will this be less of an issue with a partition of let's say 1 TB?
I'll do some more research as it's not very clear for me yet (due to my lack of experience regarding XFS, or filesystems in general). Your information was very insightful though, so thank you for that!
Quoted 5 lines
Also for what it's worth - the sparse inodes feature, which is default on newer filesystems, alleviates this problem. When mounted, what does
xfs_info/mount/point say, does it contain output for "spinodes?"
Currently it's set to 0: spinodes=0
Quoted 3 lines
If you need 2k inodes, you probaboy want to get userspace+kernel that can support sparse inode allocati
I will look into this
Quoted 3 lines
-Eric