XFS "no space left" problem

Hi dear list,

I recently migrated a 1.3Tbyte partition from reiserfs to XFS. As the 1.3Tb partition was full, my idea was to shrink reiserfs partition of 100Gig at a time, and then grow the XFS one.

This has been working since 1.1To for XFS partition. Now I'm in the following situation:

/dev/mapper/crypt-data

                      1.2T  1.1T  134G  89% /jail2

dunno jail2 # touch a
dunno jail2 # touch b
touch: cannot touch `b': No space left on device
dunno jail2 #

will be running
again into the "no space left".

first:
/dev/mapper/crypt-data

                     561850144  158335 561691809    1% /jail2

is outputing from df -i.

umount/remount of the partition doesn't change anything.

here are some information about the filesystem details:

dunno jail2 # xfs_info /jail2
meta-data=/dev/crypt/data isize=256 agcount=188,
agsize=1638400 blks

         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=306708480,
imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal               bsize=4096   blocks=12800, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

dunno / # xfs_db /dev/crypt/data
xfs_db> frag
actual 140099, ideal 134926, fragmentation factor 3.69% xfs_db> freesp
from to extents blocks pct

      1       1     801     801   0.00
      2       3       2       5   0.00
     32      63       2      89   0.00
    128     255       3     517   0.00
    256     511       4    1676   0.00
    512    1023       5    3776   0.01
   1024    2047       7   10765   0.03

262144 524287 23 9200216 26.21
1048576 1638400 16 25888649 73.74
xfs_db>

I can't think of a Fix solution from here and I'm stuck.

As this box is in production and remotely managed; Go to the place, take the hard drives and backup/restore is not a solution.

Any help/comment/fix/idea would be very appreciated. Any additionnal information you need could be asked :)

TIA,
Best regards,

wildcat

So you've filled up the lower terabyte of the partition before you added the space above 1TB. The problem
is almost certainly going to be that there's no room left to allocate inodes below the 1TB mark (32 bit inode filesystem) because all that space is taken up by data....

Mostly large blocks. I bet they are all in AGs above the 1TB mark. One thing you can do is look at the freespace per AG:

# for ag in `seq 0 1 188`; do

And see how many of the AGs below 1TB (~6.4GB per AG, so those below about about AG 160) have no free space. To allocate new inodes you need a contiguous extent of 16k plus another 4 free blocks, so you need at _least_ 8 free blocks in an AG to allocate a new inode chunk on disk.

If this is the case, then you need to delete some files to get space free below 1TB for more inodes and then copy a large file or two so they are moved above the 1TB mark, then remove the original copy to make space for inodes below 1TB.

Or get a 64bit machine and use inode64. ;)

HTH.

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group

David, can you tell more in detail what you mean with "use inode64"?

We also just migrated our server to xfs, the largest partition has a size of 4TB. Lets say in the future a problem occurs and we could solve this by a 64-bit system? Migrating to 64-bit wouldn't be difficult, since the server systems are already opterons. Only since we never bothered to migrate the installation to x86_64 its not 64-bit yet. Would we need to tell xfs somehow that it update its internal values to 64-bit integers or will it do that automatically?
Btw, if you need 64-bit integers, why don't you use long long, or even better with C99 int64_t?

Thanks,
Bernd

xfs is generally fully 64 bit, even on a 32 bit machine. see more below...

32-bit machines can only do 32-bit inodes (vfs limitation) (at least 'til recently), so by default xfs doesn't let inode numbers go above 32 bits - even though -internally- it -does- use a 64-bit container.

The xfs inode number encodes its location on disk, so this restriction also restricts which blocks can be allocated as inodes.

xfs tries to reserve enough of these low blocks that you won't run out of potential inode blocks, but i've seen that get sideways. The original poster probably ran into trouble because he started with a 100G filesystem & grew it 10 or so times.

if you mount with inode64 you'll allow inodes to be allocated with more
than 32 bits set. But this can only work on a 64 bit machine.

and then you have to be sure your userspace can cope w/ 64 bit inode numbers too.

-Eric

Ah, I see, linux vfs limitation.

Well, we just also switched to a new raid device and before we bought it, we also made sure its possible to do an online expansion. So a migration from 6 TB to 12 TB will come... I guess in two or three years.

Ah, its a mount option! And even undocumented in "man 8 mount". Thanks will add this to our wiki and write a debian bugreport.

Hmm, sizeof(ino_t) is 4 on x86_32, that will be a problem. We do have quite a lot of programs, not written very well and not compilable with recent compilers. On the other side, the linux nfs client converts inode numbers to 32 bit, if it would cause an overflow. However, it doesn't do that on x86_64 for 32bit programs. I know that quite well, due to another problem.

Thanks a lot for your insights,
Bernd

Bernd Schubert
PCI / Theoretische Chemie
Universität Heidelberg
INF 229
69120 Heidelberg