Failed xfs_growfs after lvextend

I extended an LV to 12.28TB and ran xfs_growfs on the mount point, lv0.
This appeared to work fine except that after this, the filesystem didn't appear any larger.

rajones@mapdata<~>$ df -h /lv0
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-lv0   4.1T  2.9T  1.3T  70% /lv0

This was the size of the filesystem before the xfs_growfs was run and after.

Is it possible the LV or VG could be corrupted in some way? Or is the problem likely with the xfs filesystem? I've run xfs_repair on /dev/mapper/vg0-lv0 and it complete successfully. I can mount the filesystem with no errors and see the existing data but the size still shows up wrong.

The System in question:

The PVs in the VG are LUNs on an external FC attached RAID unit.
The system OS is FC3 running kernel 2.6.12-1.1381_FC3smp, xfsprogs-2.6.13-2, lvm2-2.00.25-1.01.

What happened:

root@mapdata<~>$ lvextend -l +33528 /dev/vg0/lv0 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1

After the extend I have the following LV.

root@mapdata<~>$ lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg0/lv0
  VG Name                vg0
  LV UUID                qbk6r9-RxpP-Bx9Y-joJV-4ptl-PSNE-PuXX1X
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                12.28 TB
  Current LE             50292
  Segments               11
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

root@mapdata<~>$ xfs_growfs /lv0

Results in no change in the mounted filesystem size.

root@mapdata<~>$ xfs_repair -o assume_xfs /dev/mapper/vg0-lv0
Phase 1 - find and verify superblock...
bad primary superblock - inconsistent filesystem geometry information !!!

attempting to find secondary superblock...

[cut rest of xfs_repair output]

The xfs_repair finds secondary superblock and completes successfully but when I mount I get the same result as before:

rajones@mapdata<~>$ df -h /lv0
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-lv0   4.1T  2.9T  1.3T  70% /lv0

Any suggestions?

Thank you,
Randall

..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
[email protected]      Code 610.3      301-286-2239

I looked into this problem awhile back now. I believe what you're seeing here is an inconsistency in the kernels block layer - at least, when I last looked at this, this was the problem - the size increase was indeed done inside the driver, and /sys/block/xxx/size confirmed that, but the interfaces which use /dev/xxx to get the device size do not see the increase (i.e. lseek(SEEK_END) or a BLKGETSIZE64 ioctl).

I wrote the attached program to show the issue, and sent mail to LKML to let folks know of the issue, I guess noone has got around to trying to address the problem yet though. The core of the problem was that the /dev/xxx inode size (i_size) had not been updated to reflect the change in device size, IIRC.

Hmm, actually, I just went and read through fs/block_dev.c again, and perhaps this is a device mapper bug after all - there is an interface for increasing the inode size (bd_set_size) but it doesn't seem to be called from anywhere in drivers/md/dm*...? Theres one call to the i_size_write interface on the bdev inode, but I can't tell whether it will be called on the resize - perhaps a device mapper guru can tell us? It looks like MD might do the resize correctly though, its got a call in what looks like a device-size-increasing code path.

Give this program a try though, just to see if you are seeing the same problem that I was.

cheers.

Nathan

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/mount.h>

#ifndef BLKGETSIZE64

# define BLKGETSIZE64   _IOR(0x12,114,size_t)
#endif

int main(int argc, char **argv)

{
	__uint64_t	size;
	long long	sz = -1;
	int		error, fd;

	if (argc != 2) {
		fputs("insufficient arguments\n", stderr);
		return 1;
	}
	fd = open(argv[1], O_RDONLY);
	if (!fd) {
		perror(argv[1]);
		return 1;
	}

	error = ioctl(fd, BLKGETSIZE64, &size);
	if (error >= 0) {
		/* BLKGETSIZE64 returns size in bytes not 512-byte blocks */
		sz = (long long)(size >> 9);
		printf("%lld 512 byte blocks (BLKGETSIZE64)\n", sz);
	} else {
		/* If BLKGETSIZE64 fails, try BLKGETSIZE */
		unsigned long tmpsize;

		error = ioctl(fd, BLKGETSIZE, &tmpsize);
		if (error < 0) {
			fprintf(stderr, "can't determine device size");
			return 1;
		}
		sz = (long long)tmpsize;
		printf("%lld 512 byte blocks (BLKGETSIZE)\n", sz);
	}
	return 0;
}

Ok. I looked at the things you suggested. It seems that everything is in order with the device mapper and it sees the 12.28TB device size. Your getdevicesize.c code verified this with a BLKGETSIZE64 ioctl call.

root@mapdata<~>$ cat /sys/block/dm-0/size
26367492096

root@mapdata<~>$ ~rajones/getdevicesize /dev/mapper/vg0-lv0
26367492096 512 byte blocks (BLKGETSIZE64)

26367492096 512 byte blocks is approx. 12.28TB.

So, back to xfs. Is it possible xfs_growfs is not working properly? The result after running xfs_growfs was that the primary superblock on the LV filesystem was corrupt or missing.

Is there a workaround? A 12.28TB LV with xfs filesystem should work, yes?

One idea for a workaround is to relocate the data on the misbehaving LV/filesystem and recreate the LV and filesystem from scratch to avoid using xfs_growfs. I have enough free PVs to create a temp space to move my existing data.

Are there any possibilities for fixing the problem "in place".

Thank you,
Randall

..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
[email protected]      Code 610.3      301-286-2239

Ah, excellent, sounds like that issue I had seen has been resolved.

Anythings possible, its unlikely though as we run it through regression tests every day - the core functionality is sound, its usually the interaction with drivers that gets fun.

Do you know for sure that it wasn't corrupt/missing before running xfs_growfs (iow, did you run repair vefore and after the extend)?

Hard to say, without knowing what the problem is.

Oh, definately.

It's not clear that growfs did anything wrong yet. That approach will also avoid use of lvextend, fwiw, and I'd expect recreate-from-scratch will work as thats a commonly used scenario... so I guess thats your best approach, unless you want to diagnose the problem more closely (please do, for those who come after you...)

cheers.

Nathan