Alignment size?

DISCUSSION Michael Tokarev 5 participants 11 messages Aug 12, 2010 – Aug 17, 2010

lore

Share & tools

lore

Michael Tokarev

author

2010-08-12 22:10 UTC #0 lore raw

Hello.

I used XFS for a long time on many different servers, and it works well. But now I encountered an.. unexpected problem.

The question is: on one of our servers, XFS requires different alignment size for O_DIRECT operations than on others. Usually it's 512 bytes, but on this server it is 4096 - both min_io and alignment (this is from XFS_IOC_DIOINFO ioctl).

I'm not sure what the reason for this is.
On this server, the underlying block device is raid5 (linux sw raid), but we had other machines with raid5 which didn't have that alignment requiriments.

The problem with that is that Oracle db, which we use with XFS alot, refuses to work on this machine, or, rather, XFS refuses to process I/O in 512-byte chunks from oracle (control files and redolog files).

I know it is a frequent combination which is used in production in many places, and is used here alot too, but I haven't seen anyone mentioning this issue we have now, with "larger than usual" alignment size requiriments.

Is there a way to remedy this somehow, without reformatting whole 600+ gb?

Thank you!

/mjt

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner

2010-08-12 23:48 UTC #1 lore raw

Quoted 11 lines · Michael Tokarev

Hello.
I used XFS for a long time on many different servers, and it works well. But now I encountered an.. unexpected problem.
The question is: on one of our servers, XFS requires different alignment size for O_DIRECT operations than on others. Usually it's 512 bytes, but on this server it is 4096 - both min_io and alignment (this is from XFS_IOC_DIOINFO ioctl).

It'll be a filesystem set up with a 4k sector size, then. Check the output of xfs_info.

Quoted 9 lines

I'm not sure what the reason for this is.
On this server, the underlying block device is raid5 (linux sw raid), but we had other machines with raid5 which didn't have that alignment requiriments.
The problem with that is that Oracle db, which we use with XFS alot, refuses to work on this machine, or, rather, XFS refuses to process I/O in 512-byte chunks from oracle (control files and redolog files).

A clear case of application failure. I guess Oracle have some work to do to support 4k sector drives where they won't be able to do 512 byte direct IOs at all....

Quoted 2 lines

Is there a way to remedy this somehow, without reformatting whole 600+ gb?

Not really. If it is 4k sector size, then there is some extremely dangerous voodoo that you could do to realign and resize the AG headers, followed by a full xfs_repair run to fix up all the block accounting. This is not something I'd recommend anyone ever does, and for only 600GB of data it would probably take more time to work out how to do it correctly (using disposable filesystem images) than it would to dump, mkfs and restore...

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Michael Tokarev

author

2010-08-13 06:24 UTC #2 lore raw

Quoted 7 lines · Dave Chinner

Quoted 6 lines · Michael Tokarev
Hello.
I used XFS for a long time on many different servers, and it works well. But now I encountered an.. unexpected problem.

Quoted 8 lines

Quoted 5 lines
The question is: on one of our servers, XFS requires different alignment size for O_DIRECT operations than on others. Usually it's 512 bytes, but on this server it is 4096 - both min_io and alignment (this is from XFS_IOC_DIOINFO ioctl).
It'll be a filesystem set up with a 4k sector size, then. Check the output of xfs_info.

yes, xfs_info reports sectsz=4096, I noticed this yesterday.

Quoted 5 lines

Quoted 5 lines
I'm not sure what the reason for this is.
On this server, the underlying block device is raid5 (linux sw raid), but we had other machines with raid5 which didn't have that alignment requiriments.

Quoted 8 lines

Quoted 4 lines
The problem with that is that Oracle db, which we use with XFS alot, refuses to work on this machine, or, rather, XFS refuses to process I/O in 512-byte chunks from oracle (control files and redolog files).
A clear case of application failure. I guess Oracle have some work to do to support 4k sector drives where they won't be able to do 512 byte direct IOs at all....

Sure thing, that's oracle10, and at least at that time there was no way to determine the size of I/O in a generic way. Now there is, and I hope in oracle12 there will be support for various different sectors.

But this is not the point..
.

Quoted 10 lines

Quoted 2 lines
Is there a way to remedy this somehow, without reformatting whole 600+ gb?
Not really. If it is 4k sector size, then there is some extremely dangerous voodoo that you could do to realign and resize the AG headers, followed by a full xfs_repair run to fix up all the block accounting. This is not something I'd recommend anyone ever does, and for only 600GB of data it would probably take more time to work out how to do it correctly (using disposable filesystem images) than it would to dump, mkfs and restore...

Ugh. I see. Well, I was afraid of that, but I'm already sorta-prepared for that, after "sleeping with this idea"... ;) It'll take ages for sure, but there's no other choice for now.

So the question that remains is: why?

It's an old machine (PIV era), with old scsi disks (74Gb non-hotswap), -- the same disks as used on numerous other machines out there, where there's no such issue.

Plain old linux software raid array, also as used on many other systems.

At that time, all stuff were in 512 bytes for sure.

The array and filesystem were re-created last year (we added another drive to it), but I don't think at that time there were a kernel version that supported >512 sector sizes either (it was 2.6.27 I think).

So why xfs decided the block size is 4K??

And a related question, -- is there a way to create xfs fs with the right sector size? The filesystem were ok in years, not only on this machine, and I'm quite afraid to replace it with something else (e.g. ext4) in a hurry without good prior testing.

By the way, how one can check the "sector size" of a block device nowadays? I think I saw something about sysfs, but I see nothing of that sort in 2.6.32 kernel (which is used on this and other systems).

Thanks!

/mjt

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Stan Hoeppner

2010-08-13 10:22 UTC #3 lore raw

Michael Tokarev put forth on 8/13/2010 1:24 AM:

Quoted 1 line

So the question that remains is: why?

4096 is the default block size and has been since at least 2.6.26 when I started using XFS. From "man mkfs.xfs":

OPTIONS
-b block_size_options
This option specifies the fundamental block size of the filesystem. The valid block_size_options are: log=value or size=value and only one can be supplied. The block size is specified either as a base two logarithm value with log=, or in bytes with size=. The default value is 4096 bytes (4 KiB), the minimum is 512, and the maximum is 65536 (64 KiB). XFS on Linux currently only supports pagesize or smaller blocks.

Quoted 1 line

So why xfs decided the block size is 4K??

See above. It's the default. Dave, Eric, Alex and others may be able to explain why 4096 was chosen as the default. I'm guessing it has to do with the best all around performance across a wide variety of storage systems.

Quoted 2 lines

And a related question, -- is there a way to create xfs fs with the right sector size?

Yes.

-s sector_size
This option specifies the fundamental sector size of the filesystem. The sector_size is specified either as a value in bytes with size=value or as a base two logarithm value with log=value. The default sector_size is 512 bytes. The minimum value for sector size is 512; the maximum is 32768 (32 KiB). The sector_size must be a power of 2 size and cannot be made larger than the filesystem block size.

Note that the default is 512. This would lead me to believe that whoever created this 600GB XFS filesystem manually specified "-s 4096" on the command line when creating it.

Quoted 7 lines

The filesystem
were ok in years, not only on this machine, and I'm quite afraid to replace it with something else (e.g. ext4) in a hurry without good prior testing.
By the way, how one can check the "sector size" of a block device nowadays?

cat /sys/block/[device]/queue/hw_sector_size

That will give you the hardware sector size. As mentioned above, the XFS sector size can be manually specified during FS creation. Thus they may not match, which is likely the case with the 600GB FS you're having the problems with.

Signature · 6 lines

Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Michael Tokarev

author

2010-08-13 10:59 UTC #4 lore raw

Quoted 9 lines · Stan Hoeppner

Michael Tokarev put forth on 8/13/2010 1:24 AM:
Quoted 1 line
So the question that remains is: why?
4096 is the default block size and has been since at least 2.6.26 when I started using XFS. From "man mkfs.xfs":
OPTIONS
-b block_size_options

This is block size. But XFS_IOC_DIOINFO returns _sector_ size. All other XFS filesystems we have are made with the same 4096 _block_ size.

Quoted 3 lines

[..] The default value is 4096 bytes (4 KiB)
Quoted 1 line
So why xfs decided the block size is 4K??

That was the wrong question. The right one is about _sector_ size, not _block_ size.
The filesystem in question has _sector_ size =4096, all the rest has it =512.

Quoted 9 lines

Quoted 2 lines
And a related question, -- is there a way to create xfs fs with the right sector size?
Yes.
-s sector_size
This option specifies the fundamental sector size of the filesystem. The sector_size is specified either as a value in bytes with size=value or as a base two logarithm value with log=value. The default sector_size is 512

Yeah, the default is 512, my manpage agrees. But yet I've a filesystem that has it =4096...

But maybe it were specified during filesystem creation. I re-read the mkfs.xfs manpage yesterday, but somehow missed the sector size option (!), which you quoted above. Maybe we used it year ago when creating the filesystem, for yet to be determined reason... ;)

I just tried to create an xfs filesystem on this machine (on a small reserved partition) - it uses sector size = 512 as expected.

Quoted 4 lines

Quoted 2 lines
By the way, how one can check the "sector size" of a block device nowadays?
cat /sys/block/[device]/queue/hw_sector_size

And it shows 512 even for the md array in question.

Quoted 3 lines

That will give you the hardware sector size. As mentioned above, the XFS sector size can be manually specified during FS creation. Thus they may not match, which is likely the case with the 600GB FS you're having the problems with.

Yup!

Thank you all for the information, and please excuse me for the noize - just too many stuff at once... ;)

/mjt

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Roger Willcocks

2010-08-13 11:35 UTC #5 lore raw

In reply to Michael Tokarev

Quoted 1 line · Michael Tokarev

So why xfs decided the block size is 4K??

We had a similar problem with direct io here, with 2.6.9; I quote from the Bugzilla:

"mkfs.xfs has md-specific code (!) that looks at the raid flavour to figure stripe parameters, alignment requirements, etc.

"Raid flavours 4,5 and 6 force the alignment to be the same as the file system block size (which is 4096 bytes)."

Here's a program to test the alignment requirements:

----

#include <xfs/libxfs.h>
#include <fcntl.h>

int main(int argc, char* argv[])

{
	struct dioattr	dio;

	int tfd = open((argc == 2) ? argv[1] : "/mnt/disk1", O_RDONLY, 0666);

	if (ioctl(tfd, XFS_IOC_DIOINFO, &dio) < 0)
		perror("ioctl");
	else {
		printf("min io size = %d\n", dio.d_miniosz);
		printf("max io size = %d\n", dio.d_maxiosz);
		printf("align = %d\n", dio.d_mem);
	}
	close(tfd);
	return 0;
}

----

The same disk set returned 'align = 4096' for raid 5, but 'align = 512' for raid 0.

--
Roger

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner

2010-08-13 11:38 UTC #6 lore raw

In reply to Michael Tokarev

Quoted 18 lines · Michael Tokarev

Quoted 7 lines · Dave Chinner
Quoted 6 lines · Michael Tokarev
Hello.
I used XFS for a long time on many different servers, and it works well. But now I encountered an.. unexpected problem.
Quoted 8 lines
Quoted 5 lines
The question is: on one of our servers, XFS requires different alignment size for O_DIRECT operations than on others. Usually it's 512 bytes, but on this server it is 4096 - both min_io and alignment (this is from XFS_IOC_DIOINFO ioctl).
It'll be a filesystem set up with a 4k sector size, then. Check the output of xfs_info.
yes, xfs_info reports sectsz=4096, I noticed this yesterday.

....

Quoted 5 lines

So the question that remains is: why?
It's an old machine (PIV era), with old scsi disks (74Gb non-hotswap), -- the same disks as used on numerous other machines out there, where there's no such issue.

If the software was as old as the machine, then that's the likely reason. The old md raid5 implementation did not handle sub-page size aligned IO very well - a change of IO alignment would cause the stripe cache to be purged and cause performance to be terrible. Hence every time XFS wrote the superblock or an AG header it would purge the stripe cache.

The workaround old versions of mkfs.xfs used was to create the fs with a sector size of 4k when it detected md raid5 underneath it so the sb and ag headers were all 4k aligned and sized, just like the rest of the filesystem....

Quoted 5 lines

And a related question, -- is there a way to create xfs fs with the right sector size? The filesystem were ok in years, not only on this machine, and I'm quite afraid to replace it with something else (e.g. ext4) in a hurry without good prior testing.

# mkfs.xfs -s <size> ....

if you want to set it manually. YOu shouldn't need to with any relatively recent mkfs.xfs...

Quoted 4 lines

By the way, how one can check the "sector size" of a block device nowadays? I think I saw something about sysfs, but I see nothing of that sort in 2.6.32 kernel (which is used on this and other systems).

/sys/block/<dev>/queue/hw_sector_size

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Christoph Hellwig

2010-08-13 15:15 UTC #7 lore raw

Quoted 4 lines · Dave Chinner

The workaround old versions of mkfs.xfs used was to create the fs with a sector size of 4k when it detected md raid5 underneath it so the sb and ag headers were all 4k aligned and sized, just like the rest of the filesystem....

That workaround is still in latests mkfs.xfs if you build against the internal libdisk instead of libblkid. And that's the case at least for Debian, and probably a few other distributions as well.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Michael Tokarev

author

2010-08-17 00:18 UTC #8 lore raw

In reply to Dave Chinner

Quoted 1 line · Dave Chinner

On Fri, Aug 13, 2010 at 10:24:46AM +0400, Michael Tokarev wrote:

[]

Quoted 10 lines

Quoted 5 lines
And a related question, -- is there a way to create xfs fs with the right sector size? The filesystem were ok in years, not only on this machine, and I'm quite afraid to replace it with something else (e.g. ext4) in a hurry without good prior testing.
# mkfs.xfs -s <size> ....
if you want to set it manually. YOu shouldn't need to with any relatively recent mkfs.xfs...

Um. It appears that mkfs.xfs ignores -s size=512 on this raid5 array, and silently creates a filesystem with 4096 sector size, regardless of various -s size=nn and -s log=mm options.

This is xfsprogs 3.1.2-1 (debian squeeze package).

So the question stands...

Thanks!

/mjt

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Michael Tokarev

author

2010-08-17 00:30 UTC #9 lore raw

Quoted 5 lines · Michael Tokarev

Quoted 4 lines
# mkfs.xfs -s <size> ....
if you want to set it manually. YOu shouldn't need to with any relatively recent mkfs.xfs...

Quoted 8 lines

Um. It appears that mkfs.xfs ignores -s size=512 on this raid5 array, and silently creates a filesystem with 4096 sector size, regardless of various -s size=nn and -s log=mm options.
This is xfsprogs 3.1.2-1 (debian squeeze package).
So the question stands...

Debian builds it with internal blkid. Rebuilding it with libblkid fixes that.

Thanks!

/mjt

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave Chinner

2010-08-17 00:30 UTC #10 lore raw

In reply to Michael Tokarev

Quoted 14 lines · Michael Tokarev

Quoted 1 line · Dave Chinner
On Fri, Aug 13, 2010 at 10:24:46AM +0400, Michael Tokarev wrote:
[]
Quoted 10 lines
Quoted 5 lines
And a related question, -- is there a way to create xfs fs with the right sector size? The filesystem were ok in years, not only on this machine, and I'm quite afraid to replace it with something else (e.g. ext4) in a hurry without good prior testing.
# mkfs.xfs -s <size> ....
if you want to set it manually. YOu shouldn't need to with any relatively recent mkfs.xfs...

Quoted 8 lines

Um. It appears that mkfs.xfs ignores -s size=512 on this raid5 array, and silently creates a filesystem with 4096 sector size, regardless of various -s size=nn and -s log=mm options.
This is xfsprogs 3.1.2-1 (debian squeeze package).
So the question stands...

IIRC, the current debian xfsprogs package is still being built with the old detection library. If you install libblkid and build xfsprogs yourself, it should use the newer detection code and behave as expected.

Cheers,

Dave.

Signature · 7 lines

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs