Skip to contentSkip to conversation

xfs: allow logical-sector sized O_DIRECT for any fs sector size

PATCH Superseded Eric Sandeen 5 participants 16 messages Jan 15, 2014 – Jan 20, 2014 XFS filesystem

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Eric Sandeen

author
#0 lore raw

PATCH 1/1 xfs: allow logical-sector sized O_DIRECT for any fs sector size

Some time ago, mkfs.xfs started picking the storage physical sector size as the default filesystem "sector size" in order to avoid RMW costs incurred by doing IOs at logical sector size alignments.

However, this means that for a filesystem made with i.e. a 4k sector size on an "advanced format" 4k/512 disk, 512-byte direct IOs are no longer allowed. This means that XFS has essentially turned this AF drive into a hard 4K device, from the filesystem on up.

XFS's mkfs-specified "sector size" is really just controlling the minimum size & alignment of filesystem metadata IO.

There is no real need to tightly couple XFS's minimal metadata size to the minimum allowed direct IO size; XFS can continue doing metadata in optimal sizes, but still allow smaller DIOs for apps which issue them, for whatever reason.

This patch adds 2 new fields to the xfs_buftarg, so that we now track 3 sizes:

1) The device logical sector size
2) The device physical sector size
3) The filesystem sector size, which is the minimum unit and alignment of IO which will be performed by metadata operations.

The first is used for the minimum allowed direct IO alignment, the 2nd is used to report DIO sizes in XFS_IOC_DIOINFO (the theory being, if an app actually asks, we can give them the optimal answer, even if we allow smaller IOs), and the 3rd is used internally by the filesystem for metadata IOs.

This has passed xfstests on filesystems made with 4k sectors, including when run under the patch I sent to ignore XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly tested end of block behavior on preallocated, sparse, and existing files when we do a 512 IO into a 4k file on a 4k-sector filesystem, to be sure there were no unexpected behaviors.

Signed-off-by Eric Sandeen <[email protected]>

---

NB: This depends on this patch which is in the xfs tree, but not yet upstream:
xfs: simplify xfs_setsize_buftarg callchain; remove unused arg

fs/xfs/xfs_buf.c +4 -0
index 9fccfb5..a89dcdf 100644
@@ -1599,6 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
+	/* Set up filesystem block and sector sizes */
 	btp->bt_bsize = blocksize;
 	btp->bt_sshift = ffs(sectorsize) - 1;
 	btp->bt_smask = sectorsize - 1;

@@ -1614,6 +1615,9 @@ xfs_setsize_buftarg(
 		return EINVAL;
 	}
 
+	/* Set up device logical & physical sector size info */
+	btp->bt_lsmask = bdev_logical_block_size(btp->bt_bdev) - 1;
+	btp->bt_pssize = bdev_physical_block_size(btp->bt_bdev);
 	return 0;
 }
fs/xfs/xfs_buf.h +17 -3
index 1cf21a4..29a0db9 100644
@@ -88,14 +88,28 @@ typedef unsigned int xfs_buf_flags_t;
  */
 #define XFS_BSTATE_DISPOSE	 (1 << 0)	/* buffer being discarded */
 
+/*
+ * The xfs_buftarg contains 3 notions of "sector size" -
+ *
+ * 1) The device logical sector size
+ * 2) The device physical sector size
+ * 3) The filesystem sector size, which is the minimum unit and
+ *    alignment of IO which will be performed by metadata operations.
+ *
+ * The latter is specified at mkfs time, stored on-disk in the
+ * superblock's sb_sectsize, and is set from there.
+ */
+
 typedef struct xfs_buftarg {
 	dev_t			bt_dev;
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_bsize;
-	unsigned int		bt_sshift;
-	size_t			bt_smask;
+	unsigned int		bt_bsize;	/* fs block size */
+	unsigned int		bt_sshift;	/* fs sector size shift */
+	size_t			bt_smask;	/* fs sector size mask */
+	size_t			bt_lsmask;	/* dev logical sectsz mask */
+	unsigned int		bt_pssize;	/* dev physical sector size */
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +5 -2
index 52c91e1..09f9df9 100644
@@ -261,7 +261,8 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos & target->bt_smask) || (size & target->bt_smask)) {
+		/* DIO must be aligned to device logical sector size */
+		if ((pos & target->bt_lsmask) || (size & target->bt_lsmask)) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

@@ -641,9 +642,11 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
 
-	if ((pos & target->bt_smask) || (count & target->bt_smask))
+	/* DIO must be aligned to device logical sector size */
+	if ((pos & target->bt_lsmask) || (count & target->bt_lsmask))
 		return -XFS_ERROR(EINVAL);
 
+	/* "unaligned" here means not aligned to a filesystem block */
 	if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
 		unaligned_io = 1;
fs/xfs/xfs_ioctl.c +6 -1
index 33ad9a7..1f3431f 100644
@@ -1587,7 +1587,12 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		/*
+		 * Report device physical sector size as "optimal" minimum,
+		 * unless blocksize is smaller than that.
+		 */
+		da.d_miniosz = min(target->bt_pssize, target->bt_bsize);
+		da.d_mem = da.d_miniosz;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 11 lines · Eric Sandeen

Some time ago, mkfs.xfs started picking the storage physical sector size as the default filesystem "sector size" in order to avoid RMW costs incurred by doing IOs at logical sector size alignments.

However, this means that for a filesystem made with i.e. a 4k sector size on an "advanced format" 4k/512 disk, 512-byte direct IOs are no longer allowed. This means that XFS has essentially turned this AF drive into a hard 4K device, from the filesystem on up.

Quoted 8 lines

XFS's mkfs-specified "sector size" is really just controlling the minimum size & alignment of filesystem metadata IO.

There is no real need to tightly couple XFS's minimal metadata size to the minimum allowed direct IO size; XFS can continue doing metadata in optimal sizes, but still allow smaller DIOs for apps which issue them, for whatever reason.

Given that we already serialise sub-block IO completely, doing "sub-sector" IO will also be serialised so there shouldn't be any new issues introduced by this change.

Quoted 7 lines

This patch adds 2 new fields to the xfs_buftarg, so that we now track 3 sizes:

1) The device logical sector size
2) The device physical sector size
3) The filesystem sector size, which is the minimum unit and alignment of IO which will be performed by metadata operations.

I wouldn't call it the "filesystem sector size" because it's clear that it doesn't apply to everything in the filesystem. I'd prefer that we call it the "metadata sector size", similar to the "log sector size" we keep for situations where the external log device has a different sector size to the data device.

Quoted 5 lines

The first is used for the minimum allowed direct IO alignment, the 2nd is used to report DIO sizes in XFS_IOC_DIOINFO (the theory being, if an app actually asks, we can give them the optimal answer, even if we allow smaller IOs), and the 3rd is used internally by the filesystem for metadata IOs.

Terminology nit: the 3rd is set by mkfs to define the physical format constraints that metadata in the filesystem must obey. If we ever have to allocate sector sized metadata, then it will be used for more than just IO....

Quoted 22 lines

This has passed xfstests on filesystems made with 4k sectors, including when run under the patch I sent to ignore XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly tested end of block behavior on preallocated, sparse, and existing files when we do a 512 IO into a 4k file on a 4k-sector filesystem, to be sure there were no unexpected behaviors.

Signed-off-by Eric Sandeen <[email protected]>

---

NB: This depends on this patch which is in the xfs tree, but not yet upstream:
xfs: simplify xfs_setsize_buftarg callchain; remove unused arg

fs/xfs/xfs_buf.c +0 -0
index 9fccfb5..a89dcdf 100644

fs/xfs/xfs_buf.c xfs_setsize_buftarg( · line 1604

Quoted 6 lines
fs/xfs/xfs_buf.c +1 -0
@@ -1599,6 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
+	/* Set up filesystem block and sector sizes */
 	btp->bt_bsize = blocksize;
 	btp->bt_sshift = ffs(sectorsize) - 1;

The two places that use bt_sshift convert it back to a byte count. We should change this simply to being a byte count.

Quoted 34 lines
fs/xfs/xfs_buf.c +3 -0
 	btp->bt_smask = sectorsize - 1;

@@ -1614,6 +1615,9 @@ xfs_setsize_buftarg(
 		return EINVAL;
 	}
 
+	/* Set up device logical & physical sector size info */
+	btp->bt_lsmask = bdev_logical_block_size(btp->bt_bdev) - 1;
+	btp->bt_pssize = bdev_physical_block_size(btp->bt_bdev);
 	return 0;
 }
fs/xfs/xfs_buf.h +12 -0
index 1cf21a4..29a0db9 100644
@@ -88,14 +88,28 @@ typedef unsigned int xfs_buf_flags_t;
  */
 #define XFS_BSTATE_DISPOSE	 (1 << 0)	/* buffer being discarded */
 
+/*
+ * The xfs_buftarg contains 3 notions of "sector size" -
+ *
+ * 1) The device logical sector size
+ * 2) The device physical sector size
+ * 3) The filesystem sector size, which is the minimum unit and
+ *    alignment of IO which will be performed by metadata operations.
+ *
+ * The latter is specified at mkfs time, stored on-disk in the
+ * superblock's sb_sectsize, and is set from there.
+ */
+
 typedef struct xfs_buftarg {
 	dev_t			bt_dev;
 	struct block_device	*bt_bdev;

fs/xfs/xfs_buf.h typedef unsigned int xfs_buf_flags_t; · line 112

Quoted 10 lines
fs/xfs/xfs_buf.h +5 -3
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_bsize;
-	unsigned int		bt_sshift;
-	size_t			bt_smask;
+	unsigned int		bt_bsize;	/* fs block size */
+	unsigned int		bt_sshift;	/* fs sector size shift */
+	size_t			bt_smask;	/* fs sector size mask */
+	size_t			bt_lsmask;	/* dev logical sectsz mask */
+	unsigned int		bt_pssize;	/* dev physical sector size */

This patch makes bt_smask unused, so it can be removed. bt_bsize is also unused, so that should be removed, too. indeed, the buftarg has a backpointer to the xfs_mount, so we can get the block size from there if it is ever necessary.

And I think we should make these names a little more descriptive while we are touching them so comments to describe them are unnecessary:

	unsigned int		bt_meta_sectorsize;
	unsigned int		bt_physical_sectorsize;
	size_t			bt_logical_sectormask;
Quoted 5 lines
fs/xfs/xfs_ioctl.c +0 -0
index 33ad9a7..1f3431f 100644

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1594

Quoted 9 lines
fs/xfs/xfs_ioctl.c +5 -1
@@ -1587,7 +1587,12 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		/*
+		 * Report device physical sector size as "optimal" minimum,
+		 * unless blocksize is smaller than that.
+		 */
+		da.d_miniosz = min(target->bt_pssize, target->bt_bsize);

Just grab the filesysetm block size from the xfs_mount:

		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1595

Quoted 1 line
fs/xfs/xfs_ioctl.c +1 -0
+		da.d_mem = da.d_miniosz;

I'd suggest that this should be PAGE_SIZE - it's for memory buffer alignment, not IO alignment, so using the IO alignment just seems wrong to me...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw
Quoted 33 lines · Dave Chinner
Quoted 11 lines · Eric Sandeen

Some time ago, mkfs.xfs started picking the storage physical sector size as the default filesystem "sector size" in order to avoid RMW costs incurred by doing IOs at logical sector size alignments.

However, this means that for a filesystem made with i.e. a 4k sector size on an "advanced format" 4k/512 disk, 512-byte direct IOs are no longer allowed. This means that XFS has essentially turned this AF drive into a hard 4K device, from the filesystem on up.

Quoted 8 lines

XFS's mkfs-specified "sector size" is really just controlling the minimum size & alignment of filesystem metadata IO.

There is no real need to tightly couple XFS's minimal metadata size to the minimum allowed direct IO size; XFS can continue doing metadata in optimal sizes, but still allow smaller DIOs for apps which issue them, for whatever reason.

Given that we already serialise sub-block IO completely, doing "sub-sector" IO will also be serialised so there shouldn't be any new issues introduced by this change.

Quoted 7 lines

This patch adds 2 new fields to the xfs_buftarg, so that we now track 3 sizes:

1) The device logical sector size
2) The device physical sector size
3) The filesystem sector size, which is the minimum unit and alignment of IO which will be performed by metadata operations.

Quoted 5 lines

I wouldn't call it the "filesystem sector size" because it's clear that it doesn't apply to everything in the filesystem. I'd prefer that we call it the "metadata sector size", similar to the "log sector size" we keep for situations where the external log device has a different sector size to the data device.

Ok, fair enough.

Quoted 10 lines
Quoted 5 lines

The first is used for the minimum allowed direct IO alignment, the 2nd is used to report DIO sizes in XFS_IOC_DIOINFO (the theory being, if an app actually asks, we can give them the optimal answer, even if we allow smaller IOs), and the 3rd is used internally by the filesystem for metadata IOs.

Terminology nit: the 3rd is set by mkfs to define the physical format constraints that metadata in the filesystem must obey. If we ever have to allocate sector sized metadata, then it will be used for more than just IO....

Ok. I can fix comments. :D

Quoted 17 lines
Quoted 17 lines

This has passed xfstests on filesystems made with 4k sectors, including when run under the patch I sent to ignore XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly tested end of block behavior on preallocated, sparse, and existing files when we do a 512 IO into a 4k file on a 4k-sector filesystem, to be sure there were no unexpected behaviors.

Signed-off-by Eric Sandeen <[email protected]>

---

NB: This depends on this patch which is in the xfs tree, but not yet upstream:
xfs: simplify xfs_setsize_buftarg callchain; remove unused arg

Quoted 14 lines

fs/xfs/xfs_buf.c xfs_setsize_buftarg( · line 1604

Quoted 11 lines
fs/xfs/xfs_buf.c +1 -0
index 9fccfb5..a89dcdf 100644
@@ -1599,6 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
+	/* Set up filesystem block and sector sizes */
 	btp->bt_bsize = blocksize;
 	btp->bt_sshift = ffs(sectorsize) - 1;

The two places that use bt_sshift convert it back to a byte count. We should change this simply to being a byte count.

Ok, yeah, good point. I think I considered that along the way.

Quoted 41 lines
Quoted 41 lines
fs/xfs/xfs_buf.c +3 -0
 	btp->bt_smask = sectorsize - 1;

@@ -1614,6 +1615,9 @@ xfs_setsize_buftarg(
 		return EINVAL;
 	}
 
+	/* Set up device logical & physical sector size info */
+	btp->bt_lsmask = bdev_logical_block_size(btp->bt_bdev) - 1;
+	btp->bt_pssize = bdev_physical_block_size(btp->bt_bdev);
 	return 0;
 }
fs/xfs/xfs_buf.h +14 -3
index 1cf21a4..29a0db9 100644
@@ -88,14 +88,28 @@ typedef unsigned int xfs_buf_flags_t;
  */
 #define XFS_BSTATE_DISPOSE	 (1 << 0)	/* buffer being discarded */
 
+/*
+ * The xfs_buftarg contains 3 notions of "sector size" -
+ *
+ * 1) The device logical sector size
+ * 2) The device physical sector size
+ * 3) The filesystem sector size, which is the minimum unit and
+ *    alignment of IO which will be performed by metadata operations.
+ *
+ * The latter is specified at mkfs time, stored on-disk in the
+ * superblock's sb_sectsize, and is set from there.
+ */
+
 typedef struct xfs_buftarg {
 	dev_t			bt_dev;
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_bsize;
-	unsigned int		bt_sshift;
-	size_t			bt_smask;
+	unsigned int		bt_bsize;	/* fs block size */
+	unsigned int		bt_sshift;	/* fs sector size shift */
Quoted 8 lines

fs/xfs/xfs_buf.h typedef unsigned int xfs_buf_flags_t; · line 112

Quoted 3 lines
fs/xfs/xfs_buf.h +3 -0
+	size_t			bt_smask;	/* fs sector size mask */
+	size_t			bt_lsmask;	/* dev logical sectsz mask */
+	unsigned int		bt_pssize;	/* dev physical sector size */

This patch makes bt_smask unused, so it can be removed. bt_bsize is also unused, so that should be removed, too. indeed, the buftarg has a backpointer to the xfs_mount, so we can get the block size from there if it is ever necessary.

bt_bsize is used, until your suggestion below. :) But I can remove it with that change, sure.

Quoted 7 lines

And I think we should make these names a little more descriptive while we are touching them so comments to describe them are unnecessary:

	unsigned int		bt_meta_sectorsize;
	unsigned int		bt_physical_sectorsize;
	size_t			bt_logical_sectormask;

Ok. I was just sticking with the tradtional (lack of) verbosity.

Quoted 17 lines
Quoted 5 lines
fs/xfs/xfs_ioctl.c +0 -0
index 33ad9a7..1f3431f 100644

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1594

Quoted 9 lines
fs/xfs/xfs_ioctl.c +5 -1
@@ -1587,7 +1587,12 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		/*
+		 * Report device physical sector size as "optimal" minimum,
+		 * unless blocksize is smaller than that.
+		 */
+		da.d_miniosz = min(target->bt_pssize, target->bt_bsize);

Just grab the filesysetm block size from the xfs_mount:

Quoted 7 lines
		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1595

Quoted 1 line
fs/xfs/xfs_ioctl.c +1 -0
+		da.d_mem = da.d_miniosz;

I'd suggest that this should be PAGE_SIZE - it's for memory buffer alignment, not IO alignment, so using the IO alignment just seems wrong to me...

Ok. Was just sticking close to what we had before.

So:

 		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);
		da.d_mem = PAGE_SIZE;

? Then we can have a minimum IO size of 512, and a memory alignment of 4k, isn't that a little odd?

(IOWs we could do 512-aligned memory before, right? What's the downside, or the value in changing it now?)

Thanks,
-Eric

Quoted 4 lines

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
Quoted 30 lines · Eric Sandeen
Quoted 18 lines · Dave Chinner
Quoted 5 lines · Eric Sandeen
fs/xfs/xfs_ioctl.c +0 -0
index 33ad9a7..1f3431f 100644

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1594

Quoted 9 lines
fs/xfs/xfs_ioctl.c +5 -1
@@ -1587,7 +1587,12 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		/*
+		 * Report device physical sector size as "optimal" minimum,
+		 * unless blocksize is smaller than that.
+		 */
+		da.d_miniosz = min(target->bt_pssize, target->bt_bsize);

Just grab the filesysetm block size from the xfs_mount:

Quoted 7 lines
		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1595

Quoted 1 line
fs/xfs/xfs_ioctl.c +1 -0
+		da.d_mem = da.d_miniosz;

I'd suggest that this should be PAGE_SIZE - it's for memory buffer alignment, not IO alignment, so using the IO alignment just seems wrong to me...

Ok. Was just sticking close to what we had before.

So:

Quoted 8 lines
 		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);
		da.d_mem = PAGE_SIZE;

? Then we can have a minimum IO size of 512, and a memory alignment of 4k, isn't that a little odd?

(IOWs we could do 512-aligned memory before, right? What's the downside, or the value in changing it now?)

We can do arbitrary byte aligned buffers if I understand get_user_pages() correctly - it just maps the page under the buffer into the kernel address space and then the bio is pointed at it. AFAICT, the reason for the "memory buffer needs 512 byte alignment" is simply that this is the minimum IO size supported.

However, for large IOs, 512 byte alignment is not really optimal. If we don't align the buffer to PAGE_SIZE, then we have partial head and tail pages, so for a 512k IO we need to map 129 pages into a bio instead of 128. When you have hardware that can only handle 128 segments in a DMA transfer, that means the 512k IO needs to be sent in two IOs rather than one.

There's quite a bit of hardware out there that have a limit of 128 segments to each IO....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 37 lines · Dave Chinner
Quoted 36 lines · Eric Sandeen
Quoted 18 lines · Dave Chinner
Quoted 5 lines · Eric Sandeen
fs/xfs/xfs_ioctl.c +0 -0
index 33ad9a7..1f3431f 100644

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1594

Quoted 9 lines
fs/xfs/xfs_ioctl.c +5 -1
@@ -1587,7 +1587,12 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		/*
+		 * Report device physical sector size as "optimal" minimum,
+		 * unless blocksize is smaller than that.
+		 */
+		da.d_miniosz = min(target->bt_pssize, target->bt_bsize);

Just grab the filesysetm block size from the xfs_mount:

Quoted 7 lines
		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);

fs/xfs/xfs_ioctl.c xfs_file_ioctl( · line 1595

Quoted 1 line
fs/xfs/xfs_ioctl.c +1 -0
+		da.d_mem = da.d_miniosz;

I'd suggest that this should be PAGE_SIZE - it's for memory buffer alignment, not IO alignment, so using the IO alignment just seems wrong to me...

Ok. Was just sticking close to what we had before.

So:

 		da.d_miniosz = min(target->bt_pssize, mp->m_sb.sb_blocksize);
		da.d_mem = PAGE_SIZE;

? Then we can have a minimum IO size of 512, and a memory alignment of 4k, isn't that a little odd?

Quoted 8 lines
Quoted 2 lines

(IOWs we could do 512-aligned memory before, right? What's the downside, or the value in changing it now?)

We can do arbitrary byte aligned buffers if I understand get_user_pages() correctly - it just maps the page under the buffer into the kernel address space and then the bio is pointed at it. AFAICT, the reason for the "memory buffer needs 512 byte alignment" is simply that this is the minimum IO size supported.

Actually, it's fs/direct-io.c which enforces this (not sure why I couldn't find that before), in do_blockdev_direct_IO(); the *enforced* minimum memory alignment is the size of the bev's logical block size.

Quoted 6 lines

However, for large IOs, 512 byte alignment is not really optimal. If we don't align the buffer to PAGE_SIZE, then we have partial head and tail pages, so for a 512k IO we need to map 129 pages into a bio instead of 128. When you have hardware that can only handle 128 segments in a DMA transfer, that means the 512k IO needs to be sent in two IOs rather than one.

Ok, but I have a bit of a problem with changing what XFS_IOC_DIOINFO reports. (I had originally thought to change the minimum IO size, but I have talked myself out of that, too).

The xfsctl(3) manpage says that XFS_IOC_DIOINFO: "Get(s) information required to perform direct I/O on the specified file descriptor." and "the user’s data buffer must conform to the same type of constraints as required for accessing a raw disk partition."

IOWs, the ioctl is documented as returning minimum, not optimal, requirements, and it has always been implemented as such. Changing its meaning now seems wrong. At least, I would not like to do so as part of this functional change; to do so would probably be best in a new ioctl which reports both minimum & optimal sizes. And at that point we should just do a vfs interface. :)

Of course, applications don't have to use the minimum sizes reported by the ioctl; they are free to be smarter and do larger sizes and alignments. But if the ioctl was designed and documented to report required *minimums*, I think we should leave it as such.

I'm going to resend the change, split up a bit more to separate cleanups from functional changes, and maybe we can discuss the ioctl change as a potential additional patch.

Thanks,
-Eric

Quoted 7 lines

There's quite a bit of hardware out there that have a limit of 128 segments to each IO....

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

v2 cover xfs: allow logical-sector sized O_DIRECT for any fs sector size

In reply to Eric Sandeen

Resend of the previous patch, now in 3 parts, incorporating some of Dave's suggestions, and split up for SUPER-easy review. ;)

I have explicitly not changed XFS_IOC_DIOINFO output for now; it still reports the actual minimum sizes and alignments which will be allowed by the filesystem (and block layer beneath it).

Thanks,
-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw

PATCH v2 1/3 xfs: clean up xfs_buftarg

Clean up the xfs_buftarg structure a bit:
- remove bt_bsize, which is never used
- replace bt_sshift with bt_ssize; we only ever shift it back to a size anyway

Signed-off-by Eric Sandeen <[email protected]>

---

fs/xfs/xfs_buf.c +2 -3
index 9fccfb5..b664bce 100644
@@ -445,7 +445,7 @@ _xfs_buf_find(
 	numbytes = BBTOB(numblks);
 
 	/* Check for IOs smaller than the sector size / not sector aligned */
-	ASSERT(!(numbytes < (1 << btp->bt_sshift)));
+	ASSERT(!(numbytes < btp->bt_ssize));
 	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_smask));
 
 	/*

@@ -1599,8 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
-	btp->bt_bsize = blocksize;
-	btp->bt_sshift = ffs(sectorsize) - 1;
+	btp->bt_ssize = sectorsize;
 	btp->bt_smask = sectorsize - 1;
 
 	if (set_blocksize(btp->bt_bdev, sectorsize)) {
fs/xfs/xfs_buf.h +1 -2
index 1cf21a4..4ef949a 100644
@@ -93,8 +93,7 @@ typedef struct xfs_buftarg {
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_bsize;
-	unsigned int		bt_sshift;
+	unsigned int		bt_ssize;
 	size_t			bt_smask;
 
 	/* LRU control structures */
fs/xfs/xfs_ioctl.c +1 -1
index 33ad9a7..39c38ee 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		da.d_mem = da.d_miniosz = target->bt_ssize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw

PATCH v2 2/3 xfs: rename xfs_buftarg structure members

In reply to Eric Sandeen

In preparation for adding new members to the xfs_buftarg structure, give these old ones more descriptive names:

	bt_ssize -> bt_meta_sectorsize
	bt_smask -> bt_meta_sectormask

(Note the slight change in logic format in xfs_file_aio_read(() and xfs_file_dio_aio_write(); this was done just for formatting, and should not change actual logic or behavior.)

Signed-off-by Eric Sandeen <[email protected]>

---

fs/xfs/xfs_buf.c +4 -4
index b664bce..a526f8d 100644
@@ -445,8 +445,8 @@ _xfs_buf_find(
 	numbytes = BBTOB(numblks);
 
 	/* Check for IOs smaller than the sector size / not sector aligned */
-	ASSERT(!(numbytes < btp->bt_ssize));
-	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_smask));
+	ASSERT(!(numbytes < btp->bt_meta_sectorsize));
+	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_meta_sectormask));
 
 	/*
 	 * Corrupted block numbers can get through to here, unfortunately, so we

@@ -1599,8 +1599,8 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
-	btp->bt_ssize = sectorsize;
-	btp->bt_smask = sectorsize - 1;
+	btp->bt_meta_sectorsize = sectorsize;
+	btp->bt_meta_sectormask = sectorsize - 1;
 
 	if (set_blocksize(btp->bt_bdev, sectorsize)) {
 		char name[BDEVNAME_SIZE];
fs/xfs/xfs_buf.h +2 -2
index 4ef949a..d5d88dd 100644
@@ -93,8 +93,8 @@ typedef struct xfs_buftarg {
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_ssize;
-	size_t			bt_smask;
+	unsigned int		bt_meta_sectorsize;
+	size_t			bt_meta_sectormask;
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +2 -2
index 52c91e1..61a7de0 100644
@@ -261,7 +261,7 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos & target->bt_smask) || (size & target->bt_smask)) {
+		if ((pos || size) & target->bt_meta_sectormask) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

@@ -641,7 +641,7 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
 
-	if ((pos & target->bt_smask) || (count & target->bt_smask))
+	if ((pos || count) & target->bt_meta_sectormask)
 		return -XFS_ERROR(EINVAL);
 
 	if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
fs/xfs/xfs_ioctl.c +1 -1
index 39c38ee..64ca8e9 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = target->bt_ssize;
+		da.d_mem = da.d_miniosz = target->bt_meta_sectorsize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#13 lore raw

PATCH v2 3/3 xfs: allow logical-sector sized O_DIRECT IOs

In reply to Eric Sandeen

Some time ago, mkfs.xfs started picking the storage physical sector size as the default filesystem "sector size" in order to avoid RMW costs incurred by doing IOs at logical sector size alignments.

However, this means that for a filesystem made with i.e. a 4k sector size on an "advanced format" 4k/512 disk, 512-byte direct IOs are no longer allowed. This means that XFS has essentially turned this AF drive into a hard 4K device, from the filesystem on up.

XFS's mkfs-specified "sector size" is really just controlling the minimum size & alignment of filesystem metadata.

There is no real need to tightly couple XFS's minimal metadata size to the minimum allowed direct IO size; XFS can continue doing metadata in optimal sizes, but still allow smaller DIOs for apps which issue them, for whatever reason.

This patch adds new information to the xfs_buftarg, so that we now track 2 sizes:

1) The metadata sector size, which is the minimum unit and alignment of IO which will be performed by metadata operations.
2) The device logical sector size.

The first is used internally by the file system for metadata alignment and IOs.
The second is used for the minimum allowed direct IO size and alignment.

This has passed xfstests on filesystems made with 4k sectors, including when run under the patch I sent to ignore XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly tested end of block behavior on preallocated, sparse, and existing files when we do a 512 IO into a 4k file on a 4k-sector filesystem, to be sure there were no unexpected behaviors.

Signed-off-by Eric Sandeen <[email protected]>
fs/xfs/xfs_buf.c +5 -0
index a526f8d..5175711 100644
@@ -1599,6 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
+	/* Set up metadata sector size info */
 	btp->bt_meta_sectorsize = sectorsize;
 	btp->bt_meta_sectormask = sectorsize - 1;
 

@@ -1613,6 +1614,10 @@ xfs_setsize_buftarg(
 		return EINVAL;
 	}
 
+	/* Set up device logical sector size mask */
+	btp->bt_logical_sectorsize = bdev_logical_block_size(btp->bt_bdev);
+	btp->bt_logical_sectormask = bdev_logical_block_size(btp->bt_bdev) - 1;
+
 	return 0;
 }
fs/xfs/xfs_buf.h +15 -0
index d5d88dd..9953395 100644
@@ -88,6 +88,19 @@ typedef unsigned int xfs_buf_flags_t;
  */
 #define XFS_BSTATE_DISPOSE	 (1 << 0)	/* buffer being discarded */
 
+/*
+ * The xfs_buftarg contains 2 notions of "sector size" -
+ *
+ * 1) The metadata sector size, which is the minimum unit and
+ *    alignment of IO which will be performed by metadata operations.
+ * 2) The device logical sector size
+ *
+ * The first is specified at mkfs time, and is stored on-disk in the
+ * superblock's sb_sectsize.
+ *
+ * The latter is derived from the underlying device, and controls direct IO
+ * alignment constraints.
+ */
 typedef struct xfs_buftarg {
 	dev_t			bt_dev;
 	struct block_device	*bt_bdev;

@@ -95,6 +108,8 @@ typedef struct xfs_buftarg {
 	struct xfs_mount	*bt_mount;
 	unsigned int		bt_meta_sectorsize;
 	size_t			bt_meta_sectormask;
+	size_t			bt_logical_sectorsize;
+	size_t			bt_logical_sectormask;
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +5 -2
index 61a7de0..5507420 100644
@@ -261,7 +261,8 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos || size) & target->bt_meta_sectormask) {
+		/* DIO must be aligned to device logical sector size */
+		if ((pos || size) & target->bt_logical_sectormask) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

@@ -641,9 +642,11 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
 
-	if ((pos || count) & target->bt_meta_sectormask)
+	/* DIO must be aligned to device logical sector size */
+	if ((pos || count) & target->bt_logical_sectormask)
 		return -XFS_ERROR(EINVAL);
 
+	/* "unaligned" here means not aligned to a filesystem block */
 	if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
 		unaligned_io = 1;
fs/xfs/xfs_ioctl.c +1 -1
index 64ca8e9..f7ac335 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = target->bt_meta_sectorsize;
+		da.d_mem =  da.d_miniosz = target->bt_logical_sectorsize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw

Re: [PATCH 2/3] xfs: rename xfs_buftarg structure members

Quoted 5 lines · Roger Willcocks

- if ((pos & target->bt_smask) || (size & target->bt_smask)) {
+ if ((pos || size) & target->bt_meta_sectormask) {

nope...

Quoted 9 lines

|| needs to be |

(pos || size) yields a boolean

Roger Willcocks <[email protected]>

Oof. Brain fart. Thank you :)

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw

PATCH v2 2/3 xfs: rename xfs_buftarg structure members

In reply to Eric Sandeen

In preparation for adding new members to the xfs_buftarg structure, give these old ones more descriptive names:

	bt_ssize -> bt_meta_sectorsize
	bt_smask -> bt_meta_sectormask

(Note the slight change in logic format in xfs_file_aio_read(() and xfs_file_dio_aio_write(); this was done just for formatting, and should not change actual logic or behavior.)

Signed-off-by Eric Sandeen <[email protected]>

---

V2: Really make the logic equivalent, thanks Roger!

fs/xfs/xfs_buf.c +4 -4
index b664bce..a526f8d 100644
@@ -445,8 +445,8 @@ _xfs_buf_find(
 	numbytes = BBTOB(numblks);
 
 	/* Check for IOs smaller than the sector size / not sector aligned */
-	ASSERT(!(numbytes < btp->bt_ssize));
-	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_smask));
+	ASSERT(!(numbytes < btp->bt_meta_sectorsize));
+	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_meta_sectormask));
 
 	/*
 	 * Corrupted block numbers can get through to here, unfortunately, so we

@@ -1599,8 +1599,8 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
-	btp->bt_ssize = sectorsize;
-	btp->bt_smask = sectorsize - 1;
+	btp->bt_meta_sectorsize = sectorsize;
+	btp->bt_meta_sectormask = sectorsize - 1;
 
 	if (set_blocksize(btp->bt_bdev, sectorsize)) {
 		char name[BDEVNAME_SIZE];
fs/xfs/xfs_buf.h +2 -2
index 4ef949a..d5d88dd 100644
@@ -93,8 +93,8 @@ typedef struct xfs_buftarg {
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_ssize;
-	size_t			bt_smask;
+	unsigned int		bt_meta_sectorsize;
+	size_t			bt_meta_sectormask;
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +2 -2
index 52c91e1..61a7de0 100644
@@ -261,7 +261,7 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos & target->bt_smask) || (size & target->bt_smask)) {
+		if ((pos | size) & target->bt_meta_sectormask) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

@@ -641,7 +641,7 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
 
-	if ((pos & target->bt_smask) || (count & target->bt_smask))
+	if ((pos | count) & target->bt_meta_sectormask)
 		return -XFS_ERROR(EINVAL);
 
 	if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
fs/xfs/xfs_ioctl.c +1 -1
index 39c38ee..64ca8e9 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = target->bt_ssize;
+		da.d_mem = da.d_miniosz = target->bt_meta_sectorsize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by on v2 1/3
#7 lore raw

Re: [PATCH 1/3] xfs: clean up xfs_buftarg

In reply to Eric Sandeen
Quoted 7 lines · Eric Sandeen

Clean up the xfs_buftarg structure a bit:
- remove bt_bsize, which is never used
- replace bt_sshift with bt_ssize; we only ever shift it back to a size anyway

Signed-off-by Eric Sandeen <[email protected]>

---

Reviewed-by Brian Foster <[email protected]>
Quoted 52 lines
fs/xfs/xfs_buf.c +2 -3
index 9fccfb5..b664bce 100644
@@ -445,7 +445,7 @@ _xfs_buf_find(
 	numbytes = BBTOB(numblks);
 
 	/* Check for IOs smaller than the sector size / not sector aligned */
-	ASSERT(!(numbytes < (1 << btp->bt_sshift)));
+	ASSERT(!(numbytes < btp->bt_ssize));
 	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_smask));
 
 	/*

@@ -1599,8 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
-	btp->bt_bsize = blocksize;
-	btp->bt_sshift = ffs(sectorsize) - 1;
+	btp->bt_ssize = sectorsize;
 	btp->bt_smask = sectorsize - 1;
 
 	if (set_blocksize(btp->bt_bdev, sectorsize)) {
fs/xfs/xfs_buf.h +1 -2
index 1cf21a4..4ef949a 100644
@@ -93,8 +93,7 @@ typedef struct xfs_buftarg {
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_bsize;
-	unsigned int		bt_sshift;
+	unsigned int		bt_ssize;
 	size_t			bt_smask;
 
 	/* LRU control structures */
fs/xfs/xfs_ioctl.c +1 -1
index 33ad9a7..39c38ee 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = 1 << target->bt_sshift;
+		da.d_mem = da.d_miniosz = target->bt_ssize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by on v2 2/3
#12 lore raw

Re: [PATCH 2/3 V2] xfs: rename xfs_buftarg structure members

In reply to Eric Sandeen
Quoted 6 lines · Eric Sandeen

In preparation for adding new members to the xfs_buftarg structure, give these old ones more descriptive names:

	bt_ssize -> bt_meta_sectorsize
	bt_smask -> bt_meta_sectormask
Quoted 6 lines

(Note the slight change in logic format in xfs_file_aio_read(() and xfs_file_dio_aio_write(); this was done just for formatting, and should not change actual logic or behavior.)

Signed-off-by Eric Sandeen <[email protected]>

---

Reviewed-by Brian Foster <[email protected]>
Quoted 87 lines

V2: Really make the logic equivalent, thanks Roger!

fs/xfs/xfs_buf.c +4 -4
index b664bce..a526f8d 100644
@@ -445,8 +445,8 @@ _xfs_buf_find(
 	numbytes = BBTOB(numblks);
 
 	/* Check for IOs smaller than the sector size / not sector aligned */
-	ASSERT(!(numbytes < btp->bt_ssize));
-	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_smask));
+	ASSERT(!(numbytes < btp->bt_meta_sectorsize));
+	ASSERT(!(BBTOB(blkno) & (xfs_off_t)btp->bt_meta_sectormask));
 
 	/*
 	 * Corrupted block numbers can get through to here, unfortunately, so we

@@ -1599,8 +1599,8 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
-	btp->bt_ssize = sectorsize;
-	btp->bt_smask = sectorsize - 1;
+	btp->bt_meta_sectorsize = sectorsize;
+	btp->bt_meta_sectormask = sectorsize - 1;
 
 	if (set_blocksize(btp->bt_bdev, sectorsize)) {
 		char name[BDEVNAME_SIZE];
fs/xfs/xfs_buf.h +2 -2
index 4ef949a..d5d88dd 100644
@@ -93,8 +93,8 @@ typedef struct xfs_buftarg {
 	struct block_device	*bt_bdev;
 	struct backing_dev_info	*bt_bdi;
 	struct xfs_mount	*bt_mount;
-	unsigned int		bt_ssize;
-	size_t			bt_smask;
+	unsigned int		bt_meta_sectorsize;
+	size_t			bt_meta_sectormask;
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +2 -2
index 52c91e1..61a7de0 100644
@@ -261,7 +261,7 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos & target->bt_smask) || (size & target->bt_smask)) {
+		if ((pos | size) & target->bt_meta_sectormask) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

@@ -641,7 +641,7 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
 
-	if ((pos & target->bt_smask) || (count & target->bt_smask))
+	if ((pos | count) & target->bt_meta_sectormask)
 		return -XFS_ERROR(EINVAL);
 
 	if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
fs/xfs/xfs_ioctl.c +1 -1
index 39c38ee..64ca8e9 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = target->bt_ssize;
+		da.d_mem = da.d_miniosz = target->bt_meta_sectorsize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#14 lore raw

Re: [PATCH 3/3] xfs: allow logical-sector sized O_DIRECT IOs

In reply to Eric Sandeen
Quoted 112 lines · Eric Sandeen

Some time ago, mkfs.xfs started picking the storage physical sector size as the default filesystem "sector size" in order to avoid RMW costs incurred by doing IOs at logical sector size alignments.

However, this means that for a filesystem made with i.e. a 4k sector size on an "advanced format" 4k/512 disk, 512-byte direct IOs are no longer allowed. This means that XFS has essentially turned this AF drive into a hard 4K device, from the filesystem on up.

XFS's mkfs-specified "sector size" is really just controlling the minimum size & alignment of filesystem metadata.

There is no real need to tightly couple XFS's minimal metadata size to the minimum allowed direct IO size; XFS can continue doing metadata in optimal sizes, but still allow smaller DIOs for apps which issue them, for whatever reason.

This patch adds new information to the xfs_buftarg, so that we now track 2 sizes:

1) The metadata sector size, which is the minimum unit and alignment of IO which will be performed by metadata operations.
2) The device logical sector size.

The first is used internally by the file system for metadata alignment and IOs.
The second is used for the minimum allowed direct IO size and alignment.

This has passed xfstests on filesystems made with 4k sectors, including when run under the patch I sent to ignore XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly tested end of block behavior on preallocated, sparse, and existing files when we do a 512 IO into a 4k file on a 4k-sector filesystem, to be sure there were no unexpected behaviors.

Signed-off-by Eric Sandeen <[email protected]>
fs/xfs/xfs_buf.c +5 -0
index a526f8d..5175711 100644
@@ -1599,6 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
+	/* Set up metadata sector size info */
 	btp->bt_meta_sectorsize = sectorsize;
 	btp->bt_meta_sectormask = sectorsize - 1;
 

@@ -1613,6 +1614,10 @@ xfs_setsize_buftarg(
 		return EINVAL;
 	}
 
+	/* Set up device logical sector size mask */
+	btp->bt_logical_sectorsize = bdev_logical_block_size(btp->bt_bdev);
+	btp->bt_logical_sectormask = bdev_logical_block_size(btp->bt_bdev) - 1;
+
 	return 0;
 }
fs/xfs/xfs_buf.h +15 -0
index d5d88dd..9953395 100644
@@ -88,6 +88,19 @@ typedef unsigned int xfs_buf_flags_t;
  */
 #define XFS_BSTATE_DISPOSE	 (1 << 0)	/* buffer being discarded */
 
+/*
+ * The xfs_buftarg contains 2 notions of "sector size" -
+ *
+ * 1) The metadata sector size, which is the minimum unit and
+ *    alignment of IO which will be performed by metadata operations.
+ * 2) The device logical sector size
+ *
+ * The first is specified at mkfs time, and is stored on-disk in the
+ * superblock's sb_sectsize.
+ *
+ * The latter is derived from the underlying device, and controls direct IO
+ * alignment constraints.
+ */
 typedef struct xfs_buftarg {
 	dev_t			bt_dev;
 	struct block_device	*bt_bdev;

@@ -95,6 +108,8 @@ typedef struct xfs_buftarg {
 	struct xfs_mount	*bt_mount;
 	unsigned int		bt_meta_sectorsize;
 	size_t			bt_meta_sectormask;
+	size_t			bt_logical_sectorsize;
+	size_t			bt_logical_sectormask;
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +2 -1
index 61a7de0..5507420 100644
@@ -261,7 +261,8 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos || size) & target->bt_meta_sectormask) {
+		/* DIO must be aligned to device logical sector size */
+		if ((pos || size) & target->bt_logical_sectormask) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

fs/xfs/xfs_file.c xfs_file_dio_aio_write( · line 648

Quoted 8 lines
fs/xfs/xfs_file.c +2 -1
@@ -641,9 +642,11 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
 
-	if ((pos || count) & target->bt_meta_sectormask)
+	/* DIO must be aligned to device logical sector size */
+	if ((pos || count) & target->bt_logical_sectormask)
 		return -XFS_ERROR(EINVAL);

Looks like this requires a v2 to preserve the fixes made in v2 of patch 2, unless I missed a newer version of this somewhere.

Brian

Quoted 18 lines
fs/xfs/xfs_file.c +1 -0
+	/* "unaligned" here means not aligned to a filesystem block */
 	if ((pos & mp->m_blockmask) || ((pos + count) & mp->m_blockmask))
 		unaligned_io = 1;
fs/xfs/xfs_ioctl.c +1 -1
index 64ca8e9..f7ac335 100644
@@ -1587,7 +1587,7 @@ xfs_file_ioctl(
 			XFS_IS_REALTIME_INODE(ip) ?
 			mp->m_rtdev_targp : mp->m_ddev_targp;
 
-		da.d_mem = da.d_miniosz = target->bt_meta_sectorsize;
+		da.d_mem =  da.d_miniosz = target->bt_logical_sectorsize;
 		da.d_maxiosz = INT_MAX & ~(da.d_miniosz - 1);
 
 		if (copy_to_user(arg, &da, sizeof(da)))
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#15 lore raw

Re: [PATCH 3/3] xfs: allow logical-sector sized O_DIRECT IOs

Quoted 116 lines · Brian Foster
Quoted 115 lines · Eric Sandeen

Some time ago, mkfs.xfs started picking the storage physical sector size as the default filesystem "sector size" in order to avoid RMW costs incurred by doing IOs at logical sector size alignments.

However, this means that for a filesystem made with i.e. a 4k sector size on an "advanced format" 4k/512 disk, 512-byte direct IOs are no longer allowed. This means that XFS has essentially turned this AF drive into a hard 4K device, from the filesystem on up.

XFS's mkfs-specified "sector size" is really just controlling the minimum size & alignment of filesystem metadata.

There is no real need to tightly couple XFS's minimal metadata size to the minimum allowed direct IO size; XFS can continue doing metadata in optimal sizes, but still allow smaller DIOs for apps which issue them, for whatever reason.

This patch adds new information to the xfs_buftarg, so that we now track 2 sizes:

1) The metadata sector size, which is the minimum unit and alignment of IO which will be performed by metadata operations.
2) The device logical sector size.

The first is used internally by the file system for metadata alignment and IOs.
The second is used for the minimum allowed direct IO size and alignment.

This has passed xfstests on filesystems made with 4k sectors, including when run under the patch I sent to ignore XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly tested end of block behavior on preallocated, sparse, and existing files when we do a 512 IO into a 4k file on a 4k-sector filesystem, to be sure there were no unexpected behaviors.

Signed-off-by Eric Sandeen <[email protected]>
fs/xfs/xfs_buf.c +5 -0
index a526f8d..5175711 100644
@@ -1599,6 +1599,7 @@ xfs_setsize_buftarg(
 	unsigned int		blocksize,
 	unsigned int		sectorsize)
 {
+	/* Set up metadata sector size info */
 	btp->bt_meta_sectorsize = sectorsize;
 	btp->bt_meta_sectormask = sectorsize - 1;
 

@@ -1613,6 +1614,10 @@ xfs_setsize_buftarg(
 		return EINVAL;
 	}
 
+	/* Set up device logical sector size mask */
+	btp->bt_logical_sectorsize = bdev_logical_block_size(btp->bt_bdev);
+	btp->bt_logical_sectormask = bdev_logical_block_size(btp->bt_bdev) - 1;
+
 	return 0;
 }
fs/xfs/xfs_buf.h +15 -0
index d5d88dd..9953395 100644
@@ -88,6 +88,19 @@ typedef unsigned int xfs_buf_flags_t;
  */
 #define XFS_BSTATE_DISPOSE	 (1 << 0)	/* buffer being discarded */
 
+/*
+ * The xfs_buftarg contains 2 notions of "sector size" -
+ *
+ * 1) The metadata sector size, which is the minimum unit and
+ *    alignment of IO which will be performed by metadata operations.
+ * 2) The device logical sector size
+ *
+ * The first is specified at mkfs time, and is stored on-disk in the
+ * superblock's sb_sectsize.
+ *
+ * The latter is derived from the underlying device, and controls direct IO
+ * alignment constraints.
+ */
 typedef struct xfs_buftarg {
 	dev_t			bt_dev;
 	struct block_device	*bt_bdev;

@@ -95,6 +108,8 @@ typedef struct xfs_buftarg {
 	struct xfs_mount	*bt_mount;
 	unsigned int		bt_meta_sectorsize;
 	size_t			bt_meta_sectormask;
+	size_t			bt_logical_sectorsize;
+	size_t			bt_logical_sectormask;
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
fs/xfs/xfs_file.c +2 -1
index 61a7de0..5507420 100644
@@ -261,7 +261,8 @@ xfs_file_aio_read(
 		xfs_buftarg_t	*target =
 			XFS_IS_REALTIME_INODE(ip) ?
 				mp->m_rtdev_targp : mp->m_ddev_targp;
-		if ((pos || size) & target->bt_meta_sectormask) {
+		/* DIO must be aligned to device logical sector size */
+		if ((pos || size) & target->bt_logical_sectormask) {
 			if (pos == i_size_read(inode))
 				return 0;
 			return -XFS_ERROR(EINVAL);

@@ -641,9 +642,11 @@ xfs_file_dio_aio_write(
 	struct xfs_buftarg	*target = XFS_IS_REALTIME_INODE(ip) ?
 					mp->m_rtdev_targp : mp->m_ddev_targp;
Quoted 8 lines

fs/xfs/xfs_file.c xfs_file_dio_aio_write( · line 648

Quoted 5 lines
fs/xfs/xfs_file.c +2 -1
-	if ((pos || count) & target->bt_meta_sectormask)
+	/* DIO must be aligned to device logical sector size */
+	if ((pos || count) & target->bt_logical_sectormask)
 		return -XFS_ERROR(EINVAL);

Looks like this requires a v2 to preserve the fixes made in v2 of patch 2, unless I missed a newer version of this somewhere.

You're right. 2 diligence demerits for me. :(

(Hopefully it just wouldn't have applied...)

Maybe I'll just resend the whole series.

Thanks,
-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs