Skip to contentSkip to conversation

fix FITRIM with non-discard capable RT device v2

PATCH In review Christoph Hellwig 2 participants 9 messages Aug 16, 2024 – Aug 20, 2024 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi all,

this series fixes FITRIM when an RT device that does not support discards is present.

Changes since v1:
- explicitly reject the discard capable RT device and non-discard capable main device case because it is too much of a mess to handle and not practically useful

Diffstat:

 xfs_discard.c |   57 +++++++++++++++++++++++++++++++--------------------------
 1 file changed, 31 insertions(+), 26 deletions(-)
#1 lore raw

PATCH 1/2 xfs: remove a stale comment in xfs_ioc_trim

There is no truncating down going on here, the code has changed multiple times since the comment was added with the initial FITRIM implementation and it doesn't make sense in the current context.

---
 fs/xfs/xfs_discard.c | 7 -------
 1 file changed, 7 deletions(-)
fs/xfs/xfs_discard.c +0 -7
index 80586336276c19..d56efe9eae2cef 100644
@@ -720,13 +720,6 @@ xfs_ioc_trim(
 	range.minlen = max_t(u64, granularity, range.minlen);
 	minlen = XFS_B_TO_FSB(mp, range.minlen);
 
-	/*
-	 * Truncating down the len isn't actually quite correct, but using
-	 * BBTOB would mean we trivially get overflows for values
-	 * of ULLONG_MAX or slightly lower.  And ULLONG_MAX is the default
-	 * used by the fstrim application.  In the end it really doesn't
-	 * matter as trimming blocks is an advisory interface.
-	 */
 	max_blocks = mp->m_sb.sb_dblocks + mp->m_sb.sb_rblocks;
 	if (range.start >= XFS_FSB_TO_B(mp, max_blocks) ||
 	    range.minlen > XFS_FSB_TO_B(mp, mp->m_ag_max_usable) ||
Signature · 1 line
2.43.0
#2 lore raw

PATCH 2/2 xfs: don't extend the FITRIM range if the rt device does not support discard

In reply to Christoph Hellwig

Fix the newly added discard support to only offer a FITRIM range that spans the RT device in addition to the main device if the RT device actually supports discard. Without this we'll incorrectly accept a larger range than actually supported and confuse user space if the RT device does not support discard. This can easily happen when the main device is a SSD but the RT device is a hard driver.

Move the code around a bit to keep the max_blocks and granularity assignments together and explicitly reject that case where only the RT device supports discard, as that does not fit the way the FITRIM ABI works very well and is a very fringe use case.

Fixes 3ba3ab1f6719 ("xfs: enable FITRIM on the realtime device")
---
 fs/xfs/xfs_discard.c | 50 +++++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 19 deletions(-)
fs/xfs/xfs_discard.c +31 -19
index d56efe9eae2cef..77d9d2b39f9b00 100644
@@ -685,27 +685,18 @@ xfs_ioc_trim(
 	struct xfs_mount		*mp,
 	struct fstrim_range __user	*urange)
 {
-	unsigned int		granularity =
-		bdev_discard_granularity(mp->m_ddev_targp->bt_bdev);
+	struct block_device	*bdev = mp->m_ddev_targp->bt_bdev;
 	struct block_device	*rt_bdev = NULL;
+	unsigned int		granularity = bdev_discard_granularity(bdev);
+	xfs_rfsblock_t		max_blocks = mp->m_sb.sb_dblocks;
 	struct fstrim_range	range;
 	xfs_daddr_t		start, end;
 	xfs_extlen_t		minlen;
-	xfs_rfsblock_t		max_blocks;
 	uint64_t		blocks_trimmed = 0;
 	int			error, last_error = 0;
 
 	if (!capable(CAP_SYS_ADMIN))
 		return -EPERM;
-	if (mp->m_rtdev_targp &&
-	    bdev_max_discard_sectors(mp->m_rtdev_targp->bt_bdev))
-		rt_bdev = mp->m_rtdev_targp->bt_bdev;
-	if (!bdev_max_discard_sectors(mp->m_ddev_targp->bt_bdev) && !rt_bdev)
-		return -EOPNOTSUPP;
-
-	if (rt_bdev)
-		granularity = max(granularity,
-				  bdev_discard_granularity(rt_bdev));
 
 	/*
 	 * We haven't recovered the log, so we cannot use our bnobt-guided
@@ -714,13 +705,36 @@ xfs_ioc_trim(
 	if (xfs_has_norecovery(mp))
 		return -EROFS;
 
+	/*
+	 * If the main device doesn't support discards, fail the entire ioctl
+	 * as the RT blocks are mapped after the main device blocks, and we'd
+	 * have to fake operation results for the main device for the ABI to
+	 * work.
+	 *
+	 * Note that while a main device that supports discards (SSD) and a RT
+	 * device that does not (HDD) is a fairly common setup, the reverse is
+	 * theoretically possible but rather odd, so this is not much of a loss.
+	 */
+	if (!bdev_max_discard_sectors(bdev))
+		return -EOPNOTSUPP;
+
+	if (mp->m_rtdev_targp) {
+		rt_bdev = mp->m_rtdev_targp->bt_bdev;
+		if (!bdev_max_discard_sectors(rt_bdev))
+			rt_bdev = NULL;
+	}
+	if (rt_bdev) {
+		max_blocks += mp->m_sb.sb_rblocks;
+		granularity =
+			max(granularity, bdev_discard_granularity(rt_bdev));
+	}
+
 	if (copy_from_user(&range, urange, sizeof(range)))
 		return -EFAULT;
 
 	range.minlen = max_t(u64, granularity, range.minlen);
 	minlen = XFS_B_TO_FSB(mp, range.minlen);
 
-	max_blocks = mp->m_sb.sb_dblocks + mp->m_sb.sb_rblocks;
 	if (range.start >= XFS_FSB_TO_B(mp, max_blocks) ||
 	    range.minlen > XFS_FSB_TO_B(mp, mp->m_ag_max_usable) ||
 	    range.len < mp->m_sb.sb_blocksize)
@@ -729,12 +743,10 @@ xfs_ioc_trim(
 	start = BTOBB(range.start);
 	end = start + BTOBBT(range.len) - 1;
 
-	if (bdev_max_discard_sectors(mp->m_ddev_targp->bt_bdev)) {
-		error = xfs_trim_datadev_extents(mp, start, end, minlen,
-				&blocks_trimmed);
-		if (error)
-			last_error = error;
-	}
+	error = xfs_trim_datadev_extents(mp, start, end, minlen,
+			&blocks_trimmed);
+	if (error)
+		last_error = error;
 
 	if (rt_bdev && !xfs_trim_should_stop()) {
 		error = xfs_trim_rtdev_extents(mp, start, end, minlen,
Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH 2/2] xfs: don't extend the FITRIM range if the rt device does not support discard

Quoted 11 lines · Christoph Hellwig

Fix the newly added discard support to only offer a FITRIM range that spans the RT device in addition to the main device if the RT device actually supports discard. Without this we'll incorrectly accept a larger range than actually supported and confuse user space if the RT device does not support discard. This can easily happen when the main device is a SSD but the RT device is a hard driver.

Move the code around a bit to keep the max_blocks and granularity assignments together and explicitly reject that case where only the RT device supports discard, as that does not fit the way the FITRIM ABI works very well and is a very fringe use case.

Is there more to this than generic/260 failing? And if not, does the following patch things up for you?

--D

From: Darrick J. Wong <[email protected]>
Subject: [PATCH] generic/260: fix for multi-device xfs with mixed discard support

Fix this test so that it can handle XFS filesystems with a realtime volume when the data and rt devices do not both support discard.

---
 common/rc  |   12 ++++++++++-
 common/xfs |   65 +++++++++++++++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 64 insertions(+), 13 deletions(-)
common/rc +11 -1
index b718030a59..426f2de43b 100644
@@ -4207,6 +4207,16 @@ _require_batched_discard()
 	fi
 }
 
+_bdev_queue_property()
+{
+	local dev="$1"
+	local property="$2"
+	local default="$3"
+
+	local fname="/sys/block/$(_short_dev "$dev")/queue/$property"
+	cat "$fname" 2>/dev/null || echo "$default"
+}
+
 # Given a mountpoint and the device associated with that mountpoint, return the
 # maximum start offset that the FITRIM command will accept, in units of 1024
 # byte blocks.
@@ -4214,7 +4224,7 @@ _discard_max_offset_kb()
 {
 	case "$FSTYP" in
 	xfs)
-		_xfs_discard_max_offset_kb "$1"
+		_xfs_discard_max_offset_kb "$@"
 		;;
 	*)
 		$DF_PROG -k | awk -v dev="$2" -v mnt="$1" '$1 == dev && $7 == mnt { print $3 }'
common/xfs +53 -12

91 lines not on this page · raw message

#4 lore raw

Re: [PATCH 2/2] xfs: don't extend the FITRIM range if the rt device does not support discard

Quoted 1 line · Darrick J. Wong

Is there more to this than generic/260 failing?

The only other users of the detection is generic/251, which doesn't seem to care about the exact value.

Quoted 2 lines

And if not, does the
following patch things up for you?

This works. OTOH it will break again with the zoned RT subvolume which can't support FITRIM even on devices that claim it. And for actual users that care (and not just xfstests) these kinds of hacks don't seem very palatable..

#5 lore raw

Re: [PATCH 2/2] xfs: don't extend the FITRIM range if the rt device does not support discard

Quoted 6 lines · Christoph Hellwig
Quoted 1 line · Darrick J. Wong

Is there more to this than generic/260 failing?

The only other users of the detection is generic/251, which doesn't seem to care about the exact value.

Quoted 7 lines
Quoted 2 lines

And if not, does the
following patch things up for you?

This works. OTOH it will break again with the zoned RT subvolume which can't support FITRIM even on devices that claim it. And for actual users that care (and not just xfstests) these kinds of hacks don't seem very palatable..

What does discard do on a zoned device? Is that how you reset the write pointer? And does that mean that either you tell the device to discard everything it's written in a zone, or it will do nothing?

I see that this test really just puts a lower bound on how much FITRIM can report that it discarded from an empty fs. That number is already rather meaningless on XFS because the amount "discarded" is roughly (free space - (amount of pending gc work + discard work already in progress)) and the user has no visibility into the amount of pending gc work. And you can repeatedly FITRIM and it reports the same number since we have no idea if the device actually *did* anything.

Hmm. No manpage for FITRIM. Why don't we return the number of bytes in the space map that we iterated as range.len? Or perhaps leave it unchanged?

--D

#6 lore raw

Re: [PATCH 2/2] xfs: don't extend the FITRIM range if the rt device does not support discard

Quoted 8 lines · Darrick J. Wong
Quoted 4 lines

This works. OTOH it will break again with the zoned RT subvolume which can't support FITRIM even on devices that claim it. And for actual users that care (and not just xfstests) these kinds of hacks don't seem very palatable..

What does discard do on a zoned device? Is that how you reset the write pointer? And does that mean that either you tell the device to discard everything it's written in a zone, or it will do nothing?

On an actual zone device it will probably do nothing. But at least for NVMe the command used to implement discard is mandatory, so all devices will show support. We also support the zoned mode on conventional devices, but instead of through FITRIM we want to issue it instad of a zone reset when the whole rtg has been garbage collected.

Quoted 3 lines

Hmm. No manpage for FITRIM. Why don't we return the number of bytes in the space map that we iterated as range.len? Or perhaps leave it unchanged?

The above would seem sensible. Not sure if we can still pull it off, though.

#7 lore raw

Re: [PATCH 2/2] xfs: don't extend the FITRIM range if the rt device does not support discard

Quoted 16 lines · Christoph Hellwig
Quoted 8 lines · Darrick J. Wong
Quoted 4 lines

This works. OTOH it will break again with the zoned RT subvolume which can't support FITRIM even on devices that claim it. And for actual users that care (and not just xfstests) these kinds of hacks don't seem very palatable..

What does discard do on a zoned device? Is that how you reset the write pointer? And does that mean that either you tell the device to discard everything it's written in a zone, or it will do nothing?

On an actual zone device it will probably do nothing. But at least for NVMe the command used to implement discard is mandatory, so all devices will show support. We also support the zoned mode on conventional devices, but instead of through FITRIM we want to issue it instad of a zone reset when the whole rtg has been garbage collected.

Quoted 6 lines
Quoted 3 lines

Hmm. No manpage for FITRIM. Why don't we return the number of bytes in the space map that we iterated as range.len? Or perhaps leave it unchanged?

The above would seem sensible. Not sure if we can still pull it off, though.

It seems to have survived testing on TOT overnight, so I'll bake it into djwong-dev when I go through and remove the rtgroups/rtsb feature bits today. And I guess the rtgroups xarray conversion too.

--D

#8 lore raw

Re: [PATCH 2/2] xfs: don't extend the FITRIM range if the rt device does not support discard

Quoted 2 lines · Darrick J. Wong

It seems to have survived testing on TOT overnight, so I'll bake it into djwong-dev

Note that this a fix for the new RT discard code in 6.11 and it would be kinda nice to get the fix into the 6.11 tree so that we don't have a release with the current state.