Skip to contentSkip to conversation

update BDI {io,ra}_pages values based on the RT device limits v2

PATCH Applied Christoph Hellwig 4 participants 8 messages Jul 20 – Jul 21

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the io_pages value that can't be inspected or tuned from userspace.

Changes since v1:
- restore the old value at unmount, unless the BDI values changed since mount

#1 lore raw

PATCH 1/1 xfs: update BDI {io,ra}_pages values based on the RT device limits

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account at mount an restoring the old settings at unmount time, unless the BDI settings have changed from those set by XFS.

---
 fs/xfs/xfs_mount.h |  7 ++++++
 fs/xfs/xfs_super.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 59 insertions(+), 1 deletion(-)
fs/xfs/xfs_mount.h +7 -0
index 66a02d1b9ad7..216a38a354e7 100644
@@ -349,6 +349,13 @@ typedef struct xfs_mount {
 
 	/* Index of uuid record in the uuid xarray. */
 	unsigned int		m_uuid_table_index;
+
+	/*
+	 * Old io_pages/ra_pages valued in the main bdev BDI, and our initial
+	 * calculated values.
+	 */
+	unsigned long		m_old_io_pages, m_initial_io_pages;
+	unsigned long		m_old_ra_pages, m_initial_ra_pages;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_super.c +52 -1

72 lines not on this page · raw message

Signature · 1 line
2.53.0
Reviewed-by on v1 1/1
#2 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 11 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account at mount an restoring the old settings at unmount time, unless the BDI settings have changed from those set by XFS.

[...]

Quoted 9 lines

+/*
+ * When using a RT device some or all data I/O is using the RT device, but
+ * the BDI is inherited from the main data device. When the underlying block
+ * device for the RT device has larger I/O sizes, the BDI settings might be
+ * incorrect, which is especially bad if the main device is a SSD and the
+ * RT device is a HDD, as the io_opt fixup in blk_apply_bdi_limits is missing
+ * for this case.
+ *
+ * Update the BDI values to the max of the data and RT device to cover our

s/data/main ?
I am still confused about the proper name for the device holding metadata :)

Other than this, this looks OK to me.

Signature · 2 lines
Damien Le Moal
Western Digital Research
#3 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 6 lines · Damien Le Moal
Quoted 1 line

+ * Update the BDI values to the max of the data and RT device to cover our

s/data/main ?
I am still confused about the proper name for the device holding metadata :)

Other than this, this looks OK to me.

Historically the main device is called the data device, which is also reflected in all kinds of field naming. But yes, it is confusing especially for zoned xfs setups. Maybe I should do a little global naming sweep..

maintainer
#4 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Damien Le Moal
Quoted 1 line

+ * Update the BDI values to the max of the data and RT device to cover our

s/data/main ?
I am still confused about the proper name for the device holding metadata :)

Quoted 8 lines
Quoted 1 line

Other than this, this looks OK to me.

Historically the main device is called the data device, which is also reflected in all kinds of field naming. But yes, it is confusing especially for zoned xfs setups. Maybe I should do a little global naming sweep..

+1, please keep /data/ here, this makes much more sense in xfs lingo... Perhaps we should start naming them to something like ndata (for normal data) and zdata for zoned data, but that's talk for another thread.

#5 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 16 lines · Carlos Maiolino
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Damien Le Moal
Quoted 1 line

+ * Update the BDI values to the max of the data and RT device to cover our

s/data/main ?
I am still confused about the proper name for the device holding metadata :)

Quoted 8 lines
Quoted 1 line

Other than this, this looks OK to me.

Historically the main device is called the data device, which is also reflected in all kinds of field naming. But yes, it is confusing especially for zoned xfs setups. Maybe I should do a little global naming sweep..

Quoted 3 lines

+1, please keep /data/ here, this makes much more sense in xfs lingo... Perhaps we should start naming them to something like ndata (for normal data) and zdata for zoned data, but that's talk for another thread.

I've been trying to be consistent at calling them the 'datadev' and 'rtdev' (no spaces) so it's more obvious that I'm using a jargon. But 'rt' makes no sense these days and I never did come up with a good retcon for it though.

"RelaTive"? Because the rtdev is "related" to the datadev by metadata?

:D

--D

maintainer Reviewed-by on v1 1/1
#6 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account at mount an restoring the old settings at unmount time, unless the BDI settings have changed from those set by XFS.

Quoted 106 lines
---
 fs/xfs/xfs_mount.h |  7 ++++++
 fs/xfs/xfs_super.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 59 insertions(+), 1 deletion(-)
fs/xfs/xfs_mount.h +7 -0
index 66a02d1b9ad7..216a38a354e7 100644
@@ -349,6 +349,13 @@ typedef struct xfs_mount {
 
 	/* Index of uuid record in the uuid xarray. */
 	unsigned int		m_uuid_table_index;
+
+	/*
+	 * Old io_pages/ra_pages valued in the main bdev BDI, and our initial
+	 * calculated values.
+	 */
+	unsigned long		m_old_io_pages, m_initial_io_pages;
+	unsigned long		m_old_ra_pages, m_initial_ra_pages;
 } xfs_mount_t;
 
 #define M_IGEO(mp)		(&(mp)->m_ino_geo)
fs/xfs/xfs_super.c +52 -1
index 8531d526fc44..63c4bcbe6c2b 100644
@@ -548,6 +548,52 @@ xfs_open_devices(
 	return error;
 }
 
+/*
+ * When using a RT device some or all data I/O is using the RT device, but
+ * the BDI is inherited from the main data device.  When the underlying block
+ * device for the RT device has larger I/O sizes, the BDI settings might be
+ * incorrect, which is especially bad if the main device is a SSD and the
+ * RT device is a HDD, as the io_opt fixup in blk_apply_bdi_limits is missing
+ * for this case.
+ *
+ * Update the BDI values to the max of the data and RT device to cover our
+ * bases.
+ */
+static void
+xfs_update_bdi_rahead(
+	struct xfs_mount	*mp)
+{
+	struct backing_dev_info	*rt_bdi =
+		mp->m_rtdev_targp->bt_bdev->bd_disk->bdi;
+	struct backing_dev_info	*sb_bdi = mp->m_super->s_bdi;
+
+	mp->m_old_io_pages = sb_bdi->io_pages;
+	mp->m_old_ra_pages = sb_bdi->ra_pages;
+
+	sb_bdi->io_pages = mp->m_initial_io_pages =
+		max(sb_bdi->io_pages, rt_bdi->io_pages);
+	sb_bdi->ra_pages = mp->m_initial_ra_pages =
+		max(sb_bdi->ra_pages, rt_bdi->ra_pages);
+}
+
+static void
+xfs_restore_bdi_rahead(
+	struct xfs_mount	*mp)
+{
+	struct backing_dev_info	*sb_bdi = mp->m_super->s_bdi;
+
+	if (sb_bdi->io_pages == mp->m_initial_io_pages)
+		sb_bdi->io_pages = mp->m_old_io_pages;
+	else
+		xfs_info(mp, "io_pages changed from %lu to %lu, not restoring.",
+				mp->m_initial_io_pages, sb_bdi->io_pages);
+	if (sb_bdi->ra_pages == mp->m_initial_ra_pages)
+		sb_bdi->ra_pages = mp->m_old_ra_pages;
+	else
+		xfs_info(mp, "ra_pages changed from %lu to %lu, not restoring.",
+				mp->m_initial_ra_pages, sb_bdi->ra_pages);
+}
+
 /*
  * Setup xfs_mount buffer target pointers based on superblock
  */
@@ -585,6 +631,7 @@ xfs_setup_devices(
 				mp->m_sb.sb_sectsize, mp->m_sb.sb_rblocks);
 		if (error)
 			return error;
+		xfs_update_bdi_rahead(mp);
 	}
 
 	return 0;
@@ -2283,8 +2330,12 @@ static void
 xfs_kill_sb(
 	struct super_block		*sb)
 {
+	struct xfs_mount		*mp = XFS_M(sb);
+
+	if (mp->m_rtdev_targp && mp->m_rtdev_targp != mp->m_ddev_targp)
+		xfs_restore_bdi_rahead(mp);
 	kill_block_super(sb);
-	xfs_mount_free(XFS_M(sb));
+	xfs_mount_free(mp);
 }
 
 static struct file_system_type xfs_fs_type = {
Signature · 1 line
2.53.0
maintainer Applied
#7 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

In reply to Christoph Hellwig
Quoted 10 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account at mount an restoring the old settings at unmount time, unless the BDI settings have changed from those set by XFS.

[...]

Applied to for-next, thanks!

[1/1] xfs: update BDI {io,ra}_pages values based on the RT device limits

      commit: 39de8c9f3e4bc059bb0bc85a39affcf52664c1ed

Best regards,

Signature · 1 line
Carlos Maiolino <[email protected]>