Skip to contentSkip to conversation

update BDI {io,ra}_pages values based on the RT device limits

PATCH In review Christoph Hellwig 4 participants 13 messages Jun 23 – Jul 20

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the io_pages value that can't be inspected or tuned from userspace.

This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:

1) support multiple BDIs per file system.

This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases

2) stop propagating these values through the BDI

Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.

Thoughts?

#1 lore raw

PATCH 1/1 xfs: update BDI {io,ra}_pages values based on the RT device limits

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the ra_pages value).

---
 fs/xfs/xfs_super.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
fs/xfs/xfs_super.c +24 -0
index e6781f86d867..88edd3822872 100644
@@ -546,6 +546,29 @@ xfs_open_devices(
 	return error;
 }
 
+/*
+ * When using a RT device some or all data I/O is using the RT device, but
+ * the BDI is inherited from the main data device.  When the underlying block
+ * device for the RT device has larger I/O sizes, the BDI settings might be
+ * incorrect, which is especially bad if the main device is a SSD and the
+ * RT device is a HDD, as the io_opt fixup in blk_apply_bdi_limits is missing
+ * for this case.
+ *
+ * Update the BDI values to the max of the data and RT device to cover our
+ * bases.
+ */
+static void
+xfs_update_bdi_rahead(
+	struct xfs_mount	*mp)
+{
+	struct backing_dev_info	*rt_bdi =
+		mp->m_rtdev_targp->bt_bdev->bd_disk->bdi;
+	struct backing_dev_info	*sb_bdi = mp->m_super->s_bdi;
+
+	sb_bdi->ra_pages = max(sb_bdi->ra_pages, rt_bdi->ra_pages);
+	sb_bdi->io_pages = max(sb_bdi->io_pages, rt_bdi->io_pages);
+}
+
 /*
  * Setup xfs_mount buffer target pointers based on superblock
  */
@@ -583,6 +606,7 @@ xfs_setup_devices(
 				mp->m_sb.sb_sectsize, mp->m_sb.sb_rblocks);
 		if (error)
 			return error;
+		xfs_update_bdi_rahead(mp);
 	}
 
 	return 0;
Signature · 1 line
2.53.0
Reviewed-by on v1 1/1
#2 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 5 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Quoted 8 lines

Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the ra_pages value).

Until we have a better API to do this, given the performance improvements this gives with HDDs, I am all for it.

Signature · 2 lines
Damien Le Moal
Western Digital Research
maintainer
#3 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

In reply to Christoph Hellwig
Quoted 10 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the ra_pages value).

Sorry a 'very late reply', but...
On a second thought, wouldn't be wise to cache the original value in memory and restore it during unmount/filesystem shutdown?

I could send a patch to complement this one if you guys agree, otherwise just ignore me.

Quoted 52 lines
---
 fs/xfs/xfs_super.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
fs/xfs/xfs_super.c +24 -0
index e6781f86d867..88edd3822872 100644
@@ -546,6 +546,29 @@ xfs_open_devices(
 	return error;
 }
 
+/*
+ * When using a RT device some or all data I/O is using the RT device, but
+ * the BDI is inherited from the main data device.  When the underlying block
+ * device for the RT device has larger I/O sizes, the BDI settings might be
+ * incorrect, which is especially bad if the main device is a SSD and the
+ * RT device is a HDD, as the io_opt fixup in blk_apply_bdi_limits is missing
+ * for this case.
+ *
+ * Update the BDI values to the max of the data and RT device to cover our
+ * bases.
+ */
+static void
+xfs_update_bdi_rahead(
+	struct xfs_mount	*mp)
+{
+	struct backing_dev_info	*rt_bdi =
+		mp->m_rtdev_targp->bt_bdev->bd_disk->bdi;
+	struct backing_dev_info	*sb_bdi = mp->m_super->s_bdi;
+
+	sb_bdi->ra_pages = max(sb_bdi->ra_pages, rt_bdi->ra_pages);
+	sb_bdi->io_pages = max(sb_bdi->io_pages, rt_bdi->io_pages);
+}
+
 /*
  * Setup xfs_mount buffer target pointers based on superblock
  */
@@ -583,6 +606,7 @@ xfs_setup_devices(
 				mp->m_sb.sb_sectsize, mp->m_sb.sb_rblocks);
 		if (error)
 			return error;
+		xfs_update_bdi_rahead(mp);
 	}
 
 	return 0;
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 12 lines · Carlos Maiolino
Quoted 10 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the ra_pages value).

Quoted 6 lines

Sorry a 'very late reply', but...
On a second thought, wouldn't be wise to cache the original value in memory and restore it during unmount/filesystem shutdown?

I could send a patch to complement this one if you guys agree, otherwise just ignore me.

As mentioned this does seem like a valid option, there's tradeoff both ways. Although we'd also want a flag to avoid restoring them if they were changed while mounted so it would get a bit more complicated. Happy to respin it for that if there is a general preference.

maintainer
#5 lore raw

Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits

Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Carlos Maiolino
Quoted 10 lines · Christoph Hellwig

When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.

Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the ra_pages value).

Sorry a 'very late reply', but...
On a second thought, wouldn't be wise to cache the original value in memory and restore it during unmount/filesystem shutdown?

Quoted 9 lines
Quoted 2 lines

I could send a patch to complement this one if you guys agree, otherwise just ignore me.

As mentioned this does seem like a valid option, there's tradeoff both ways. Although we'd also want a flag to avoid restoring them if they were changed while mounted so it would get a bit more complicated. Happy to respin it for that if there is a general preference.

Sorry coming back late to it, I don't want to give you extra work after we've already discussed the implementation.

I was going to merge this patch today, but it came to my mind that might not be a good idea to just quietly update BDI settings while XFS is using it and just keep whatever change we made persist after the mount.

I can still merge it this week and we decide what to do later or just wait a bit for extra thoughts on this.
Perhaps let's see if we get other comments soon and we make a decision later this week.

maintainer
#6 lore raw
In reply to Christoph Hellwig
Quoted 13 lines · Christoph Hellwig

Hi all,

we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the io_pages value that can't be inspected or tuned from userspace.

This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:

Quoted 6 lines

1) support multiple BDIs per file system.

This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases

Giving the current tendency of filesystems being multi-device, this doesn't sound bad IMHO. Wouldn't accessing io_pages of each BDI also be worth even for a journal dev? I wonder if what you ran into wouldn't be possible if somebody would be using just a SSD for journal and a non-rt XFS on a HDD or a different/slower device.

I don't know how stupid that sounds but perhaps it wouldn't be that complicated to support multiple BDIs without making it unpleasant for filesystems that don't care?
sb->bdi could be turned into a dynamic array and a new sb->s_devcount fields to keep track of it on multi-device filesystems. Filesystems who don't care about multiple BDIs would have it pointing to a single BDI struct. Again I feel I'm missing something, so it might sound really stupid :)

Quoted 8 lines

2) stop propagating these values through the BDI

Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.

It doesn't sound bad, but to me, not better than keeping multiple BDIs. But my lack of knowledge on details about BDI handling might be hiding something from me.

Quoted 3 lines

Thoughts?

#7 lore raw
Quoted 5 lines · Carlos Maiolino

Giving the current tendency of filesystems being multi-device, this doesn't sound bad IMHO. Wouldn't accessing io_pages of each BDI also be worth even for a journal dev? I wonder if what you ran into wouldn't be possible if somebody would be using just a SSD for journal and a non-rt XFS on a HDD or a different/slower device.

Nothing looks at the value for the log device.

Quoted 9 lines

I don't know how stupid that sounds but perhaps it wouldn't be that complicated to support multiple BDIs without making it unpleasant for filesystems that don't care?
sb->bdi could be turned into a dynamic array and a new sb->s_devcount fields to keep track of it on multi-device filesystems. Filesystems who don't care about multiple BDIs would have it pointing to a single BDI struct. Again I feel I'm missing something, so it might sound really stupid :)

This will get complicated really soon..

reviewer
#8 lore raw
In reply to Christoph Hellwig

Hi Christoph!

Quoted 9 lines · Christoph Hellwig

we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the io_pages value that can't be inspected or tuned from userspace.

This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:

A cleaner but still relatively simple way how to fix your problem would be to do something similar like e.g. btrfs does. That is when you have a filesystem with multiple backing devices, you create a new BDI (using super_setup_bdi()) and set its parameters based on the combination of devices you have.

The only downside to this is that blk-wbt has a layering violation in it when it tries to guess whether there are task dirty-throttled for the device which stops working for when the filesystem uses this synthetic bdi. But I don't think that's too serious to worry about.

Quoted 6 lines

1) support multiple BDIs per file system.

This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases

Since looking up the bdi is abstracted through inode_to_bdi() (which currently goes through the superblock), this would be relatively easy to do but either the inode would have to store the bdi pointer or we'd have to have an operation to query it. Just at this point I'm not fully convinced the additional complexity / memory cost is worth it compared to what can be achieved with a synthetic bdi.

Quoted 7 lines

2) stop propagating these values through the BDI

Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.

Similarly as above. It makes sense but I'm not sure there are strong enough usecases to justify this.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
Quoted 10 lines · Jan Kara

A cleaner but still relatively simple way how to fix your problem would be to do something similar like e.g. btrfs does. That is when you have a filesystem with multiple backing devices, you create a new BDI (using super_setup_bdi()) and set its parameters based on the combination of devices you have.

The only downside to this is that blk-wbt has a layering violation in it when it tries to guess whether there are task dirty-throttled for the device which stops working for when the filesystem uses this synthetic bdi. But I don't think that's too serious to worry about.

The big downside of that is that it breaks the userspace ABI - any previous changes to BDI paramters, most importantly the read-ahead size, would now only apply to the BDI for s_bdev which isn't actually used by anything.

And yeah, the blk-wbt thing actually looks really broken and could use fixing..

Quoted 7 lines
Quoted 6 lines

1) support multiple BDIs per file system.

This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases

Quoted 6 lines

Since looking up the bdi is abstracted through inode_to_bdi() (which currently goes through the superblock), this would be relatively easy to do but either the inode would have to store the bdi pointer or we'd have to have an operation to query it. Just at this point I'm not fully convinced the additional complexity / memory cost is worth it compared to what can be achieved with a synthetic bdi.

Same.

Quoted 10 lines
Quoted 7 lines

2) stop propagating these values through the BDI

Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.

Similarly as above. It makes sense but I'm not sure there are strong enough usecases to justify this.

It would at least be a lot less complete than option 1. And get us out of a bit more of all that bdi weirdness.

reviewer
#10 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Jan Kara

A cleaner but still relatively simple way how to fix your problem would be to do something similar like e.g. btrfs does. That is when you have a filesystem with multiple backing devices, you create a new BDI (using super_setup_bdi()) and set its parameters based on the combination of devices you have.

The big downside of that is that it breaks the userspace ABI - any previous changes to BDI paramters, most importantly the read-ahead size, would now only apply to the BDI for s_bdev which isn't actually used by anything.

Hum, right. XFS users are used to tweak /sys/block/<dev>/queue/read_ahead_kb
which would stop influencing anything. I guess that could indeed break some people's expectations.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

1) support multiple BDIs per file system.

This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases

Quoted 8 lines
Quoted 6 lines

Since looking up the bdi is abstracted through inode_to_bdi() (which currently goes through the superblock), this would be relatively easy to do but either the inode would have to store the bdi pointer or we'd have to have an operation to query it. Just at this point I'm not fully convinced the additional complexity / memory cost is worth it compared to what can be achieved with a synthetic bdi.

Same.

One more idea: We already cache bdi's ra_pages in struct file_ra_state on file open. It wouldn't be that hard to allow filesystem to override the values in its open method (currently it isn't possible only because file_ra_state_init() is called after ->open). For io_pages we currently always go to the bdi so we'd have to add it to struct file_ra_state.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw
Quoted 4 lines · Jan Kara

Hum, right. XFS users are used to tweak /sys/block/<dev>/queue/read_ahead_kb
which would stop influencing anything. I guess that could indeed break some people's expectations.

Yeah.

Quoted 5 lines

One more idea: We already cache bdi's ra_pages in struct file_ra_state on file open. It wouldn't be that hard to allow filesystem to override the values in its open method (currently it isn't possible only because file_ra_state_init() is called after ->open). For io_pages we currently always go to the bdi so we'd have to add it to struct file_ra_state.

It has to be called after ->open because ->open can change file->f_mapping. Now we could require any instance changing it to also call file_ra_state_init, but I think bloating every struct file with a new field might not win a lot of friends. OTOH having a local copy of ra_pages but not io_pages is really weird. But so is this whole cascading set of indirections for the readahead parameters.

Quoted 5 lines

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR

---end quoted text---

#12 lore raw
In reply to Christoph Hellwig

So, I guess we're stuck with this approach? Or any other ideas?

Quoted 22 lines · Christoph Hellwig

Hi all,

we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the io_pages value that can't be inspected or tuned from userspace.

This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:

1) support multiple BDIs per file system.

This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases

2) stop propagating these values through the BDI

Quoted 7 lines

Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.

Thoughts?

---end quoted text---