update BDI {io,ra}_pages values based on the RT device limits
Subsystems (2)
Hi all,
we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the io_pages value that can't be inspected or tuned from userspace.
This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:
1) support multiple BDIs per file system.
This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases
2) stop propagating these values through the BDI
Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.
Thoughts?
PATCH 1/1 xfs: update BDI {io,ra}_pages values based on the RT device limits
When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.
Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the ra_pages value).
--- fs/xfs/xfs_super.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
fs/xfs/xfs_super.c +24 -0
@@ -546,6 +546,29 @@ xfs_open_devices(return error; } +/* +* When using a RT device some or all data I/O is using the RT device, but +* the BDI is inherited from the main data device. When the underlying block +* device for the RT device has larger I/O sizes, the BDI settings might be +* incorrect, which is especially bad if the main device is a SSD and the +* RT device is a HDD, as the io_opt fixup in blk_apply_bdi_limits is missing +* for this case. +* +* Update the BDI values to the max of the data and RT device to cover our +* bases. +*/ +static void +xfs_update_bdi_rahead ( +struct xfs_mount *mp) +{ +struct backing_dev_info *rt_bdi = + mp->m_rtdev_targp->bt_bdev->bd_disk->bdi; +struct backing_dev_info *sb_bdi = mp->m_super->s_bdi; + + sb_bdi->ra_pages =max (sb_bdi->ra_pages, rt_bdi->ra_pages); + sb_bdi->io_pages =max (sb_bdi->io_pages, rt_bdi->io_pages); +} +/* * Setup xfs_mount buffer target pointers based on superblock */
@@ -583,6 +606,7 @@ xfs_setup_devices( mp->m_sb.sb_sectsize, mp->m_sb.sb_rblocks);if (error)return error; +xfs_update_bdi_rahead (mp); }return 0 ;
Signature · 1 line
2.53.0
Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits
Quoted 5 lines · Christoph Hellwig
When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.
Quoted 8 lines
Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the
ra_pagesvalue).Reported-by Filip Blagojevic <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>
Until we have a better API to do this, given the performance improvements this gives with HDDs, I am all for it.
Signature · 2 lines
Damien Le Moal Western Digital Research
Carlos Maiolino
maintainerRe: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits
In reply to Christoph HellwigQuoted 10 lines · Christoph Hellwig
When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.
Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the
ra_pagesvalue).
Sorry a 'very late reply', but...
On a second thought, wouldn't be wise to cache the original value in memory and restore it during unmount/filesystem shutdown?
I could send a patch to complement this one if you guys agree, otherwise just ignore me.
Quoted 52 lines
Reported-by Filip Blagojevic <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>--- fs/xfs/xfs_super.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)fs/xfs/xfs_super.c +24
-0@@ -546,6 +546,29 @@ xfs_open_devices(return error; } +/* +* When using a RT device some or all data I/O is using the RT device, but +* the BDI is inherited from the main data device. When the underlying block +* device for the RT device has larger I/O sizes, the BDI settings might be +* incorrect, which is especially bad if the main device is a SSD and the +* RT device is a HDD, as the io_opt fixup in blk_apply_bdi_limits is missing +* for this case. +* +* Update the BDI values to the max of the data and RT device to cover our +* bases. +*/ +static void +xfs_update_bdi_rahead ( +struct xfs_mount *mp) +{ +struct backing_dev_info *rt_bdi = + mp->m_rtdev_targp->bt_bdev->bd_disk->bdi; +struct backing_dev_info *sb_bdi = mp->m_super->s_bdi; + + sb_bdi->ra_pages =max (sb_bdi->ra_pages, rt_bdi->ra_pages); + sb_bdi->io_pages =max (sb_bdi->io_pages, rt_bdi->io_pages); +} +/* * Setup xfs_mount buffer target pointers based on superblock */ @@ -583,6 +606,7 @@ xfs_setup_devices( mp->m_sb.sb_sectsize, mp->m_sb.sb_rblocks);if (error)return error; +xfs_update_bdi_rahead (mp); }return 0 ;Signature · 1 line
2.53.0
Re: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits
Quoted 12 lines · Carlos Maiolino
Quoted 10 lines · Christoph Hellwig
When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.
Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the
ra_pagesvalue).
Quoted 6 lines
Sorry a 'very late reply', but...
On a second thought, wouldn't be wise to cache the original value in memory and restore it during unmount/filesystem shutdown?I could send a patch to complement this one if you guys agree, otherwise just ignore me.
As mentioned this does seem like a valid option, there's tradeoff both ways. Although we'd also want a flag to avoid restoring them if they were changed while mounted so it would get a bit more complicated. Happy to respin it for that if there is a general preference.
Carlos Maiolino
maintainerRe: [PATCH] xfs: update BDI {io,ra}_pages values based on the RT device limits
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Carlos Maiolino
Quoted 10 lines · Christoph Hellwig
When using XFS with a main device on an SSD that stores metadata and a RT device to store data on a HDD, we fail to take the I/O sizes for the RT device into accounting, leading to up to 5% slower read performance when using an SSD for metadata vs storing data and metadata on the HDD.
Fix this up by taking the RT settings into account. Note that this updates the BDI owned by the main device, and leaves those settings in place even when the file system is unmounted. This is a bit unexpected but not different from manual tuning through sysfs (although that is only possible for the
ra_pagesvalue).Sorry a 'very late reply', but...
On a second thought, wouldn't be wise to cache the original value in memory and restore it during unmount/filesystem shutdown?
Quoted 9 lines
Quoted 2 lines
I could send a patch to complement this one if you guys agree, otherwise just ignore me.
As mentioned this does seem like a valid option, there's tradeoff both ways. Although we'd also want a flag to avoid restoring them if they were changed while mounted so it would get a bit more complicated. Happy to respin it for that if there is a general preference.
Sorry coming back late to it, I don't want to give you extra work after we've already discussed the implementation.
I was going to merge this patch today, but it came to my mind that might not be a good idea to just quietly update BDI settings while XFS is using it and just keep whatever change we made persist after the mount.
I can still merge it this week and we decide what to do later or just wait a bit for extra thoughts on this.
Perhaps let's see if we get other comments soon and we make a decision later this week.
Carlos Maiolino
maintainerQuoted 13 lines · Christoph Hellwig
Hi all,
we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the
io_pagesvalue that can't be inspected or tuned from userspace.This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:
Quoted 6 lines
1) support multiple BDIs per file system.
This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases
Giving the current tendency of filesystems being multi-device, this doesn't sound bad IMHO. Wouldn't accessing io_pages of each BDI also be worth even for a journal dev? I wonder if what you ran into wouldn't be possible if somebody would be using just a SSD for journal and a non-rt XFS on a HDD or a different/slower device.
I don't know how stupid that sounds but perhaps it wouldn't be that complicated to support multiple BDIs without making it unpleasant for filesystems that don't care?sb->bdi could be turned into a dynamic array and a new sb->s_devcount fields to keep track of it on multi-device filesystems. Filesystems who don't care about multiple BDIs would have it pointing to a single BDI struct. Again I feel I'm missing something, so it might sound really stupid :)
Quoted 8 lines
2) stop propagating these values through the BDI
Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.
It doesn't sound bad, but to me, not better than keeping multiple BDIs. But my lack of knowledge on details about BDI handling might be hiding something from me.
Quoted 3 lines
Thoughts?
Quoted 5 lines · Carlos Maiolino
Giving the current tendency of filesystems being multi-device, this doesn't sound bad IMHO. Wouldn't accessing
io_pagesof each BDI also be worth even for a journal dev? I wonder if what you ran into wouldn't be possible if somebody would be using just a SSD for journal and a non-rt XFS on a HDD or a different/slower device.
Nothing looks at the value for the log device.
Quoted 9 lines
I don't know how stupid that sounds but perhaps it wouldn't be that complicated to support multiple BDIs without making it unpleasant for filesystems that don't care?
sb->bdicould be turned into a dynamic array and a newsb->s_devcountfields to keep track of it on multi-device filesystems. Filesystems who don't care about multiple BDIs would have it pointing to a single BDI struct. Again I feel I'm missing something, so it might sound really stupid :)
This will get complicated really soon..
Jan Kara
reviewerHi Christoph!
Quoted 9 lines · Christoph Hellwig
we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the
io_pagesvalue that can't be inspected or tuned from userspace.This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:
A cleaner but still relatively simple way how to fix your problem would be to do something similar like e.g. btrfs does. That is when you have a filesystem with multiple backing devices, you create a new BDI (using super_setup_bdi()) and set its parameters based on the combination of devices you have.
The only downside to this is that blk-wbt has a layering violation in it when it tries to guess whether there are task dirty-throttled for the device which stops working for when the filesystem uses this synthetic bdi. But I don't think that's too serious to worry about.
Quoted 6 lines
1) support multiple BDIs per file system.
This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases
Since looking up the bdi is abstracted through inode_to_bdi() (which currently goes through the superblock), this would be relatively easy to do but either the inode would have to store the bdi pointer or we'd have to have an operation to query it. Just at this point I'm not fully convinced the additional complexity / memory cost is worth it compared to what can be achieved with a synthetic bdi.
Quoted 7 lines
2) stop propagating these values through the BDI
Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.
Similarly as above. It makes sense but I'm not sure there are strong enough usecases to justify this.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 10 lines · Jan Kara
A cleaner but still relatively simple way how to fix your problem would be to do something similar like e.g. btrfs does. That is when you have a filesystem with multiple backing devices, you create a new BDI (using
super_setup_bdi()) and set its parameters based on the combination of devices you have.The only downside to this is that blk-wbt has a layering violation in it when it tries to guess whether there are task dirty-throttled for the device which stops working for when the filesystem uses this synthetic bdi. But I don't think that's too serious to worry about.
The big downside of that is that it breaks the userspace ABI - any previous changes to BDI paramters, most importantly the read-ahead size, would now only apply to the BDI for s_bdev which isn't actually used by anything.
And yeah, the blk-wbt thing actually looks really broken and could use fixing..
Quoted 7 lines
Quoted 6 lines
1) support multiple BDIs per file system.
This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases
Quoted 6 lines
Since looking up the bdi is abstracted through
inode_to_bdi()(which currently goes through the superblock), this would be relatively easy to do but either the inode would have to store the bdi pointer or we'd have to have an operation to query it. Just at this point I'm not fully convinced the additional complexity / memory cost is worth it compared to what can be achieved with a synthetic bdi.
Same.
Quoted 10 lines
Quoted 7 lines
2) stop propagating these values through the BDI
Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.
Similarly as above. It makes sense but I'm not sure there are strong enough usecases to justify this.
It would at least be a lot less complete than option 1. And get us out of a bit more of all that bdi weirdness.
Jan Kara
reviewerQuoted 11 lines · Christoph Hellwig
Quoted 5 lines · Jan Kara
A cleaner but still relatively simple way how to fix your problem would be to do something similar like e.g. btrfs does. That is when you have a filesystem with multiple backing devices, you create a new BDI (using
super_setup_bdi()) and set its parameters based on the combination of devices you have.The big downside of that is that it breaks the userspace ABI - any previous changes to BDI paramters, most importantly the read-ahead size, would now only apply to the BDI for
s_bdevwhich isn't actually used by anything.
Hum, right. XFS users are used to tweak /sys/block/<dev>/queue/read_ahead_kb
which would stop influencing anything. I guess that could indeed break some people's expectations.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
1) support multiple BDIs per file system.
This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases
Quoted 8 lines
Quoted 6 lines
Since looking up the bdi is abstracted through
inode_to_bdi()(which currently goes through the superblock), this would be relatively easy to do but either the inode would have to store the bdi pointer or we'd have to have an operation to query it. Just at this point I'm not fully convinced the additional complexity / memory cost is worth it compared to what can be achieved with a synthetic bdi.Same.
One more idea: We already cache bdi's ra_pages in struct file_ra_state on file open. It wouldn't be that hard to allow filesystem to override the values in its open method (currently it isn't possible only because file_ra_state_init() is called after ->open). For io_pages we currently always go to the bdi so we'd have to add it to struct file_ra_state.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 4 lines · Jan Kara
Hum, right. XFS users are used to tweak /sys/block/<dev>/queue/read_ahead_kb
which would stop influencing anything. I guess that could indeed break some people's expectations.
Yeah.
Quoted 5 lines
One more idea: We already cache bdi's
ra_pagesin structfile_ra_stateon file open. It wouldn't be that hard to allow filesystem to override the values in its open method (currently it isn't possible only becausefile_ra_state_init()is called after ->open). Forio_pageswe currently always go to the bdi so we'd have to add it to structfile_ra_state.
It has to be called after ->open because ->open can change file->f_mapping. Now we could require any instance changing it to also call file_ra_state_init, but I think bloating every struct file with a new field might not win a lot of friends. OTOH having a local copy of ra_pages but not io_pages is really weird. But so is this whole cascading set of indirections for the readahead parameters.
Quoted 5 lines
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
---end quoted text---
So, I guess we're stuck with this approach? Or any other ideas?
Quoted 22 lines · Christoph Hellwig
Hi all,
we ran into a bit of a funny case where adding an SSD to store metadata to a (zoned) XFS file system reduced the performance vs using a HDD for both data and metadata. It turns out this is due to readahead code looking at the BDI limits, including the
io_pagesvalue that can't be inspected or tuned from userspace.This patch has the simplest fix for that by just updating the values from XFS, but for a long-term solution this feels a bit ugly. Other, a lot more invasive options would be:
1) support multiple BDIs per file system.
This would work pretty well for XFS, as data for a given file is always entirely on one device, and the VFS writeback code is not used for metadata. But it probably doesn't work well for other cases
2) stop propagating these values through the BDI
Quoted 7 lines
Have a way to query these parameters from the file system, either through a method if we want to be fully dynamic, or through fields instead of going through the BDI. The downside would be that sysfs modifications of the readahead size would not work after the file system initially queried them.
Thoughts?
---end quoted text---