questions about hybird xfs wih ssd/hdd by realtime subvol
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
There are some questions about how to control the data to save into normal vol or realtime subvol firstly.
1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?
2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?
3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
Best Regards
Wang Yugui ([email protected])
2022/08/29
Carlos Maiolino
maintainerQuoted 7 lines · Wang Yugui
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
Quoted 7 lines
There are some questions about how to control the data to save into normal vol or realtime subvol firstly.
1, man xfsctl
here isXFS_XFLAG_REALTIMEin man xfsctl of xfsprogs 5.0 , but there is noXFS_XFLAG_REALTIMEin xfsprogs 5.14/5.19.xfsctl(XFS_XFLAG_REALTIME)will be removed in the further?
It's been a while since XFS uses FS_XFLAG features directly, so, what you're specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:
#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME
FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.
Quoted 2 lines
2, Is there some tool to do
xfsctl(XFS_XFLAG_REALTIME)?
You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.
Quoted 6 lines
3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.
Cheers.
Signature · 1 line
Carlos Maiolino
Hi,
Quoted 22 lines · Wang Yugui
Quoted 7 lines
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunderQuoted 7 lines
There are some questions about how to control the data to save into normal vol or realtime subvol firstly.
1, man xfsctl
here isXFS_XFLAG_REALTIMEin man xfsctl of xfsprogs 5.0 , but there is noXFS_XFLAG_REALTIMEin xfsprogs 5.14/5.19.xfsctl(XFS_XFLAG_REALTIME)will be removed in the further?It's been a while since XFS uses
FS_XFLAGfeatures directly, so, what you're specifically looking for isFS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:#define
XFS_XFLAG_REALTIMEFS_XFLAG_REALTIME
Quoted 13 lines
FS_XFLAG_REALTIMEis part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.Quoted 2 lines
2, Is there some tool to do
xfsctl(XFS_XFLAG_REALTIME)?You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.
Quoted 3 lines
3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.
Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev device
Quoted 8 lines
Quoted 2 lines
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.
Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.
# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
typedef struct xfs_sb {
...
uint64_t sb_fdblocks; /* free data blocks */
uint64_t sb_frextents; /* free realtime extents */
...
}And based the info from Carlos Maiolino
FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:
https://lore.kernel.org/linux-xfs/[email protected]/
With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.
fs/xfs/xfs_inode.c +3 -0
@@ -868,6 +868,9 @@ xfs_init_new_inode( flags |=XFS_ILOG_DEV ;break ;case S_IFREG : +if (xfs_has_realtime (ip->i_mount)) + ip->i_diflags |=XFS_DIFLAG_REALTIME ; + fallthrough;case S_IFDIR :if (pip && (pip->i_diflags &XFS_DIFLAG_ANY ))xfs_inode_inherit_flags (ip, pip);
Thanks a lot.
Best Regards
Wang Yugui ([email protected])
2022/08/30
Quoted 68 lines · Wang Yugui
Hi,
Quoted 22 lines · Wang Yugui
Quoted 7 lines
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunderQuoted 7 lines
There are some questions about how to control the data to save into normal vol or realtime subvol firstly.
1, man xfsctl
here isXFS_XFLAG_REALTIMEin man xfsctl of xfsprogs 5.0 , but there is noXFS_XFLAG_REALTIMEin xfsprogs 5.14/5.19.xfsctl(XFS_XFLAG_REALTIME)will be removed in the further?It's been a while since XFS uses
FS_XFLAGfeatures directly, so, what you're specifically looking for isFS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:#define
XFS_XFLAG_REALTIMEFS_XFLAG_REALTIMEQuoted 13 lines
FS_XFLAG_REALTIMEis part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.Quoted 2 lines
2, Is there some tool to do
xfsctl(XFS_XFLAG_REALTIME)?You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.
Quoted 3 lines
3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.
Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev deviceQuoted 8 lines
Quoted 2 lines
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.
Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.
# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents' typedef struct xfs_sb { ... uint64_t sb_fdblocks; /* free data blocks */ uint64_t sb_frextents; /* free realtime extents */ ... }And based the info from Carlos Maiolino
FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:
Quoted 5 lines
https://lore.kernel.org/linux-xfs/[email protected]/
With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p/dev/sdb8|grep 'fdblocks\|frextents'.
mkfs.xfs -d rtinherit=1...
--D
Quoted 21 lines
fs/xfs/xfs_inode.c +3
-0@@ -868,6 +868,9 @@ xfs_init_new_inode( flags |=XFS_ILOG_DEV ;break ;case S_IFREG : +if (xfs_has_realtime (ip->i_mount)) + ip->i_diflags |=XFS_DIFLAG_REALTIME ; + fallthrough;case S_IFDIR :if (pip && (pip->i_diflags &XFS_DIFLAG_ANY ))xfs_inode_inherit_flags (ip, pip);Thanks a lot.
Best Regards
Wang Yugui ([email protected])
2022/08/30
Hi,
Quoted 69 lines · Wang Yugui
Quoted 68 lines
Hi,
Quoted 22 lines · Wang Yugui
Quoted 7 lines
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunderQuoted 7 lines
There are some questions about how to control the data to save into normal vol or realtime subvol firstly.
1, man xfsctl
here isXFS_XFLAG_REALTIMEin man xfsctl of xfsprogs 5.0 , but there is noXFS_XFLAG_REALTIMEin xfsprogs 5.14/5.19.xfsctl(XFS_XFLAG_REALTIME)will be removed in the further?It's been a while since XFS uses
FS_XFLAGfeatures directly, so, what you're specifically looking for isFS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:#define
XFS_XFLAG_REALTIMEFS_XFLAG_REALTIMEQuoted 13 lines
FS_XFLAG_REALTIMEis part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.Quoted 2 lines
2, Is there some tool to do
xfsctl(XFS_XFLAG_REALTIME)?You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.
Quoted 3 lines
3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.
Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev deviceQuoted 8 lines
Quoted 2 lines
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.
Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.
# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents' typedef struct xfs_sb { ... uint64_t sb_fdblocks; /* free data blocks */ uint64_t sb_frextents; /* free realtime extents */ ... }And based the info from Carlos Maiolino
FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:
Quoted 7 lines
Quoted 5 lines
https://lore.kernel.org/linux-xfs/[email protected]/
With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p/dev/sdb8|grep 'fdblocks\|frextents'.mkfs.xfs -d rtinherit=1...
'mkfs.xfs -d rtinherit=1' works very well. Thanks a lot.
I noticed that 'reflink not supported with realtime devices' both in mkfs.xfs and in kernel/xfs(c14632ddac98 xfs: don't allow reflink + realtime filesystems).
Is this a limit of current design?
or we just disable reflink when realtime devices just for QoS/realtime?
Best Regards
Wang Yugui ([email protected])
2022/08/31
Quoted 79 lines · Wang Yugui
Hi,
Quoted 69 lines · Wang Yugui
Quoted 68 lines
Hi,
Quoted 22 lines · Wang Yugui
Quoted 7 lines
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunderQuoted 7 lines
There are some questions about how to control the data to save into normal vol or realtime subvol firstly.
1, man xfsctl
here isXFS_XFLAG_REALTIMEin man xfsctl of xfsprogs 5.0 , but there is noXFS_XFLAG_REALTIMEin xfsprogs 5.14/5.19.xfsctl(XFS_XFLAG_REALTIME)will be removed in the further?It's been a while since XFS uses
FS_XFLAGfeatures directly, so, what you're specifically looking for isFS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:#define
XFS_XFLAG_REALTIMEFS_XFLAG_REALTIMEQuoted 13 lines
FS_XFLAG_REALTIMEis part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.Quoted 2 lines
2, Is there some tool to do
xfsctl(XFS_XFLAG_REALTIME)?You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.
Quoted 3 lines
3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.
Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev deviceQuoted 8 lines
Quoted 2 lines
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.
Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.
# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents' typedef struct xfs_sb { ... uint64_t sb_fdblocks; /* free data blocks */ uint64_t sb_frextents; /* free realtime extents */ ... }And based the info from Carlos Maiolino
FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:
Quoted 7 lines
Quoted 5 lines
https://lore.kernel.org/linux-xfs/[email protected]/
With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p/dev/sdb8|grep 'fdblocks\|frextents'.mkfs.xfs -d rtinherit=1...
Quoted 8 lines
'mkfs.xfs -d rtinherit=1' works very well. Thanks a lot.
I noticed that 'reflink not supported with realtime devices' both in mkfs.xfs and in kernel/xfs(c14632ddac98 xfs: don't allow reflink + realtime filesystems).
Is this a limit of current design?
or we just disable reflink when realtime devices just for QoS/realtime?
No, just unfinished features. :/
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink
--D
Quoted 5 lines
Best Regards
Wang Yugui ([email protected])
2022/08/31
Quoted 6 lines · Wang Yugui
Hi,
I saw some info about hybird xfs wih ssd/hdd by realtime subvol.
Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
....
Quoted 2 lines
Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?
FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:
https://lore.kernel.org/linux-xfs/[email protected]/
-Dave.
Signature · 2 lines
Dave Chinner [email protected]