Skip to contentSkip to conversation

questions about hybird xfs wih ssd/hdd by realtime subvol

DISCUSSION Wang Yugui 4 participants 7 messages Aug 29, 2022 – Aug 31, 2022 XFS filesystem

lore
Share & tools
lore

Wang Yugui

author
#0 lore raw

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

There are some questions about how to control the data to save into normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

Best Regards
Wang Yugui ([email protected])
2022/08/29

maintainer
#1 lore raw
Quoted 7 lines · Wang Yugui

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

Quoted 7 lines

There are some questions about how to control the data to save into normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

It's been a while since XFS uses FS_XFLAG features directly, so, what you're specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:

#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME

FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.

Quoted 2 lines

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.

Quoted 6 lines

3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.

Cheers.

Signature · 1 line
Carlos Maiolino
author
#2 lore raw

Hi,

Quoted 22 lines · Wang Yugui
Quoted 7 lines

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

Quoted 7 lines

There are some questions about how to control the data to save into normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

It's been a while since XFS uses FS_XFLAG features directly, so, what you're specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:

#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME

Quoted 13 lines

FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.

Quoted 2 lines

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.

Quoted 3 lines

3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.

Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev device

Quoted 8 lines
Quoted 2 lines

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.

Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.

# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
typedef struct xfs_sb {
...
	uint64_t	sb_fdblocks;	/* free data blocks */
	uint64_t	sb_frextents;	/* free realtime extents */
...
}

And based the info from Carlos Maiolino

FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:

https://lore.kernel.org/linux-xfs/[email protected]/

With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.

fs/xfs/xfs_inode.c +3 -0
index f4cc8a1aaeb4..d19e0fa34c1a 100644
@@ -868,6 +868,9 @@ xfs_init_new_inode(
                flags |= XFS_ILOG_DEV;
                break;
        case S_IFREG:
+               if (xfs_has_realtime(ip->i_mount))
+                       ip->i_diflags |= XFS_DIFLAG_REALTIME;
+               fallthrough;
        case S_IFDIR:
                if (pip && (pip->i_diflags & XFS_DIFLAG_ANY))
                        xfs_inode_inherit_flags(ip, pip);

Thanks a lot.

Best Regards
Wang Yugui ([email protected])
2022/08/30

#3 lore raw
Quoted 68 lines · Wang Yugui

Hi,

Quoted 22 lines · Wang Yugui
Quoted 7 lines

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

Quoted 7 lines

There are some questions about how to control the data to save into normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

It's been a while since XFS uses FS_XFLAG features directly, so, what you're specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:

#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME

Quoted 13 lines

FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.

Quoted 2 lines

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.

Quoted 3 lines

3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.

Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev device

Quoted 8 lines
Quoted 2 lines

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.

Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.

# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
typedef struct xfs_sb {
...
	uint64_t	sb_fdblocks;	/* free data blocks */
	uint64_t	sb_frextents;	/* free realtime extents */
...
}

And based the info from Carlos Maiolino

FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:

Quoted 5 lines

https://lore.kernel.org/linux-xfs/[email protected]/

With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.

mkfs.xfs -d rtinherit=1...

--D

Quoted 21 lines
fs/xfs/xfs_inode.c +3 -0
index f4cc8a1aaeb4..d19e0fa34c1a 100644
@@ -868,6 +868,9 @@ xfs_init_new_inode(
                flags |= XFS_ILOG_DEV;
                break;
        case S_IFREG:
+               if (xfs_has_realtime(ip->i_mount))
+                       ip->i_diflags |= XFS_DIFLAG_REALTIME;
+               fallthrough;
        case S_IFDIR:
                if (pip && (pip->i_diflags & XFS_DIFLAG_ANY))
                        xfs_inode_inherit_flags(ip, pip);

Thanks a lot.

Best Regards
Wang Yugui ([email protected])
2022/08/30

author
#4 lore raw

Hi,

Quoted 69 lines · Wang Yugui
Quoted 68 lines

Hi,

Quoted 22 lines · Wang Yugui
Quoted 7 lines

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

Quoted 7 lines

There are some questions about how to control the data to save into normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

It's been a while since XFS uses FS_XFLAG features directly, so, what you're specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:

#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME

Quoted 13 lines

FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.

Quoted 2 lines

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.

Quoted 3 lines

3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.

Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev device

Quoted 8 lines
Quoted 2 lines

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.

Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.

# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
typedef struct xfs_sb {
...
	uint64_t	sb_fdblocks;	/* free data blocks */
	uint64_t	sb_frextents;	/* free realtime extents */
...
}

And based the info from Carlos Maiolino

FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:

Quoted 7 lines
Quoted 5 lines

https://lore.kernel.org/linux-xfs/[email protected]/

With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.

mkfs.xfs -d rtinherit=1...

'mkfs.xfs -d rtinherit=1' works very well. Thanks a lot.

I noticed that 'reflink not supported with realtime devices' both in mkfs.xfs and in kernel/xfs(c14632ddac98 xfs: don't allow reflink + realtime filesystems).

Is this a limit of current design?
or we just disable reflink when realtime devices just for QoS/realtime?

Best Regards
Wang Yugui ([email protected])
2022/08/31

#5 lore raw
Quoted 79 lines · Wang Yugui

Hi,

Quoted 69 lines · Wang Yugui
Quoted 68 lines

Hi,

Quoted 22 lines · Wang Yugui
Quoted 7 lines

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

Quoted 7 lines

There are some questions about how to control the data to save into normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19. xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

It's been a while since XFS uses FS_XFLAG features directly, so, what you're specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a preprocessor define:

#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME

Quoted 13 lines

FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without the realtime filesystems going first, so, unlikely it's gonna happen.

Quoted 2 lines

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

You can use xfs_io's chattr command to add/remote the REALTIME attribute of a file.

Quoted 3 lines

3, we build a xfs filesystem with 1G device and 1G rtdev device. and then we can save 2G data into this xfs filesystem.

Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev device

Quoted 8 lines
Quoted 2 lines

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

I didn't watch the talk you mentioned above, but when use an rt device, you don't use the 'normal' one then the rt later, or vice-versa, the rt-device is used to store data blocks for those files marked with the xattr above. For those files you want to store in the realtime device, you should add the above xattr to them.

Although I still fail to check/set the attr by 'lsattr/chattr', but I can check the free space of 'normal' and realtime subvol now.

# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
typedef struct xfs_sb {
...
	uint64_t	sb_fdblocks;	/* free data blocks */
	uint64_t	sb_frextents;	/* free realtime extents */
...
}

And based the info from Carlos Maiolino

FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:

Quoted 7 lines
Quoted 5 lines

https://lore.kernel.org/linux-xfs/[email protected]/

With a dirty patch below for test only , Now realtime subvol will be used as I expected, and that can be confirmed by
#xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.

mkfs.xfs -d rtinherit=1...

Quoted 8 lines

'mkfs.xfs -d rtinherit=1' works very well. Thanks a lot.

I noticed that 'reflink not supported with realtime devices' both in mkfs.xfs and in kernel/xfs(c14632ddac98 xfs: don't allow reflink + realtime filesystems).

Is this a limit of current design?
or we just disable reflink when realtime devices just for QoS/realtime?

No, just unfinished features. :/

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink

--D

Quoted 5 lines

Best Regards
Wang Yugui ([email protected])
2022/08/31

#6 lore raw
In reply to Wang Yugui
Quoted 6 lines · Wang Yugui

Hi,

I saw some info about hybird xfs wih ssd/hdd by realtime subvol.

Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

....

Quoted 2 lines

Is there any tool/kernel option/kernel patch to control the data to save into normal vol or realtime subvol firstly?

FB were running a modified kernel that selected the rt dev based on the initial allocation size. Behaviour for them was predictable because they also controlled the application that was storing the data. See:

https://lore.kernel.org/linux-xfs/[email protected]/

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]