XFS real-time device tweaks

Taking another run at this based on all the feedback (much appreciated).

- Replaced rtdefault with rtdisable, this yields similar operational benefits when combined with the existing mkfs time setting of the inheritance flag on the root directory. Allows temporary disabling of real-time allocation without having to walk entire FS to remove flags (which could be time consuming). I still don't think it's super obvious to an admin the real-time flag was put there at mkfs time (vs. rtdefault being in mount flags), but this gets me half of what I'm after.
- rtstatfs flag is removed, instead per Dave's suggestion we look for the inheritance flag on the directory inode, if so we fill the statfs struct with the real-time block info, otherwise data block info. Open to making this behavior a flag if folks are worried it might be a jarring change to folks used to the old behavior (i.e. data device info no matter what).
- rtfallocmin no changes, need to think more about this. Still a pretty big fan of this option for reasons already stated; at least until a more elegant solution such as preferred AGs (we'd need a tunable size for the "preferred" AG, since our SSD partitions are a fraction of the size of a normal AG) can be implemented. The only other idea I have is to make a new ioctl e.g. "norealtime", which causes the RT bits to stay cleared regardless of inheritance bits on the containing directory. This would allowing the "steering" of files to the data device (e.g. SSD); this is probably a safer design than defaulting to SSD and steering to the HDD via the realtime ioctl.

Richard Wareing (3):
fs/xfs: Add rtdisable option
fs/xfs: Add real-time device support to statfs
fs/xfs: Add rtfallocmin mount option

 fs/xfs/xfs_file.c  | 16 ++++++++++++++++
 fs/xfs/xfs_inode.c |  6 ++++--
 fs/xfs/xfs_ioctl.c |  7 +++++--
 fs/xfs/xfs_mount.h |  2 ++
 fs/xfs/xfs_super.c | 33 ++++++++++++++++++++++++++++++++-
 5 files changed, 59 insertions(+), 5 deletions(-)

2.9.3

- Adds rtdisable mount option to ignore any real-time inheritance flag set on directories. Designed to be used as a "kill-switch" for this behavior, negating the need to walk/kill flags on FS.

---
 fs/xfs/xfs_inode.c |  6 ++++--
 fs/xfs/xfs_ioctl.c |  7 +++++--
 fs/xfs/xfs_mount.h |  1 +
 fs/xfs/xfs_super.c | 14 +++++++++++++-
 4 files changed, 23 insertions(+), 5 deletions(-)

index ec9826c..dc53731 100644
@@ -878,7 +878,8 @@ xfs_ialloc(
 			uint		di_flags = 0;
 
 			if (S_ISDIR(mode)) {
-				if (pip->i_d.di_flags & XFS_DIFLAG_RTINHERIT)
+				if (!(mp->m_flags & XFS_MOUNT_RTDISABLE) &&
+					pip->i_d.di_flags & XFS_DIFLAG_RTINHERIT)
 					di_flags |= XFS_DIFLAG_RTINHERIT;
 				if (pip->i_d.di_flags & XFS_DIFLAG_EXTSZINHERIT) {
 					di_flags |= XFS_DIFLAG_EXTSZINHERIT;

@@ -887,7 +888,8 @@ xfs_ialloc(
 				if (pip->i_d.di_flags & XFS_DIFLAG_PROJINHERIT)
 					di_flags |= XFS_DIFLAG_PROJINHERIT;
 			} else if (S_ISREG(mode)) {
-				if (pip->i_d.di_flags & XFS_DIFLAG_RTINHERIT)
+				if (!(mp->m_flags & XFS_MOUNT_RTDISABLE) &&
+					pip->i_d.di_flags & XFS_DIFLAG_RTINHERIT)
 					di_flags |= XFS_DIFLAG_REALTIME;
 				if (pip->i_d.di_flags & XFS_DIFLAG_EXTSZINHERIT) {
 					di_flags |= XFS_DIFLAG_EXTSIZE;

index 6190697..5a6d45d 100644
@@ -937,6 +937,7 @@ xfs_set_diflags(
 	struct xfs_inode	*ip,
 	unsigned int		xflags)
 {
+	struct xfs_mount        *mp = ip->i_mount;
 	unsigned int		di_flags;
 	uint64_t		di_flags2;
 

@@ -957,7 +958,8 @@ xfs_set_diflags(
 	if (xflags & FS_XFLAG_FILESTREAM)
 		di_flags |= XFS_DIFLAG_FILESTREAM;
 	if (S_ISDIR(VFS_I(ip)->i_mode)) {
-		if (xflags & FS_XFLAG_RTINHERIT)
+		if (!(mp->m_flags & XFS_MOUNT_RTDISABLE) &&
+			xflags & FS_XFLAG_RTINHERIT)
 			di_flags |= XFS_DIFLAG_RTINHERIT;
 		if (xflags & FS_XFLAG_NOSYMLINKS)
 			di_flags |= XFS_DIFLAG_NOSYMLINKS;

@@ -966,7 +968,8 @@ xfs_set_diflags(
 		if (xflags & FS_XFLAG_PROJINHERIT)
 			di_flags |= XFS_DIFLAG_PROJINHERIT;
 	} else if (S_ISREG(VFS_I(ip)->i_mode)) {
-		if (xflags & FS_XFLAG_REALTIME)
+		if (!(mp->m_flags & XFS_MOUNT_RTDISABLE) &&
+			xflags & FS_XFLAG_REALTIME)
 			di_flags |= XFS_DIFLAG_REALTIME;
 		if (xflags & FS_XFLAG_EXTSIZE)
 			di_flags |= XFS_DIFLAG_EXTSIZE;

index 9fa312a..8016ddb 100644
@@ -243,6 +243,7 @@ typedef struct xfs_mount {
 						   allocator */
 #define XFS_MOUNT_NOATTR2	(1ULL << 25)	/* disable use of attr2 format */
 
+#define XFS_MOUNT_RTDISABLE    (1ULL << 61)    /* Ignore RT flags */
 #define XFS_MOUNT_DAX		(1ULL << 62)	/* TEST ONLY! */
 
 

index 455a575..4dbf95c 100644
@@ -83,7 +83,7 @@ enum {
 	Opt_quota, Opt_noquota, Opt_usrquota, Opt_grpquota, Opt_prjquota,
 	Opt_uquota, Opt_gquota, Opt_pquota,
 	Opt_uqnoenforce, Opt_gqnoenforce, Opt_pqnoenforce, Opt_qnoenforce,
-	Opt_discard, Opt_nodiscard, Opt_dax, Opt_err,
+	Opt_discard, Opt_nodiscard, Opt_dax, Opt_rtdisable, Opt_err,
 };
 
 static const match_table_t tokens = {

@@ -133,6 +133,9 @@ static const match_table_t tokens = {
 
 	{Opt_dax,	"dax"},		/* Enable direct access to bdev pages */
 
+#ifdef CONFIG_XFS_RT
+	{Opt_rtdisable, "rtdisable"},   /* Ignore real-time flags */
+#endif
 	/* Deprecated mount options scheduled for removal */
 	{Opt_barrier,	"barrier"},	/* use writer barriers for log write and
 					 * unwritten extent conversion */

@@ -367,6 +370,11 @@ xfs_parseargs(
 		case Opt_nodiscard:
 			mp->m_flags &= ~XFS_MOUNT_DISCARD;
 			break;
+#ifdef CONFIG_XFS_RT
+		case Opt_rtdisable:
+			mp->m_flags |= XFS_MOUNT_RTDISABLE;
+			break;
+#endif
 #ifdef CONFIG_FS_DAX
 		case Opt_dax:
 			mp->m_flags |= XFS_MOUNT_DAX;

@@ -492,6 +500,10 @@ xfs_showargs(
 		{ XFS_MOUNT_DISCARD,		",discard" },
 		{ XFS_MOUNT_SMALL_INUMS,	",inode32" },
 		{ XFS_MOUNT_DAX,		",dax" },
+#ifdef CONFIG_XFS_RT
+		{ XFS_MOUNT_RTDISABLE,          ",rtdisable" },
+#endif
+
 		{ 0, NULL }
 	};
 	static struct proc_xfs_info xfs_info_unset[] = {

2.9.3

- Reports real-time device free blocks in statfs calls if inheritance bit is set on the inode of directory. This is a bit more intuitive, especially for use-cases which are using a much larger device for the real-time device.

---
 fs/xfs/xfs_super.c | 6 ++++++
 1 file changed, 6 insertions(+)

index 4dbf95c..a1d6968 100644
@@ -1148,6 +1148,12 @@ xfs_fs_statfs(
 	    ((mp->m_qflags & (XFS_PQUOTA_ACCT|XFS_PQUOTA_ENFD))) ==
 			      (XFS_PQUOTA_ACCT|XFS_PQUOTA_ENFD))
 		xfs_qm_statvfs(ip, statp);
+	if ((ip->i_d.di_flags & XFS_DIFLAG_RTINHERIT) &&
+		(mp->m_rtdev_targp != NULL)) {
+		statp->f_blocks = sbp->sb_rblocks;
+		statp->f_bfree = sbp->sb_frextents * sbp->sb_rextsize -
+			mp->m_alloc_set_aside;
+	}
 	return 0;
 }
 

2.9.3

- Gates real-time block device fallocation's to rtfallocmin bytes
- Use case: Allows developers to send files to the SSD with ease simply by fallocating them, if they are below rtfallocmin XFS will allocate the blocks from the non-RT device (e.g. an SSD)
- Useful to automagically store small files on the SSD vs. RT device (HDD) for tiered XFS setups without having to rely on XFS specific ioctl calls. Userland tools such as rsync can also use fallocation behavior to migrate files between SSD and RT (HDD) device without modifiction (e.g. w/ --preallocate flag).

---
 fs/xfs/xfs_file.c  | 16 ++++++++++++++++
 fs/xfs/xfs_mount.h |  1 +
 fs/xfs/xfs_super.c | 15 ++++++++++++++-
 3 files changed, 31 insertions(+), 1 deletion(-)

index 5fb5a09..a29f6e8 100644
@@ -730,6 +730,7 @@ xfs_file_fallocate(
 {
 	struct inode		*inode = file_inode(file);
 	struct xfs_inode	*ip = XFS_I(inode);
+	struct xfs_mount	*mp = ip->i_mount;
 	long			error;
 	enum xfs_prealloc_flags	flags = 0;
 	uint			iolock = XFS_IOLOCK_EXCL;

@@ -749,6 +750,21 @@ xfs_file_fallocate(
 	xfs_ilock(ip, XFS_MMAPLOCK_EXCL);
 	iolock |= XFS_MMAPLOCK_EXCL;
 
+	/*
+	 * If fallocating a file < rtfallocmin store it on the non RT device.
+	 * In a tiered storage setup, this device might be a device suitable
+	 * for better small file storage/performance (e.g. SSD).
+	 */
+	if (mp->m_rtdev_targp && mp->m_rtfallocmin && !offset &&
+			!inode->i_size) {
+		if (len >= mp->m_rtfallocmin) {
+			ip->i_d.di_flags |= XFS_DIFLAG_REALTIME;
+		/* Clear flag if inheritence or rtdefault is being used */
+		} else {
+			ip->i_d.di_flags &= ~XFS_DIFLAG_REALTIME;
+		}
+	}
+
 	if (mode & FALLOC_FL_PUNCH_HOLE) {
 		error = xfs_free_file_space(ip, offset, len);
 		if (error)

index 8016ddb..f5593bb 100644
@@ -84,6 +84,7 @@ typedef struct xfs_mount {
 	char			*m_fsname;	/* filesystem name */
 	int			m_fsname_len;	/* strlen of fs name */
 	char			*m_rtname;	/* realtime device name */
+	int			m_rtfallocmin;  /* Min size for RT fallocate */
 	char			*m_logname;	/* external log device name */
 	int			m_bsize;	/* fs logical block size */
 	xfs_agnumber_t		m_agfrotor;	/* last ag where space found */

index a1d6968..01a2ab4 100644
@@ -83,7 +83,8 @@ enum {
 	Opt_quota, Opt_noquota, Opt_usrquota, Opt_grpquota, Opt_prjquota,
 	Opt_uquota, Opt_gquota, Opt_pquota,
 	Opt_uqnoenforce, Opt_gqnoenforce, Opt_pqnoenforce, Opt_qnoenforce,
-	Opt_discard, Opt_nodiscard, Opt_dax, Opt_rtdisable, Opt_err,
+	Opt_discard, Opt_nodiscard, Opt_dax, Opt_rtdisable, Opt_rtfallocmin,
+	Opt_err,
 };
 
 static const match_table_t tokens = {

@@ -135,6 +136,9 @@ static const match_table_t tokens = {
 
 #ifdef CONFIG_XFS_RT
 	{Opt_rtdisable, "rtdisable"},   /* Ignore real-time flags */
+	{Opt_rtfallocmin, "rtfallocmin=%u"}, /* Min fallocation required
+										  * for rt device
+										  */
 #endif
 	/* Deprecated mount options scheduled for removal */
 	{Opt_barrier,	"barrier"},	/* use writer barriers for log write and

@@ -374,6 +378,10 @@ xfs_parseargs(
 		case Opt_rtdisable:
 			mp->m_flags |= XFS_MOUNT_RTDISABLE;
 			break;
+		case Opt_rtfallocmin:
+			if (match_int(args, &mp->m_rtfallocmin))
+				return -EINVAL;
+			break;
 #endif
 #ifdef CONFIG_FS_DAX
 		case Opt_dax:

@@ -538,6 +546,11 @@ xfs_showargs(
 	if (mp->m_rtname)
 		seq_show_option(m, "rtdev", mp->m_rtname);
 
+#ifdef CONFIG_XFS_RT
+	if (mp->m_rtfallocmin > 0)
+		seq_printf(m, ",rtfallocmin=%d", mp->m_rtfallocmin);
+#endif
+
 	if (mp->m_dalign > 0)
 		seq_printf(m, ",sunit=%d",
 				(int)XFS_FSB_TO_BB(mp, mp->m_dalign));

2.9.3

I'd be much happier if this was done inside the allocator, and in affect for any initial allocation, not just fallocate, as that keeps the layering and logic much cleaner.

See my comments on your reply to my cover letter. Would love to hear your thoughts on my reply.

I still don't understand this option. What is the use case of dynamically switching on/off these default to the rt device?

Jens just added a nice new fcntl to declare the life time of write streams (and in theory can add other I/O hints).

How about a a mount option that moves all I/O with a given hint to the RT device? E.g. rt=longlife would direct I/O on a file with an rw hint of RWH_WRITE_LIFE_LONG or RWH_WRITE_LIFE_EXTREME to the RT subvolume as long as there aren't any previous extents.

Say you are in a bit of an emergency, and you need IOPs *now* (incident recovery), w/ rtdisable you could funnel the IO to the SSD without having to strip the inheritance bits from all the directories (which would require two walks....one to remove and one to add them all back). I think this is about having some options during incidents, and a "kill-switch" should the need arise.

You seem to trust application developers more than I :). The problem I see with the lifetime, or allocation size as a hint is that a user could later append to the file and fill up the SSD. A "norealtime" or fallocation request is a bit more explicit and high signal about the intent vs. the lifetime or allocation size alone. It's possible, I happen to trickle writes into a file which may ultimately become very very large (e.g. logging), or perhaps introduce a performance or buffering bug which triggers smaller writes (allocations) and altered write lifetimes. With fallocmin, this won't happen as the assumption/relationship here is clear, you are clearly declaring your intent to write a file of N bytes, and based on that we promote or demote you to the appropriate tier of storage.

The other problem I see is accessibility and usability. By making these decisions buried in more generic XFS allocation mechanisms or fnctl's, few developers are going to really understand how to safely use them (e.g. without blowing up their SSD's WAF or endurance). Fallocation is a better understood notion, easier to use and has wider support amongst existing utilities. Keep in mind, we need our SSDs to last >3 years (w/ only a mere 70-80 TBW; 300TBW if we are lucky), so we want to design things such that application developers are less likely to step on land mines causing pre-mature SSD failure.

Whatever the ultimate solution here, it should be designed such that it's relatively difficult to accidentally write data to the non-RT device (e.g. SSD in our case); intent must be clear and high signal. Thus my similar "high-signal" bias in my first patchset w/ rtdefault; sure inheritance bits should be there if somebody mkfs'd, but if somehow they were removed, it could wind up costing 10's of millions of dollars in reduced SSD write lifetime at our scale. An explicit mount option makes me sleep better at night, things like chef/cfengine can enforce this through traditional policy mechanisms, and removing the behavior has a higher bar (remount + change chef/cfengine) than a trivial call to xfs_io. From a production engineering/reliability stand-point the design decision is pretty clear; inheritance bits are nearly unenforcible with policy engines such as cfengine or chef (somebody/something could remove a bit buried in the FS and you'd find it only by walking the entire FS), and as result they are bombs waiting to go off compared to the rtdefault flag.

I'd ideally like to take things even further with fallocmin, and revert to the RT device should the non-RT device fill up (subtracting some % of space for metadata); this brings the behavior more along the lines of a "preferred" device vs. a must-have.

But it /doesn't do that/. It only disables new files from writing to the rt device. All reads for data in the RT device and writes to existing files still go to the RT device.

And soon after the kill switch is triggered, your tiny data device will go ENOSPC because changing that mount option effective removed TBs of free space from the filesystem. Then things will really start going bad.

So maybe you didn't think this through properly - the last thing a typical user would expect is a filesystem reporting TBs of free space to go ENOSPC and not being able to recover, regardless of what mount options are present. iAnd they'll be especially confused when they start looking at inodes and seeing RT bits set all over the place...

It's just a recipe for confusion, unexpected behaviour and all I see here is a support and triage nightmare. Not to mention FB will move on to something else in a couple of years, and we get stuck having to maintain it forever more (*cough* filestreams *cough*).

The whole point of putting them into the XFS allocator as admin policies is that *applications developers don't need to know they exist*.

Almost every application I've seen that uses fallocate does something wrong and/or breaks a longevity or performance optimisation that filesystems have been making for years.

fallocate is "easy to understand" but *difficult to use optimally* because it's behaviour is tightly bound to the filesystem allocator algorithms. i.e. it's easy to defeat hidden filesystem optimisations with fallocate, but it's difficult to understand a sub-optimal corner case in the filesystem allocator that fallocate could be used to avoid.

In reality, we don't want people using fallocate - the filesystem algorithms should do the right thing so people don't need to modify their applications. In cases like this, having the filesystem decide automatically at first allocation what device to use is the right way to integrate the functionality, not require users to use fallocate to trigger such a decision and, as a side effect, prevent the filesystem from making all the other optimisations they still want it to make.

Hmmm. I don't think they way you are using fallocate is doing what you think it is doing.

That is, using fallocate to preallocate all files so you can direct allocation to a different device means that delayed allocation is turned off. Hence XFS cannot optimise allocation across multiple files at writeback time. This means that writeback across multiple files will be sprayed around disjointed preallocated regions. When using delayed allocation, the filesystem will allocate the blocks for all the files sequentially and so the block layer merge will them all into one big contiguous IO.

IOWs, fallocate sprays write IO around because they decouple allocation locality from temporal writeback locality and this causes non-contiguous write patterns which are a significant contributin factor to write amplification in SSDs. In comparison, delayed allocation results in large sequential IOs that minimise write amplification in the SSD...

Hence the method you describe that "maximises SSD life" won't help
- if anything it's going to actively harm the SSD life when compared to just letting the filesystem use delayed allocation and choose what device to write to at that time....

Hacking one-off high level controls into APIs like fallocate does not work. Allocation policies need to be integrated into the filesystem allocators for them to be effective and useful to administrators and applications alike. fallocate is no sustitute for the black magic that filesystems do to optimise allocation and IO patterns....

Cheers,

Dave.

Dave Chinner
[email protected]

On 9/5/17, 8:45 PM, "Dave Chinner" <[email protected]> wrote:

On Sun, Sep 03, 2017 at 10:02:41PM +0000, Richard Wareing wrote:
>
> > On Sep 3, 2017, at 1:56 AM, Christoph Hellwig
> > <[email protected]> wrote:
> >
> > On Sat, Sep 02, 2017 at 03:41:42PM -0700, Richard Wareing > > wrote:
> >> - Replaced rtdefault with rtdisable, this yields similar > >> operational benefits when combined with the existing mkfs time > >> setting of the inheritance flag on the root directory. Allows > >> temporary disabling of real-time allocation without having to > >> walk entire FS to remove flags (which could be time consuming). > >> I still don't think it's super obvious to an admin the > >> real-time flag was put there at mkfs time (vs. rtdefault being > >> in mount flags), but this gets me half of what I'm after. > >
> > I still don't understand this option. What is the use case of > > dynamically switching on/off these default to the rt device? > >
>
> Say you are in a bit of an emergency, and you need IOPs *now* > (incident recovery), w/ rtdisable you could funnel the IO to the > SSD

But it /doesn't do that/. It only disables new files from writing to the rt device. All reads for data in the RT device and writes to existing files still go to the RT device.

> without having to strip the inheritance bits from all the > directories (which would require two walks....one to remove and
> one to add them all back). I think this is about having some
> options during incidents, and a "kill-switch" should the need > arise.

And soon after the kill switch is triggered, your tiny data device will go ENOSPC because changing that mount option effective removed TBs of free space from the filesystem. Then things will really start going bad.

So maybe you didn't think this through properly - the last thing a typical user would expect is a filesystem reporting TBs of free space to go ENOSPC and not being able to recover, regardless of what mount options are present. iAnd they'll be especially confused when they start looking at inodes and seeing RT bits set all over the place...

It's just a recipe for confusion, unexpected behaviour and all I see here is a support and triage nightmare. Not to mention FB will move on to something else in a couple of years, and we get stuck having to maintain it forever more (*cough* filestreams *cough*).

Fair enough, what are your thoughts on rtdefault, if I changed it to *not* set the inheritance bits, but take over this responsibility in their place? My thinking here is this integrates better than inheritance bits w/ policy management systems such as Chef/Puppet. Inheritance bits, on the other hand don¹t really lend themselves to machine level policies; they can be sprinkled about all over the FS, and a walk would be required to enforce a machine wide policy.

Or instead of a mount option, would a sysfs option be acceptable?

My hope is we don¹t move on, but collaborate a bit more with the open-source world on these sorts of problems instead of re-inventing the proverbial FS wheel (and re-learning old lessons solved many moons ago by FS developers). Trying to do my part now, show it can be done and should be done.

The whole point of putting them into the XFS allocator as admin policies is that *applications developers don't need to know they exist*.

I get you now: *admins* need to know, but application developers not so much.

Almost every application I've seen that uses fallocate does something wrong and/or breaks a longevity or performance optimisation that filesystems have been making for years.

fallocate is "easy to understand" but *difficult to use optimally* because it's behaviour is tightly bound to the filesystem allocator algorithms. i.e. it's easy to defeat hidden filesystem optimisations with fallocate, but it's difficult to understand a sub-optimal corner case in the filesystem allocator that fallocate could be used to avoid.

In reality, we don't want people using fallocate - the filesystem algorithms should do the right thing so people don't need to modify their applications. In cases like this, having the filesystem decide automatically at first allocation what device to use is the right way to integrate the functionality, not require users to use fallocate to trigger such a decision and, as a side effect, prevent the filesystem from making all the other optimisations they still want it to make.

You make a good point here, on preventing the FS from making other optimizations. I¹m re-working this as you and others have suggested (new version tomorrow).

And xfs_fsr would be the home for code migrating the file to the real-time device once it grows beyond some tunable size.

Hmmm. I don't think they way you are using fallocate is doing what you think it is doing.

That is, using fallocate to preallocate all files so you can direct allocation to a different device means that delayed allocation is turned off. Hence XFS cannot optimise allocation across multiple files at writeback time. This means that writeback across multiple files will be sprayed around disjointed preallocated regions. When using delayed allocation, the filesystem will allocate the blocks for all the files sequentially and so the block layer merge will them all into one big contiguous IO.

IOWs, fallocate sprays write IO around because they decouple allocation locality from temporal writeback locality and this causes non-contiguous write patterns which are a significant contributin factor to write amplification in SSDs. In comparison, delayed allocation results in large sequential IOs that minimise write amplification in the SSD...

Hence the method you describe that "maximises SSD life" won't help
- if anything it's going to actively harm the SSD life when compared to just letting the filesystem use delayed allocation and choose what device to write to at that time....

Wrt to SSDs you are completely correct on this, our fallocate calls were intended to pay up front on the write path for more favorable allocations which pay off during reads on HDDs. For SSDs this clearly makes less sense, and an optimization we will need to make in our code for the reasons you point out.

Hacking one-off high level controls into APIs like fallocate does not work. Allocation policies need to be integrated into the filesystem allocators for them to be effective and useful to administrators and applications alike. fallocate is no sustitute for the black magic that filesystems do to optimise allocation and IO patterns....

Cheers,

Dave.
--
Dave Chinner
[email protected]

Thanks for the great comments, suggestions & insights. Learning a lot.

Richard

I don't think it's necessary. If you want automatic selection of the target device based on the first allocation size, then the first data allocation on a file will add the RT flag to the inode before calling into the RT allocator....

sysfs is preferable for options that are dynamically configurable.

Sure, nobody here has said what you are doing is conceptually unsound. All of the comments have been about the implementation and trying to understand what features from the implementation actually provide you with the benefit. Then we can focus in on a solid, maintainable solution...

Yeah, exactly. Sorry for not making this clearer. In general, we try to make the fs do the right thing by default and so tuning is not necessary. But if tuning is necessary, the policy is set by the admin and not the application as the admin knows a lot more about their specific hardware and execution context than an application developer.

OK.

Keep in mind that the allocation xfs_fsr does will follow whatever policy is currently in force. e.g. if a large file is on the wrong device, then just running the existing defrag operation on it should relocate the data to the correct device. Sure, fsr might need some help to recognise what "wrong device" means in it's inode scan routines, but the mechanism to move the data should be pretty much unchanged...

Cheers,

Dave.

Dave Chinner
[email protected]

FWIW, I'm still a little confused as to the need for this mechanism. What exactly is the use case for 1.) your specific environment and 2.) to a traditional realtime user?

Something like rtdefault (or an rtro option for realtime readonly behavior) seems a bit more generic to me if one wanted broad control over the feature, but your fallocate mount thingy seems to already accomplish that. I.e., if you made that thing set/clear RT on individual files based purely on file size and you had a need to quickly disable setting RT on new files, why can't you just remount without that option? It seems to me you wouldn't need to care about the RT inherit flag either way..?

I pretty much agree with everything Dave says here, along with Christoph's previous suggestion that this is better off in the allocator than in the fallocate path. In the end, I think your current environment won't know the difference because you fallocate everything up front anyways (notwithstanding Dave's explanation as to why that might not be the greatest idea, however). In fact, I think this would be much more interesting overall if we could tier per-extent allocation rather than per-file, but that of course is one of the limitations of using RT.

That said, while the implementation improvement makes sense, I'm still not necessarily convinced that this has a place in the upstream realtime feature. I'll grant you that I'm not terribly familiar with the historical realtime use case.. Dave, do you see value in such a heuristic as it relates to the realtime feature (not this tiering setup)? Is there necessarily a mapping between a large file size and a file that should be tagged realtime? E.g., I suppose somebody who is using traditional realtime (i.e., no SSD) and has a mix of legitimate realtime (streaming media) files and large sparse virt disk images or something of that nature would need to know to not use this feature (i.e., this requires documentation)..?

Brian

I don't see it much differently to the inode32 allocator policy. That separates metadata from data based on the type of allocation that is going to take place. inode32 decides on the AG for the inode data on the first data allocation (via the ag rotor), so there's already precedence for this sort of "locality selection at initial allocation" policy in the XFS allocation algorithms.

Some workloads run really well on inode32 because the metadata ends up tightly packed and you can keep lots of disks busy with a dm concat because data IO is effectively distributed over all AGs. We've never done that automatically with the rt device before, but if it allows hybrid setups to be constructed easily then I can see it being beneficial to those same sorts of worklaods....

And, FWIW, auto rtdev selection might also work quite nicely with write once large file workloads (i.e. archives) on SMR drives - data device for the PMR region for metadata and small or temporary files, rt device w/ appropriate extent size for larges files in the SMR region...

It wouldn't be enabled by default. We can't break existing rt device setups, so I don't see any issue here. And, well, someone mixing realtime and sparse virt in the same filesystem and storage isn't going to get reliable realtime response. i.e. nobody in their right mind mixes realtime streaming workloads with anything else - it's always dedicated hardware for RT....

Cheers,

Dave.

Dave Chinner
[email protected]

Ok, that sounds reasonable enough to me. Thanks.

Yes, that's just a dumb example. Let me rephrase...

Is there any legitimate realtime use case a filesystem may not want to tag all files of a particular size? E.g., this is more relevant for subsequent read requirements than anything, right? (If not, then why do we have the flag at all?) If so, then it seems to me this needs to be clearly documented...

Note that this use case defines large as >256k. Realtime use cases may have a much different definition, yes? I take it that means things like amount of physical memory and write workload may also be a significant factor in the effectiveness of this heuristic. For example, how much pagecache can we dirty before writeback occurs and does an initial allocation? How many large files are typically written in parallel? Also, what about direct I/O or extent size hints?

All I'm really saying is that I think this at least needs to consider the generic use case and have some documentation around any scnarios where this might not make sense for traditional users, what values might be sane, etc. As opposed to such users seeing an "automagic" knob, turning it on thinking it replaces the need to think about how to properly lay out the fs and then realizing later that this doesn't do what they expect. Thoughts?

Brian

Hmmm. I'm not following you here, Brian. RT has a deterministic allocator to prevent arbitrary IO delays on write, not read. The read side on RT is no different to the data device (i.e. extent lookup, read data) and as long as both allocators have given the file large contiguous extents there's no difference in the size and shape of read IOs being issued, either. So I'm not sure what you are saying needs documenting?

Also, keep in mind the RT device is not suited to small files at all. It's optimised for allocating large contiguous extents, it doesn't handle freespace fragmentation at all well so having small files come and go regularly really screws it up, and it's single threaded allocator means it can't handle the allocation demand that comes along with small file workloads, either.....

Again, if the workload is "realtime"(*) then it is not going to be using this functionality - everything needs to be tightly controlled and leave nothing to unpredictable algorithmic heuristics. Regardless, for different *data sets* the size threshold might be different, but that is for the admin who understands the environment and applications to separate workloads and set appropriate policy for each.

If you're only worried about it being a fs global setting, then start thinking about how to do it per inode/directory. Personally, though, I think we need to start moving all the allocation policy stuff (extsize hints, flags, etc) into a generic alloc policy xattr space otherwise we're going to run out of space in the inode core for all this alloc policy stuff...

Delayed allocation on large works just fine regardless of these parameter variations - that's the whole point of all the heuristics in the delalloc code to prevent fragmentation. IOWs, machine loading and worklaod should not significantly impact on what device large files are written to because it's rare that large files get allocated in tiny chunks by XFS.

Where mistakes are made, xfs_fsr can relocate the files appropriately. And the good part about having the metadata on SSD is that the xfs_fsr scan to fnd such files (i.e bulkstat) won't impact on the running workload significantly.

If you are doing direct IO, then it's up to the admin and application to make sure it it's not doing something silly. The usual raft of alloc policy controls like extent
size hints, preallocation and/or actually setting the rt inherit bits manually on data set directories can deal with issues here...

I think you're conflating "integrating new functionality in a generic manner" with "this is new generic functionality everyone should use". CRCs and reflink fall into the latter category, while allocation policies for rtdevs fall into the former....

ISTM that you are over-thinking the problem. :/

We should document how something can/should be used, not iterate all the cases where it should not be used because they vastly outnumber the valid use cases. I can see how useful a simple setup like Richard has described is for effcient long term storage in large scale storage environments. I think we should aim to support that cleanly and efficiently first, not try to make it into something that nobody is asking for....

Cheers,

Dave.

(*) <rant warning>

The "realtime" device isn't real time at all. It's a shit name and I hate it because it makes people think it's something that it isn't. It's just an alternative IO address space with a bound overhead (i.e. deterministic) allocator that is optimised for large contiguous data allocations. It's used for workloads are that are latency sensitive, not "real time". The filesystem is not real time capable and the IO subsystem is most definitely not real time capable. It's a crap name.

<end rant>

Dave Chinner
[email protected]

...

Er, Ok. I may be conflating the use cases between traditional rt and this one. Sorry, I'm also not explaining myself clearly wrt to my questions, but I think you manage to close in on them anyways...

...

Ok, so the above says that basically if somebody is using traditional RT, they shouldn't be using this mount option at all. That's the part that I think needs to be called out. :) If we add/document an rt-oriented mount option, we should probably explain that there are very special conditions where this should be used ("tiering" via SSD, archives to SMR, etc.). Either your workload closely matches these conditions or you shouldn't use this option.

That pretty much answers my question wrt to traditional realtime. It also seems like a red flag for a one off hack, but I digress (for now, more on this later). ;P Moving on from the traditional RT use case, this raises a similar question for those who might want to legitimately use this feature for the SSD use case: what are those conditions their workload needs to meet?

So we create a mount option that automatically assigns a file to the appropriate device based on the inode size at the time of the first physical allocation. This works fine for fb because they 1.) define a relatively small threshold of 256k and 2.) fallocate every file up front.

But a tunable is a tunable, so suppose another user comes along, thinks they otherwise match the conditions to use this feature on a DVR or something of that nature. The device has a smaller SSD, bigger HDD (the rtdev) and 512GB RAM. Files are either pretty small (kB-Mb) and should remain on the root SSD or multi-Gb and should go to the HDD, so the user sets a threshold of 1GB (another dumb example, just assume it's valid with respect to the dataset). This probably won't work and it's not obvious why to somebody who doesn't understand the implementation of this hack (because "file size at first alloc" is really a non-deterministic transient when it comes down to it). So is this feature simply not suitable for this environment? Does the user need to set a smaller threshold that's some percentage of physical RAM? This is the type of stuff I think needs to be described somewhere.

Repeat that scenario for another user who has a similar workload to fb, wants to ship off everything larger than a few MB to a spining rust rtdev, but otherwise have many concurrent writers of such files. This isn't a problem for fb because of their generally single-threaded workload, but all this user knows is we've advertised a mechanism that can be used to do big/small file tiering between an SSD and HDD. This user otherwise has no reason to know or care about the RT allocator. This is, of course, also not likely to perform as the user expects.

...

Yes, I understand. I'm not concerned about this feature being generic or literally enumerating all of the reasons not to use it. ;)

For one, I'm concerned that this may not be as useful for many users outside of fb, if any (based on the current XFS RT oriented design) [1], precisely because of the highly controlled/constrained workload requirements. Second, I think that highly constrained workload needs to be documented.

I understand that the realtime allocator has all these constraints and limitations as to where it should and should not be used. My point is that if we're adding a mount option on top that traditional RT users should never use and we call it the "file size tiering between SSD/HDD option," then I think we're opening the door for significant confusion for users to think they can accomplish what fb has without actually running into the limitations of the RT allocator.

IOW, users will come along with no care at all for RT and just want to do this cool SSD/HDD tiering thing. Hence, I think this non-rt, rt, tiering mount option needs to very specifically describe that those rt limitations still exist and the performance expectations might not be as expected unless they are met. Make sense?

Brian

[1] First, I'm not against merging this if you and others think there is a real use case (moreso because I don't care much about RT and will likely keep it disabled :). But as noted a couple times above, the more I think about this the more I think the current implementation of this is really not for anybody but fb. I'm not convinced the majority of users who would want to use this kind of tiering mechanism could do so in a way that navigates around the limitations of RT. I could have too insular a view of the potential use cases or be overestimating how limiting RT really is, of course. That's just my .02.