xfs_fsr, performance related tweaks

Hi,

I'm using fsr extensively.

I noticed two things:

1. in xfs_fsr.c:

  if (fsx.fsx_xflags & XFS_XFLAG_NODEFRAG) {
      if (vflag)
          fsrprintf(_("%s: marked as don't defrag, ignoring\n"),
              fname);
      return(0);

}

This check should be moved above the code that performs a stat on the file, to become the first check, this will help reduce redundant stat calls for heavily defragged fileystems. A simple patch is attached for that.

2. Files for which 'No improvement will be made' should also be marked as no-defrag, this will avoid a ton of extra work in the future. In my particular use case I have many hundreds of thousands of files on each volume (xfs_fsr runs in a never ending-loop as new files are constantly being added) and once the file is written to disk it is never changed again until deletion. Optionally do this only when a special parameter is passed to fsr at command line? (that is, if you think this is not appropriate for all scenarios).

I tried to accomplish this but it proved more difficult than I thought. While digging around in the fsr code, I didn't find out how fsr marks the file as no-defrag, I'm guessing this is done in kernel code via the ioctl that swaps the extents (xfs_swapext) ... Is that correct?

I looked at how the 'io' utility marks files as no-defrag; it calls xfsctl(path, fd, XFS_IOC_FSSETXATTR, &attr); -- this requires a path, which is not available to me when fsr is run in its default mode which traverses all xfs filesystems rather than gets to run on a single file.

Is there a way to extract the path in this case? Or possibly use another way to mark the inode as no-defrag without having to need the path -- just fd? Of course this can be done by an external script which parses the output of xfs_fsr for these inodes, looks them up and marks them as such, but that's pretty messy and very inefficient. I'd really like to do this as cleanly and efficiently as possible.

I would appreciate any feedback you have on this. Please CC me as I'm not on this list.

Thanks!

But... that file could drastically change in the future, no? Just because it can't be improved now doesn't necessarily mean that it should never be revisited on subsequent runs, does it?

-Eric

I guess one could define an additional dont-defrag (or perhaps rather already-defrag) flag that is always
cleared when the file changes. That could be safely set here.

But then I'm not sure it would be worth the effort. Why would you run fsr that often that it matters?

Also I would expect that one can easily detect in many cases an defragmented file by looking at the number of extents in the inode only and that would make it equivalent to the flag. The cases where this is not the case are probably rare too.

-Andi

Hi Eric,

In my particular case, and I'm sure for many other people, files that are stored never change again until they deleted. I hinted that there could be a command line switch to turn this functionality on, as it may not be perfect for everyone's use cases.

If nobody likes this still, I would appreciate some hints on how to mark files as no-defrag from within fsr itself given that I only have the file descriptor ... A hack like looking up the descriptor in /proc/self/fd should work, but is linux specific and is too hackish in my opinion. I'd like to at least have a nice simple patch for my own uses.

Marc

Eric,

You can use the xfs_io chattr command to mark known files as nodefrag. Using the chattr -R option can be used to recurse directories.

Regards,
Barry.

Barry,

That's right but I can't do this on a filesystem that's just been defragged say a minute ago, and in the mean time 20 new files got added (I don't know what these files are... ).

The whole filesystem has to be scanned again and here comes the issue: fsr wants to try and reduce the extents of many files for which it can't do it thus incurring lots of work and load until it can even reach the new files do defrag them...

Marc

So walk the filesystem with a script that queries the number of extents in each file, and if they have a single extent then run the xfs_io on them.

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group

David,

In my first post I already said something like that can be done but it's just an ugly hack. Don't you think it would best be handled cleanly and correctly by fsr itself?

As I said earlier, fsr already issues the ioctl you're concerned about using - I'm not sure what the issue is there - if you need to do a setxattr, Just Do It.

cheers.

--
Nathan

No, I don't - if you want files not to be defragmented, then you have to set the flags yourself in some way. You have a specific need that can be solved by some scripting to describe your defrag/no defrag policy. xfs_fsr has no place is setting defrag policy; it's function is simply to find and defrag files.

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group

I agree with you. But what about files it works on, heavily, then decides they can't be defragged further? then it tries to defrag them again and again and again. And that's the end of my story.

I wouldn't mind seeing a way to tell fsr to not worry about defragging some files based on current layout; say if the avg extent in the file is

in 3 extents (not bad) fsr will try to "fix" it for you right?

-eric

That could be easily done with command line options, I think. Define the minimum extent length or number of extents we want files to have and ignore those that are outside that criteria.

Cheers,

Dave.

Dave Chinner
Principal Engineer
SGI Australian Software Group