Skip to contentSkip to conversation

Extreme fragmentation ho!

DISCUSSION Chris Dunlop 3 participants 5 messages Dec 21, 2020 – Dec 30, 2020

lore
Share & tools
lore
#0 lore raw

Hi,

I have a 2T file fragmented into 841891 randomly placed extents. It takes 4-6 minutes (depending on what else the filesystem is doing) to delete the file. This is causing a timeout in the application doing the removal, and hilarity ensues.

The fragmentation is the result of reflinking bits and bobs from other files into the subject file, so it's probably unavoidable.

The file is sitting on XFS on LV on a raid6 comprising 6 x 5400 RPM HDD:

# xfs_info /home
meta-data=/dev/mapper/vg00-home  isize=512    agcount=32, agsize=244184192 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1
data     =                       bsize=4096   blocks=7813893120, imaxpct=5
         =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

I'm guessing the time taken to remove is not unreasonable given the speed of the underlying storage and the amount of metadata involved. Does my guess seem correct?

I'd like to do some experimentation with a facsimile of this file, e.g. try the remove on different storage subsystems, and/or with a external fast journal etc., to see how they compare.

What is the easiest way to recreate a similarly (or even better, identically) fragmented file?

One way would be to use xfs_metadump / xfs_mdrestore to create an entire copy of the original filesystem, but I'd really prefer not taking the original fs offline for the time required. I also don't have the space to restore the whole fs but perhaps using lvmthin can address the restore issue, at the cost of a slight(?) performance impact due to the extra layer.

Is it possible to using the output of xfs_bmap on the original file to drive ...something, maybe xfs_io, to recreate the fragmentation? A naive test using xfs_io pwrite didn't produce any fragmentation - unsurprisingly, given the effort XFS puts into reducing fragmentation.

Cheers,

Chris

#1 lore raw
Quoted 32 lines · Chris Dunlop

Hi,

I have a 2T file fragmented into 841891 randomly placed extents. It takes 4-6 minutes (depending on what else the filesystem is doing) to delete the file. This is causing a timeout in the application doing the removal, and hilarity ensues.

The fragmentation is the result of reflinking bits and bobs from other files into the subject file, so it's probably unavoidable.

The file is sitting on XFS on LV on a raid6 comprising 6 x 5400 RPM HDD:

# xfs_info /home
meta-data=/dev/mapper/vg00-home  isize=512    agcount=32, agsize=244184192 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1
data     =                       bsize=4096   blocks=7813893120, imaxpct=5
         =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

I'm guessing the time taken to remove is not unreasonable given the speed of the underlying storage and the amount of metadata involved. Does my guess seem correct?

I'd like to do some experimentation with a facsimile of this file, e.g. try the remove on different storage subsystems, and/or with a external fast journal etc., to see how they compare.

Quoted 9 lines

What is the easiest way to recreate a similarly (or even better, identically) fragmented file?

One way would be to use xfs_metadump / xfs_mdrestore to create an entire copy of the original filesystem, but I'd really prefer not taking the original fs offline for the time required. I also don't have the space to restore the whole fs but perhaps using lvmthin can address the restore issue, at the cost of a slight(?) performance impact due to the extra layer.

Note that xfs_metadump doesn't include file data, only metadata, so it might actually be the most time and space efficient way to replicate the large file. You would need a similarly sized block device to restore to and would not be able to change filesystem geometry and whatnot. The former can be easily worked around by restoring the image to a file on a smaller fs though, which may or may not interfere with whatever performance testing you're doing.

Quoted 5 lines

Is it possible to using the output of xfs_bmap on the original file to drive ...something, maybe xfs_io, to recreate the fragmentation? A naive test using xfs_io pwrite didn't produce any fragmentation - unsurprisingly, given the effort XFS puts into reducing fragmentation.

fstests has a helper program (xfstests-dev/src/punch-alternating) that helps create fragmented files. IIRC, you create a fully allocated file in advance and it will punch out alternating ranges based on the offset/size parameters. You might have to wait a bit for it to complete, but it's pretty easy to use (and you can always create a metadump image from the result for quicker restoration).

Yet another option might be to try a write workload that attempts to defeat the allocator heuristics. For example, do direct I/O or falloc requests in reverse order and in small sizes across a file. xfs_io has a couple flags you can pass to pwrite (i.e., -B, -R) to make that easier, but that's more manual and you may have to play around with it to get the behavior you want.

Brian

Quoted 4 lines

Cheers,

Chris

#2 lore raw
In reply to Chris Dunlop
Quoted 6 lines · Chris Dunlop

Hi,

I have a 2T file fragmented into 841891 randomly placed extents. It takes 4-6 minutes (depending on what else the filesystem is doing) to delete the file. This is causing a timeout in the application doing the removal, and hilarity ensues.

~3,000 extents/s being removed, with reflink+rmap mods being made for every extent. Seems a little slow compared to what I typically see, but...

Quoted 4 lines

The fragmentation is the result of reflinking bits and bobs from other files into the subject file, so it's probably unavoidable.

The file is sitting on XFS on LV on a raid6 comprising 6 x 5400 RPM HDD:

... probably not that unreasonable for pretty much the slowest storage configuration you can possibly come up with for small, metadata write intensive workloads.

Quoted 12 lines
# xfs_info /home
meta-data=/dev/mapper/vg00-home  isize=512    agcount=32, agsize=244184192 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1
data     =                       bsize=4096   blocks=7813893120, imaxpct=5
         =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Quoted 3 lines

I'm guessing the time taken to remove is not unreasonable given the speed of the underlying storage and the amount of metadata involved. Does my guess seem correct?

Yup.

Quoted 3 lines

I'd like to do some experimentation with a facsimile of this file, e.g. try the remove on different storage subsystems, and/or with a external fast journal etc., to see how they compare.

I think you'll find a limit at ~20,000 extents/s, regardless of your storage subsystem. Once you take away IO latency, it's basically single threaded and CPU bound so performance is largely dependent on how fast your CPUs are. IOWs, the moment you move to SSDs, it will be CPU bound and still take a minute or two to remove all the extents....

Quoted 2 lines

What is the easiest way to recreate a similarly (or even better, identically) fragmented file?

Just script xfs_io to reflink random bits and bobs from other files into a larger file?

Quoted 5 lines

One way would be to use xfs_metadump / xfs_mdrestore to create an entire copy of the original filesystem, but I'd really prefer not taking the original fs offline for the time required. I also don't have the space to restore the whole fs but perhaps using lvmthin can address the restore issue, at the cost of a slight(?) performance impact due to the extra layer.

Easiest, most space efficient way is to mdrestore to a file (ends up sparse, containing only metadata), mount it via loopback.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 6 lines · Dave Chinner
Quoted 1 line · Chris Dunlop

The file is sitting on XFS on LV on a raid6 comprising 6 x 5400 RPM HDD:

... probably not that unreasonable for pretty much the slowest storage configuration you can possibly come up with for small, metadata write intensive workloads.

[ Chris grimaces and glances over at the 8+3 erasure-encoded ceph rbd
sitting like a pitch drop experiment in the corner. ]

Speaking of slow storage and metadata write intensive workloads, what's the reason reflinks with a realtime device isn't supported? That was one approach I wanted to try, to get the metadata ops running on a small fast storage with the bulk data sitting on big slow bulk storage. But:

# mkfs.xfs -m reflink=1 -d rtinherit=1 -r rtdev=/dev/fast /dev/slow
reflink not supported with realtime devices

My naive thought was a reflink was probably "just" a block range referenced from multiple places, and probably a refcount somewhere. It seems like it should be possible to have the range, references and refcount sitting on the fast storage pointing to the actual data blocks on the slow storage.

Quoted 5 lines
Quoted 2 lines

What is the easiest way to recreate a similarly (or even better, identically) fragmented file?

Just script xfs_io to reflink random bits and bobs from other files into a larger file?

Thanks - that did it.

Cheers,

Chris

#4 lore raw
Quoted 10 lines · Chris Dunlop
Quoted 6 lines · Dave Chinner
Quoted 1 line · Chris Dunlop

The file is sitting on XFS on LV on a raid6 comprising 6 x 5400 RPM HDD:

... probably not that unreasonable for pretty much the slowest storage configuration you can possibly come up with for small, metadata write intensive workloads.

[ Chris grimaces and glances over at the 8+3 erasure-encoded ceph rbd sitting like a pitch drop experiment in the corner. ]

I would have thought that should be able to do more than the 20 IOPS the raid6 above will do on random 4kB writes.... :)

Quoted 7 lines

Speaking of slow storage and metadata write intensive workloads, what's the reason reflinks with a realtime device isn't supported? That was one approach I wanted to try, to get the metadata ops running on a small fast storage with the bulk data sitting on big slow bulk storage. But:

# mkfs.xfs -m reflink=1 -d rtinherit=1 -r rtdev=/dev/fast /dev/slow
reflink not supported with realtime devices

Yup, the realtime device is a pure data device, so all it's metadata is held externally to the device (i.e. it is held in the "data device", not the RT device). IOWs, it's a completely separate filesystem implementation within XFS, and so requires independent functional extensions to support reflink + rmap...

Quoted 4 lines

My naive thought was a reflink was probably "just" a block range referenced from multiple places, and probably a refcount somewhere. It seems like it should be possible to have the range, references and refcount sitting on the fast storage pointing to the actual data blocks on the slow storage.

Yes, it is possible, but the current reflink implementation is based on allocation group internal structures (rmap is the same), and the realtime device doesn't have these. Hence there are new metadata structures that need to be added (refcount btrees rooted in inodes, not fixed location AG headers) and a bunch of new supporting code to be written. Largely Darrick has done this already, it's just a problem of review bandwidth and validation time:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink-extsize

(which also includes realtime rmap support, a whole new internal metadata inode directory to index all the new inode btrees for the rt device, etc)

It's a pretty large chunk of shiny new code....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]