Skip to contentSkip to conversation

Making discard/fstrim reliable

DISCUSSION Richard W.M. Jones 4 participants 10 messages Mar 26, 2014 – Apr 10, 2014 VFS

lore
Share & tools
lore
#0 lore raw

virt-sparsify is a tool for trimming free space in virtual disk images. The new implementation uses vfs/kernel/qemu discard support. Essentially it does:

for each filesystem:
mount -o discard $fs /mnt
sync
fstrim /mnt
umount /mnt
sync

  # qemu is killed after sync returns

Although typing these commands by hand works fine, when you run them from a program the fstrim doesn't happen all the way down the stack reliably. Mostly it works, but sometimes it only trims some space from the host file.

It appears that when the host is slow / under load, the problem happens more frequently. Also it may happen more frequently on i686 than on x86-64 (possibly also due to speed of host).

The question is: What can I do to make sure the trim happens reliably, all the way down the stack, before qemu is killed?

I am testing this using the latest upstream kernel & qemu.

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
#1 lore raw

"Richard W.M. Jones" <[email protected]> writes:

Hi, Richard,

Quoted 4 lines

virt-sparsify is a tool for trimming free space in virtual disk images. The new implementation uses vfs/kernel/qemu discard support. Essentially it does:

Presumably there's a "start guest" step here that's missing?

Quoted 2 lines

for each filesystem:
mount -o discard $fs /mnt

What is $fs? Do you pass in a list of devices?

Also, you don't need to mount with -o discard in order to use fstrim. In fact, I'd recommend against doing that.

Quoted 1 line

sync

Interesting. Have you seen mount dirty inodes or something?

Quoted 9 lines

fstrim /mnt
umount /mnt
sync

  # qemu is killed after sync returns

Although typing these commands by hand works fine, when you run them from a program the fstrim doesn't happen all the way down the stack reliably. Mostly it works, but sometimes it only trims some space from the host file.

What is in the stack? Are you using qcow2 images, plain files, device mapper, anything else? Which file systems are you testing, and are they used in the host, the guest or both? How are you checking for success? Do you have a golden image you start with so that your test case is repeatable?

Quoted 3 lines

It appears that when the host is slow / under load, the problem happens more frequently. Also it may happen more frequently on i686 than on x86-64 (possibly also due to speed of host).

I don't know of any reason that any of the variables you listed would affect the reliability at all. As far as I can tell, fstrim is a synchronous ioctl. I believe the only reason space wouldn't be freed is if the fs is fragmented in such a way as to not meet the minimum trim granularity of the underlying device. Of course, there may be file system specific reasons too, I guess. Hopefully others can comment on that.

Cheers,
Jeff

Quoted 6 lines

The question is: What can I do to make sure the trim happens reliably, all the way down the stack, before qemu is killed?

I am testing this using the latest upstream kernel & qemu.

Rich.

#2 lore raw
Quoted 10 lines · Jeff Moyer

"Richard W.M. Jones" <[email protected]> writes:

Hi, Richard,

Quoted 4 lines

virt-sparsify is a tool for trimming free space in virtual disk images. The new implementation uses vfs/kernel/qemu discard support. Essentially it does:

Presumably there's a "start guest" step here that's missing?

Yup, it starts up a small appliance to do these operations.

Quoted 4 lines
Quoted 2 lines

for each filesystem:
mount -o discard $fs /mnt

What is $fs? Do you pass in a list of devices?

Yes and no. We examine the partitions, logical volumes and so on in order to get a list of mountable filesystems, and then the list is iterated over in this loop. The precise code for finding the filesystems is here:

https://github.com/libguestfs/libguestfs/blob/master/src/listfs.c#L45

^ That code is running on the host side. It issues various calls to the appliance side which are executed by code in multiple files here:

https://github.com/libguestfs/libguestfs/tree/master/daemon

Quoted 6 lines

Also, you don't need to mount with -o discard in order to use fstrim. In fact, I'd recommend against doing that.

Quoted 1 line

sync

Interesting. Have you seen mount dirty inodes or something?

The sync is actually not material here. However I included it for completeness because it is an effective workaround for another unreliability case where you delete some files before doing the fstrim, and ext4 is slow enough that the files you remove don't return space to the host. The relevant code is:

https://github.com/libguestfs/libguestfs/blob/master/daemon/fstrim.c#L53

Quoted 5 lines
Quoted 5 lines

fstrim /mnt
umount /mnt
sync

  # qemu is killed after sync returns
Quoted 7 lines
Quoted 4 lines

Although typing these commands by hand works fine, when you run them from a program the fstrim doesn't happen all the way down the stack reliably. Mostly it works, but sometimes it only trims some space from the host file.

What is in the stack? Are you using qcow2 images, plain files, device mapper, anything else?

In the test case it is recent kernel -> virtio-scsi -> qemu -> raw format local file stored on host filesystem (ext4 on the test machine).

Quoted 2 lines

Which file systems are you testing, and are they
used in the host, the guest or both?

ext4 guest and host in this case.

Quoted 1 line

How are you checking for success?

We measure the file size (stat.st_blocks) on the host during the test.

There are various thresholds which count as success (see test script linked below). In the case where it is failing it's hardly discarding any blocks, although it does discard some.

Quoted 2 lines

Do you have a golden image you start with so that your test case is repeatable?

We create images on the fly, but yes I'm confident that the test is repeatable (although that doesn't mean it is failing on every run -- it's a race condition of some sort). The test code is here:

https://github.com/libguestfs/libguestfs/blob/master/tests/discard/test-fstrim.pl

Quoted 9 lines
Quoted 3 lines

It appears that when the host is slow / under load, the problem happens more frequently. Also it may happen more frequently on i686 than on x86-64 (possibly also due to speed of host).

I don't know of any reason that any of the variables you listed would affect the reliability at all. As far as I can tell, fstrim is a synchronous ioctl. I believe the only reason space wouldn't be freed is if the fs is fragmented in such a way as to not meet the minimum trim granularity of the underlying device.

It's a freshly created filesystem so I guess it's not likely to be fragmented.

I suspect it's something to do with how we kill qemu. Requests are in flight somewhere. Just not sure how we sync "enough" to make sure everything is on the host. FWIW here is the elaborate sync dance we currently do to work around bugs present and past:

https://github.com/libguestfs/libguestfs/blob/master/daemon/sync.c#L54

Thanks,

Rich.

Signature · 3 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
#3 lore raw
Quoted 10 lines · Richard W.M. Jones
Quoted 4 lines
Quoted 2 lines

for each filesystem:
mount -o discard $fs /mnt

What is $fs? Do you pass in a list of devices?

Yes and no. We examine the partitions, logical volumes and so on in order to get a list of mountable filesystems, and then the list is iterated over in this loop. The precise code for finding the filesystems is here:

Quoted 6 lines

https://github.com/libguestfs/libguestfs/blob/master/src/listfs.c#L45

^ That code is running on the host side. It issues various calls to the appliance side which are executed by code in multiple files here:

https://github.com/libguestfs/libguestfs/tree/master/daemon

Sorry, that's a lot to take in. Can you distill this down to exactly the parts involved in the problem you're seeing? Pretend I don't know anything about libguestfs (I don't).

Quoted 8 lines
Quoted 2 lines

Do you have a golden image you start with so that your test case is repeatable?

We create images on the fly, but yes I'm confident that the test is repeatable (although that doesn't mean it is failing on every run -- it's a race condition of some sort). The test code is here:

https://github.com/libguestfs/libguestfs/blob/master/tests/discard/test-fstrim.pl

I suggest you create a golden image that you copy for each test that already has the data committed to it. The test would then just issue the rm and discard.

$g->fill (33, 10000000, "/data");

What's that do?

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

It appears that when the host is slow / under load, the problem happens more frequently. Also it may happen more frequently on i686 than on x86-64 (possibly also due to speed of host).

Quoted 8 lines
Quoted 5 lines

I don't know of any reason that any of the variables you listed would affect the reliability at all. As far as I can tell, fstrim is a synchronous ioctl. I believe the only reason space wouldn't be freed is if the fs is fragmented in such a way as to not meet the minimum trim granularity of the underlying device.

It's a freshly created filesystem so I guess it's not likely to be fragmented.

It's only 64MB in size, right? That could certainly affect things! Again, we'd need some input from file system folks, though. I'm hoping we're getting enough information for them to make progress.

Cheers,
Jeff

#4 lore raw
Quoted 8 lines · Jeff Moyer
Quoted 7 lines · Richard W.M. Jones

Yes and no. We examine the partitions, logical volumes and so on in order to get a list of mountable filesystems, and then the list is iterated over in this loop. The precise code for finding the filesystems is here:

https://github.com/libguestfs/libguestfs/blob/master/src/listfs.c#L45

Quoted 8 lines
Quoted 4 lines

^ That code is running on the host side. It issues various calls to the appliance side which are executed by code in multiple files here:

https://github.com/libguestfs/libguestfs/tree/master/daemon

Sorry, that's a lot to take in. Can you distill this down to exactly the parts involved in the problem you're seeing? Pretend I don't know anything about libguestfs (I don't).

Libguestfs runs up a small appliance, running the daemon above. The library side sends the daemon commands which the daemon runs. The commands that are run map nearly 1-1 to the libguestfs API.

The test program is this:

  # Add the disk to qemu.
  $g->add_drive ($disk, format => $format, readonly => 0, discard => "enable");
  # Run qemu.
  $g->launch ();

  # Run mkfs -t ext4 /dev/sda in the appliance.
  $g->mkfs ("ext4", "/dev/sda");
  # Run mount -o nodiscard /dev/sda /
  # [This is different from my earlier description, but it is done
  # deliberately so rm doesn't trim the space, because the test is
  # trying to test fstrim, not rm.  Discard of rm is tested in another test.]
  $g->mount_options ("nodiscard", "/dev/sda", "/");

  # Create a 10000000 byte file /data containing '!' characters.
  $g->fill (33, 10000000, "/data");
  # Sync the (guest) disk.
  $g->sync ();

  # The host file is examined here.
  my $full_size = (stat ($disk))[12];

  # Remove the large data file.  Because of -o nodiscard this won't trim.
  $g->rm ("/data");
  # Run fstrim in the appliance.
  $g->fstrim ("/");

  # This issues a sync for the guest disk, and kills qemu.
  $g->shutdown ();

  # The host file is examined again.
  my $trimmed_size = (stat ($disk))[12];
  if ($full_size - $trimmed_size < 1000) {
    # then print an error and die
  }
Quoted 3 lines

I suggest you create a golden image that you copy for each test that already has the data committed to it. The test would then just issue the rm and discard.

Sure I'll try this. Will take a little time to implement it and run the tests in various environments.

Quoted 1 line

It's only 64MB in size, right? That could certainly affect things!

So I'm guessing this is too small?

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
#6 lore raw
In reply to Richard W.M. Jones
Quoted 28 lines · Richard W.M. Jones

Date: Wed, 26 Mar 2014 20:47:08 +0000
From: Richard W.M. Jones <[email protected]>
To: [email protected]
Cc: [email protected]
Subject: Making discard/fstrim reliable

virt-sparsify is a tool for trimming free space in virtual disk images. The new implementation uses vfs/kernel/qemu discard support. Essentially it does:

for each filesystem:
mount -o discard $fs /mnt
sync
fstrim /mnt
umount /mnt
sync

  # qemu is killed after sync returns

Although typing these commands by hand works fine, when you run them from a program the fstrim doesn't happen all the way down the stack reliably. Mostly it works, but sometimes it only trims some space from the host file.

It appears that when the host is slow / under load, the problem happens more frequently. Also it may happen more frequently on i686 than on x86-64 (possibly also due to speed of host).

Quoted 6 lines

The question is: What can I do to make sure the trim happens reliably, all the way down the stack, before qemu is killed?

I am testing this using the latest upstream kernel & qemu.

Rich.

There is really no reliability to be had with discard. It's and advisory interface, not every file system implements it and when it does the implementation and hence the results varies wildly.

I'd suggest not to do things this way.

However let's take a look at your case. In order to determine why you think it's unreliable I'd need some data to back it up. How the file system looks like (an image would be great), when and how it was created, what is its size, what's the image size and what size difference do you expect. Also what file system type this is.

However if we're talking about raw file system images in files in the host, then much better solution would be to use fsck. Ext4 already has option -E discard which will send a discard down for ever free range (similarly as fstrim would do on mounted file system). I suspect that other fs utilities might have similar functionality.

Of course in order for it to work you need a layer to translate discard requests to punch holes to the underlying file system (such as loop device for example). But I think that if there is enough interest we might do this directly from e2fsck when we notice that we're running on the file rather than block device.

Also please note that mke2fs will issue the initial discard by default, so if you create the file system and then run fstrim on it with expectation that the size of a backing file will go down, you would be wrong. It was already trimmed down on file system creation time.

All that said, while discard is a interesting functionality and can be abused in many _many_ ways. It looks like what you really need is something that is currently available in fallocate(1) from util-linux package. The option to look for is --dig-holes:

-d, --dig-holes
Detect and dig holes. Makes the file sparse in-place, without using extra disk space. The minimal size of the hole depends on filesystem I/O block size (usually 4096 bytes). Also, when using this option, --keep-size is implied.

You can think of this as doing a "cp --sparse" and renaming the dest file as the origi‐ nal, without the need for extra disk space.

I am not sure whether util-linux version with this functionality has been released yet. But you can always checkout git repository:

https://github.com/karelzak/util-linux.git

I hope it helps.

Thanks!
-Lukas

#7 lore raw
Quoted 5 lines · Lukáš Czerner

All that said, while discard is a interesting functionality and can be abused in many _many_ ways. It looks like what you really need is something that is currently available in fallocate(1) from util-linux package. The option to look for is --dig-holes:

Quoted 7 lines

-d, --dig-holes
Detect and dig holes. Makes the file sparse in-place, without using extra disk space. The minimal size of the hole depends on filesystem I/O block size (usually 4096 bytes). Also, when using this option, --keep-size is implied.

You can think of this as doing a "cp --sparse" and renaming the dest file as the origi‐ nal, without the need for extra disk space.

This applies to single host files right?

virt-sparsify takes a disk image and sparsifies it, by examining the internals of the disk image to find unused space (not *zero* space, but space that's unused -- it could contain deleted files or other garbage).

Rich.

Signature · 8 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#8 lore raw
In reply to Lukáš Czerner

Il 03/04/2014 19:08, Lukáš Czerner ha scritto:

Quoted 7 lines

However if we're talking about raw file system images in files in the host, then much better solution would be to use fsck. Ext4 already has option -E discard which will send a discard down for ever free range (similarly as fstrim would do on mounted file system). I suspect that other fs utilities might have similar functionality.

Quoted 5 lines

Of course in order for it to work you need a layer to translate discard requests to punch holes to the underlying file system (such as loop device for example). But I think that if there is enough interest we might do this directly from e2fsck when we notice that we're running on the file rather than block device.

The e2fsck can also be done from within libguestfs easily. libguestfs runs within a VM so QEMU would handle the translation to hole-punching.

But fstrim is much faster than e2fsck.

From Richard's description, what seems to happen is that ext4 FITRIM scans the filesystem and prepares the discard requests; but then it sends them down to the filesystem after the ioctl has finished. Does that make any sense? And would that be considered a bug?

Paolo
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 4 lines · Paolo Bonzini

From Richard's description, what seems to happen is that ext4 FITRIM scans the filesystem and prepares the discard requests; but then it sends them down to the filesystem after the ioctl has finished. Does that make any sense? And would that be considered a bug?

That would account for it if it happened.

I'm still working on a good reproducer. It's remarkably hard to reproduce on fast / 64 bit hardware ...

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
#5 lore raw
In reply to Jeff Moyer

Just to finish off this subject.

It was (kind of) a bug in the test suite:

The test suite assumed that fstrim would trim the journal as well as the file content. I adjusted the test so it only measures the trimming of the file content, not any other metadata on the disk, and now the test works reliably.

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org