Skip to contentSkip to conversation

LVM & fstrim behaviour (Fedora 19)

linux-lvm Jorge Fábregas 2 participants 10 messages Sep 17, 2013 – Sep 25, 2013 ext4 file system

lore
Share & tools
lore
#0 lore raw

Hi,

I'm testing fstrim on an LVM volume but it only seems to work the first time I run it (just after mounting the volume). When I run "fstrim -v /mnt" for the first time it prints all the free blocks that is trimming (almost all of the filesystem as it is empty) but subsequent runs just output "0 bytes trimmed" no matter how many files I create/sync & delete afterwards. The "issue_discards" is set in lvm.conf.

If I do this over the raw device (/dev/sda3, same ext4 filesystem) I get the output corresponding to the last deleted files every time I run fstrim.

## THIN ##

I also created a thin pool and a logical volume on that pool. If I mount this volume with "discard", I can see that TRIM is working by doing an "lvs vgthin" (the Data Usage% grows as I create files & shrinks as I delete files). However, If I mount it without the "discard" option (in order to use fstrim) I only see the data-usage reduction when I run fstrim *just* for the first time.

Any help will be appreciated.

Thanks,
Jorge

#1 lore raw
Quoted 20 lines · Jorge Fᅵbregas

Date: Tue, 17 Sep 2013 14:38:08 -0400
From: Jorge Fᅵbregas <[email protected]>
Reply-To: LVM general discussion and development <[email protected]>
To: [email protected]
Subject: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

Hi,

I'm testing fstrim on an LVM volume but it only seems to work the first time I run it (just after mounting the volume). When I run "fstrim -v /mnt" for the first time it prints all the free blocks that is trimming (almost all of the filesystem as it is empty) but subsequent runs just output "0 bytes trimmed" no matter how many files I create/sync & delete afterwards. The "issue_discards" is set in lvm.conf.

If I do this over the raw device (/dev/sda3, same ext4 filesystem) I get the output corresponding to the last deleted files every time I run fstrim.

## THIN ##

Quoted 8 lines

I also created a thin pool and a logical volume on that pool. If I mount this volume with "discard", I can see that TRIM is working by doing an "lvs vgthin" (the Data Usage% grows as I create files & shrinks as I delete files). However, If I mount it without the "discard" option (in order to use fstrim) I only see the data-usage reduction when I run fstrim *just* for the first time.

Any help will be appreciated.

Can you blktrace output for the devivce you're doing the fstrim on ? Both when you run fstrim for the first time and then when you run it again after releasing some blocks from the file system.

blktrace -d /dev/sda -o - | blkparse -i -

viz. man blktrace

Can you share how many a what size are the files you're releasing before running fstrim again ? Are you using sync after removing those files and before running fstrim (that's pretty important since blocks are not released instantly from the ext4 file system - but you can force it with sync).

I can confirm that fstrim on ext4 works as expected with loop image and I do not see behaviour described by Jorge. I'll try thinp as well to see what's going on.

Thanks!
-Lukas

Quoted 10 lines

Thanks,
Jorge

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
Quoted 52 lines · Lukďż˝ Czerner

Date: Tue, 24 Sep 2013 15:20:13 +0200 (CEST)
From: Lukďż˝ Czerner <[email protected]>
Reply-To: LVM general discussion and development <[email protected]>
To: Jorge Fďż˝bregas <[email protected]>
Cc: [email protected]
Subject: Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

Quoted 20 lines · Jorge Fďż˝bregas

Date: Tue, 17 Sep 2013 14:38:08 -0400
From: Jorge Fďż˝bregas <[email protected]>
Reply-To: LVM general discussion and development <[email protected]>
To: [email protected]
Subject: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

Hi,

I'm testing fstrim on an LVM volume but it only seems to work the first time I run it (just after mounting the volume). When I run "fstrim -v /mnt" for the first time it prints all the free blocks that is trimming (almost all of the filesystem as it is empty) but subsequent runs just output "0 bytes trimmed" no matter how many files I create/sync & delete afterwards. The "issue_discards" is set in lvm.conf.

If I do this over the raw device (/dev/sda3, same ext4 filesystem) I get the output corresponding to the last deleted files every time I run fstrim.

## THIN ##

Quoted 8 lines

I also created a thin pool and a logical volume on that pool. If I mount this volume with "discard", I can see that TRIM is working by doing an "lvs vgthin" (the Data Usage% grows as I create files & shrinks as I delete files). However, If I mount it without the "discard" option (in order to use fstrim) I only see the data-usage reduction when I run fstrim *just* for the first time.

Any help will be appreciated.

Can you blktrace output for the devivce you're doing the fstrim on ? Both when you run fstrim for the first time and then when you run it again after releasing some blocks from the file system.

blktrace -d /dev/sda -o - | blkparse -i -

viz. man blktrace

Can you share how many a what size are the files you're releasing before running fstrim again ? Are you using sync after removing those files and before running fstrim (that's pretty important since blocks are not released instantly from the ext4 file system - but you can force it with sync).

Quoted 3 lines

I can confirm that fstrim on ext4 works as expected with loop image and I do not see behaviour described by Jorge. I'll try thinp as well to see what's going on.

I've tried it with thinp targer and it works as expected:

# mkfs.ext4 /dev/lvm_pool/thin_lv
# mount /dev/lvm_pool/thin_lv /mnt/test
# dd if=/dev/zero of=/mnt/test/file bs=1M count=1024
# dd if=/dev/zero of=/mnt/test/file1 bs=1M count=1024
# dd if=/dev/zero of=/mnt/test/file2 bs=1M count=1024
# sync
# lvs
pool    lvm_pool twi-a-tz-   9,00g              35,83                        
thin_lv lvm_pool Vwi-aotz-  50,00g pool          6,45                        
# rm -f /mnt/test/file
# sync
# fstrim -v /mnt/test
/mnt/test: 47 GiB (50508640256 bytes) trimmed
# lvs
pool    lvm_pool twi-a-tz-   9.00g              33.54                        
thin_lv lvm_pool Vwi-aotz-  50.00g pool          6.04                        
# rm -f /mnt/test/file*
# sync
# fstrim -v /mnt/test
/mnt/test: 47 GiB (50508640256 bajtďż˝) trimmed
# lvs
pool    lvm_pool twi-a-tz-   9.00g              16.44                        
thin_lv lvm_pool Vwi-aotz-  50.00g pool          2.96                        
# lvm version
LVM version:     2.02.98(2) (2012-10-15)
Library version: 1.02.77 (2012-10-15)
Driver version:  4.24.0

-Lukas

Quoted 15 lines

Thanks!
-Lukas

Quoted 10 lines

Thanks,
Jorge

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#9 lore raw
In reply to Lukáš Czerner
Quoted 3 lines · Luk� Czerner

I can confirm that fstrim on ext4 works as expected with loop image and I do not see behaviour described by Jorge. I'll try thinp as well to see what's going on.

Hi Luk�,

I'm sorry I forgot to update this thread. Fstrim over LVM is working totally fine. After a bunch of updates and a restart it started working for me again. Now the issue is really with thin LVs for which I opened a bugzilla last week:

https://bugzilla.redhat.com/show_bug.cgi?id=1009546

There you would see all the details (file sizes, package versions etc).
I can recreate the problem 100% of the time.

Please let me know if you still need any more information.

Thanks,
Jorge

#3 lore raw
In reply to Lukáš Czerner

I followed your example and trims seems to work (based on the lvs output) but as you can see, there seems to be something wrong with the "fstrim -v" output (it shows as if nothing is being trimmed).

# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   3.20

lvthinpool vgthin1 twi-a-tz- 100.68g 3.20

# mount /var/lib/libvirt/images
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 97.6 GiB (104814780416 bytes) trimmed
# dd if=/dev/zero of=/var/lib/libvirt/images/file bs=1M count=1024
# dd if=/dev/zero of=/var/lib/libvirt/images/file1 bs=1M count=1024
# dd if=/dev/zero of=/var/lib/libvirt/images/file2 bs=1M count=1024
# sync
# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   6.18

lvthinpool vgthin1 twi-a-tz- 100.68g 6.18

# rm -f /var/lib/libvirt/images/file
# sync
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 0 B (0 bytes) trimmed
# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   5.19

lvthinpool vgthin1 twi-a-tz- 100.68g 5.19

# rm -f /var/lib/libvirt/images/file*
# sync
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 0 B (0 bytes) trimmed
# lvs vgthin1

lvol1 vgthin1 Vwi-aotz- 100.78g lvthinpool 3.20

lvthinpool vgthin1 twi-a-tz- 100.68g 3.20

# lvm version
  LVM version:     2.02.98(2) (2012-10-15)
  Library version: 1.02.77 (2012-10-15)
  Driver version:  4.24.0
# dd if=/dev/zero of=/var/lib/libvirt/images/tempfile bs=1M count=40000
# lvs vgthin1
  lvol1      vgthin1 Vwi-aotz- 100.78g lvthinpool   41.84

lvthinpool vgthin1 twi-a-tz- 100.68g 41.89

# rm -f /var/lib/libvirt/images/tempfile
# sync
# fstrim -v /var/lib/libvirt/images
/var/lib/libvirt/images: 0 B (0 bytes) trimmed
# lvs vgthin1

lvol1 vgthin1 Vwi-aotz- 100.78g lvthinpool 3.08

lvthinpool vgthin1 twi-a-tz- 100.68g 3.09

As you may see, this is the behaviour I'm getting now (fstrim reports nothing being trimmed although it is being trimmed). I can't recreate the original problem when I opened the bug (fstrim reporting always 124 MiB). I'm not sure if any update had to do anything with it.

#4 lore raw
Quoted 2 lines · Jorge F�bregas

I can't recreate the original problem when I opened the bug (fstrim reporting always 124 MiB). I'm not sure if any update had to do anything with it.

Arrrgh. Please discard my last email (with my sample run). I totaly forgot that I had "discard" as a mount option in /etc/fstab (I did this as fstrim wasn't properly working). I'll remove the option now and will perform the tests again.

#6 lore raw
In reply to Lukáš Czerner

ok Luk� I think I found where the problem is. I just realized that I created that filesystem without journaling (in order to increase perform for the VM disks). I created it with:

mkfs.ext4 -O ^has_journal /dev/vgthin1/lvol2

I went back and created the filesystem again (on a thin lvol) with default options and indeed fstrim works as expected. I created ANOTHER thinvol (without journaling) and fstrim doesn't work.

Is this a bug or does fstrim requires ext4 with journal by design?

My apologies for not stating this since the beginning!

Signature · 1 line
Jorge
#7 lore raw
Quoted 10 lines · Jorge Fďż˝bregas

Date: Tue, 24 Sep 2013 14:03:12 -0400
From: Jorge Fďż˝bregas <[email protected]>
To: [email protected]
Cc: Lukďż˝ Czerner <[email protected]>
Subject: Re: [linux-lvm] LVM & fstrim behaviour (Fedora 19)

ok Lukďż˝ I think I found where the problem is. I just realized that I created that filesystem without journaling (in order to increase perform for the VM disks). I created it with:

Quoted 7 lines

mkfs.ext4 -O ^has_journal /dev/vgthin1/lvol2

I went back and created the filesystem again (on a thin lvol) with default options and indeed fstrim works as expected. I created ANOTHER thinvol (without journaling) and fstrim doesn't work.

Is this a bug or does fstrim requires ext4 with journal by design?

Oh that's useful information :) Yes, it is a bug and here is untested fix.

fs/ext4/mballoc.c +2 -2
index a41e3ba..4d113ef 100644
@@ -4794,8 +4794,8 @@ do_more:
 					 " group:%d block:%d count:%lu failed"
 					 " with %d", block_group, bit, count,
 					 err);
-		}
-
+		} else
+			EXT4_MB_GRP_CLEAR_TRIMMED(e4b.bd_info);
 
 		ext4_lock_group(sb, block_group);
 		mb_clear_bits(bitmap_bh->b_data, bit, count_clusters);
Quoted 2 lines

My apologies for not stating this since the beginning!

No problem, important is that we've got to the bottom of this. Thanks for reporting it!

-Lukas

Quoted 2 lines
#8 lore raw
Quoted 2 lines · Luk� Czerner

Oh that's useful information :) Yes, it is a bug and here is untested fix.

<snip>

That's great to know and also that it wasn't something crazy on my setup :)

Thank you for the help!

Best regards,
Jorge