lvremove does not pass discards if volume is part of thin pool

linux-lvm vaLentin chernoZemski 4 participants 10 messages Aug 10, 2015 – Jan 26, 2016

lore

Share & tools

lore

vaLentin chernoZemski

author

2015-08-10 08:54 UTC #0 lore raw

Hi folks,

I am experiencing issues with LVM thin pool and discards that should be passed down during lvremove but they are not.

Setup looks like this:

Sparse file -> Loop device -> PV -> VG -> Thin Pool -> LV

If we mount -o discard LV and fill it with data which is later deleted sparse file shrinks back ~ to the original size.

However if we directly lvremove LV, the sparse file does not shrink back its size so we are forced to use fallocate (which is slow).

According to the docs lvremove should issue discards to the underlying device but it appears that this is not the case if LV is part of thin pool

lsblk -D shows DISC-ZERO as 0 for tpool tmeta and tdata devices and all their childs which is strange.

[root@testbed ~]# lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
|-testgroup-thingroup_tmeta (dm-33) 0
4K 4G 1
| `-testgroup-thingroup-tpool (dm-35) 0
64K 64K 0

|   |-testgroup-thingroup (dm-36)                                 0 
  64K      64K         0
|   `-testgroup-testvol (dm-37)                                   0 
  64K      64K         0
`-testgroup-thingroup_tdata (dm-34)                               0 
   4K       4G         1
  `-testgroup-thingroup-tpool (dm-35)                             0 
   64K      64K         0
    |-testgroup-thingroup (dm-36)                                 0 
   64K      64K         0
    `-testgroup-testvol (dm-37)                                   0 
   64K      64K         0

Kernel version we are using is 3.12.x.

Linux 3.2 - discard support for loop devices - http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef

Linux 3.4 - discard support for thin pool - http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=104655fd4dcebd50068ef30253a001da72e3a081

/etc/lvm/lvm.conf is configured as follows

issue_discards = 1
thin_pool_discards = "passdown"

Here is a sample script that demonstrate the issue. Note the output after the second lvremove where size of "loop-block-device-file" remains unchanged even volume is removed.

set -e
file=/root/testfile
group=testgroup
thingroup=thingroup
volume_name=testvol
volume_size=200M

grep -i disca /etc/lvm/lvm.conf | grep -v \#
rpm -qa | grep lvm2
uname -r

[[ -f ${file} ]] && unlink ${file}
truncate ${file} --size 10G
loopdev=$(losetup -f --show ${file})
pvcreate --metadatasize 1M ${loopdev}
vgcreate ${group} -s 1MiB ${loopdev}
pe_size=$(vgdisplay "/dev/${group}" | grep 'PE Size' | awk '{print $3}')
thin_size=$(echo "$(vgdisplay "/dev/${group}" | grep 'Free PE' | awk '{print $5}')*${pe_size}-180" | bc -l) lvcreate --ignoreactivationskip -Z n -L ${thin_size}m -T "/dev/${group}/${thingroup}" lvcreate --ignoreactivationskip -V${volume_size} -T "${group}/${thingroup}" -n "${volume_name}" mkfs.ext4 /dev/$group/$volume_name
sync && du -hs $file
lvs $group
lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
sync && du -hs $file
mkdir -p /mnt/tmp/
mount -o discard /dev/$group/$volume_name /mnt/tmp/
dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
mount -o discard /dev/$group/$volume_name /mnt/tmp/
rm -f /mnt/tmp/fill_file
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
lvremove -f /dev/$group/$volume_name
lvcreate --ignoreactivationskip -V${volume_size} -T "${group}/${thingroup}" -n "${volume_name}" mkfs.ext4 /dev/$group/$volume_name
lvs $group
lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
sync && du -hs $file
mkdir -p /mnt/tmp/
mount -o discard /dev/$group/$volume_name /mnt/tmp/
dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
lvremove -f $group/$volume_name
echo "==== AFTER THIS LVREMOVE size should shrink but it does not ==="
sync && du -hs $file
vgchange -Kan $group
sync && du -hs $file
losetup -d $loopdev
sync && du -hs $file
echo "==== FALLOCATE does its job well but that's not the point ===="
fallocate -d $file
sync && du -hs $file

Any assistance will be highly appreciated.

Thanks,

vaLentin

Mike Snitzer

2015-08-10 17:49 UTC #1 lore raw

Quoted 43 lines · vaLentin chernoZemski

Hi folks,
I am experiencing issues with LVM thin pool and discards that should be passed down during lvremove but they are not.
Setup looks like this:
Sparse file -> Loop device -> PV -> VG -> Thin Pool -> LV
If we mount -o discard LV and fill it with data which is later deleted sparse file shrinks back ~ to the original size.
However if we directly lvremove LV, the sparse file does not shrink back its size so we are forced to use fallocate (which is slow).
According to the docs lvremove should issue discards to the underlying device but it appears that this is not the case if LV is part of thin pool
lsblk -D shows DISC-ZERO as 0 for tpool tmeta and tdata devices and all their childs which is strange.
[root@testbed ~]# lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta
NAME DISC-ALN
DISC-GRAN DISC-MAX DISC-ZERO
|-testgroup-thingroup_tmeta (dm-33) 0
4K 4G 1
| `-testgroup-thingroup-tpool (dm-35) 0
64K 64K 0
|   |-testgroup-thingroup (dm-36)                                 0
64K      64K         0
|   `-testgroup-testvol (dm-37)                                   0
64K      64K         0
`-testgroup-thingroup_tdata (dm-34)                               0
4K       4G         1
  `-testgroup-thingroup-tpool (dm-35)                             0
64K      64K         0
    |-testgroup-thingroup (dm-36)                                 0
64K      64K         0
    `-testgroup-testvol (dm-37)                                   0
64K      64K         0

Quoted 5 lines

Kernel version we are using is 3.12.x.
Linux 3.2 - discard support for loop devices - http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
Linux 3.4 - discard support for thin pool - http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=104655fd4dcebd50068ef30253a001da72e3a081

Plese verify your kernel has this commit:
19fa1a6756e ("dm thin: fix discard support to a previously shared block")

But it doesn't look like you're using snapshots so this may not matter.

Quoted 55 lines

/etc/lvm/lvm.conf is configured as follows
issue_discards = 1
thin_pool_discards = "passdown"
Here is a sample script that demonstrate the issue. Note the output after the second lvremove where size of "loop-block-device-file" remains unchanged even volume is removed.
set -e
file=/root/testfile
group=testgroup
thingroup=thingroup
volume_name=testvol
volume_size=200M
grep -i disca /etc/lvm/lvm.conf | grep -v \#
rpm -qa | grep lvm2
uname -r
[[ -f ${file} ]] && unlink ${file}
truncate ${file} --size 10G
loopdev=$(losetup -f --show ${file})
pvcreate --metadatasize 1M ${loopdev}
vgcreate ${group} -s 1MiB ${loopdev}
pe_size=$(vgdisplay "/dev/${group}" | grep 'PE Size' | awk '{print $3}') thin_size=$(echo "$(vgdisplay "/dev/${group}" | grep 'Free PE' | awk '{print $5}')*${pe_size}-180" | bc -l)
lvcreate --ignoreactivationskip -Z n -L ${thin_size}m -T "/dev/${group}/${thingroup}"
lvcreate --ignoreactivationskip -V${volume_size} -T "${group}/${thingroup}" -n "${volume_name}"
mkfs.ext4 /dev/$group/$volume_name
sync && du -hs $file
lvs $group
lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta sync && du -hs $file
mkdir -p /mnt/tmp/
mount -o discard /dev/$group/$volume_name /mnt/tmp/
dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
mount -o discard /dev/$group/$volume_name /mnt/tmp/
rm -f /mnt/tmp/fill_file
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
lvremove -f /dev/$group/$volume_name
lvcreate --ignoreactivationskip -V${volume_size} -T "${group}/${thingroup}" -n "${volume_name}"
mkfs.ext4 /dev/$group/$volume_name
lvs $group
lsblk -D | grep ^NAME ; lsblk -D | grep -A8 $group-thingroup_tmeta sync && du -hs $file

Quoted 8 lines

mkdir -p /mnt/tmp/
mount -o discard /dev/$group/$volume_name /mnt/tmp/
dd if=/dev/zero of=/mnt/tmp/fill_file count=100 bs=1M
sync && du -hs $file
umount /mnt/tmp/
sync && du -hs $file
lvremove -f $group/$volume_name
echo "==== AFTER THIS LVREMOVE size should shrink but it does not ==="

If you do have the patch I referenced above then something else is going on. You should probably run with: lvremove -vvvv to see if lvm is actually issuing a discard. Or you could use blktrace to see if the thin device you're removing is actually receiving a discard.

vaLentin chernoZemski

author

2015-08-11 08:06 UTC #2 lore raw

Quoted 4 lines

Plese verify your kernel has this commit:
19fa1a6756e ("dm thin: fix discard support to a previously shared block")
But it doesn't look like you're using snapshots so this may not matter.

The kernel we are using includes the changes listed in that commit.

Quoted 4 lines

If you do have the patch I referenced above then something else is going on. You should probably run with: lvremove -vvvv to see if lvm is actually issuing a discard. Or you could use blktrace to see if the thin device you're removing is actually receiving a discard.

lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and thingroup_tdata

However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta and tdata sections and all its child devices.

It appears to me that for some reason device mapper or kernel (not sure which should do that) are not advertising _tpool_ tmeta and tdata devices to support discards (as confirmed by lsblk). That's why during lvremove lvm skips issuing discards on those devices.

The only references in lvremove -f -vvvv that are stating discards are those

#libdm-deptree.c:2681 Suppressed testgroup-thingroup (253:36) identical table reload.
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 3 status: 3 32/2560 1679/160928 - rw discard_passdown.

#ioctl/libdm-iface.c:1795         dm message   (253:35) OF  delete 1 [16384] (*1)
#ioctl/libdm-iface.c:1795         dm message   (253:35) OF set_transaction_id 3 4 [16384] (*1)
#ioctl/libdm-iface.c:1795         dm status   (253:35) ON   [16384] (*1)

#libdm-deptree.c:1444 Thin pool transaction id: 4 status: 4 18/2560 0/160928 - rw discard_passdown.
#activate/dev_manager.c:3127 Creating CLEAN tree for thingroup.
#activate/dev_manager.c:1789 Getting device info for testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
#ioctl/libdm-iface.c:1795 dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool OF
[16384] (*1)

#ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)

Those are the name of the devices with those minor major numbers:

testgroup-thingroup (253:36)
`-testgroup-thingroup-tpool (253:35)

    |-testgroup-thingroup_tdata (253:34)
    |  `- (7:2)
    `-testgroup-thingroup_tmeta (253:33)
       `- (7:2)
testgroup-testvol (253:37)
 `-testgroup-thingroup-tpool (253:35)
    |-testgroup-thingroup_tdata (253:34)
    |  `- (7:2)
    `-testgroup-thingroup_tmeta (253:33)
       `- (7:2)

And this is the output of lsblk -D

NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
|-testgroup-thingroup_tmeta (dm-33) 0
4K 4G 1
| `-testgroup-thingroup-tpool (dm-35) 0
64K 64K 0

|   |-testgroup-thingroup (dm-36)                                 0 
  64K      64K         0
|   `-testgroup-testvol (dm-37)                                   0 
  64K      64K         0
`-testgroup-thingroup_tdata (dm-34)                               0 
   4K       4G         1
  `-testgroup-thingroup-tpool (dm-35)                             0 
   64K      64K         0
    |-testgroup-thingroup (dm-36)                                 0 
   64K      64K         0
    `-testgroup-testvol (dm-37)                                   0 
   64K      64K         0

I am yet to test with latest kernel, libdevmapper, and lvm2 but if you have any other ideas I will be happy to hear.

Sincerely,

vaLentin

Mike Snitzer

2015-08-11 14:35 UTC #3 lore raw

Quoted 12 lines · vaLentin chernoZemski

Quoted 4 lines
Plese verify your kernel has this commit:
19fa1a6756e ("dm thin: fix discard support to a previously shared block")
But it doesn't look like you're using snapshots so this may not matter.
The kernel we are using includes the changes listed in that commit.
Quoted 4 lines
If you do have the patch I referenced above then something else is going on. You should probably run with: lvremove -vvvv to see if lvm is actually issuing a discard. Or you could use blktrace to see if the thin device you're removing is actually receiving a discard.

Quoted 5 lines

lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and thingroup_tdata
However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta and tdata sections and all its child devices.

DISC-ZERO is discard_zeroes_data. DM thinp disables that. It doesn't mean discard aren't enabled. DISC-MAX and DISC-GRAN would need to be 0 for discards to be disabled.

Quoted 4 lines

It appears to me that for some reason device mapper or kernel (not sure which should do that) are not advertising _tpool_ tmeta and tdata devices to support discards (as confirmed by lsblk). That's why during lvremove lvm skips issuing discards on those devices.

Nope, that isn't it. The pool and thin device are advertising discards. You should verify that the pool is configured to passdown discards to the underlying loop device. Run: dmsetup table You should see 'discard_passdown' -- which gets enabled by default if the thin-pool's underlying data device supports discards.

Quoted 19 lines

The only references in lvremove -f -vvvv that are stating discards are those
#libdm-deptree.c:2681 Suppressed testgroup-thingroup (253:36)
identical table reload.
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 3 status: 3
32/2560 1679/160928 - rw discard_passdown.
#ioctl/libdm-iface.c:1795 dm message (253:35) OF delete 1
[16384] (*1)
#ioctl/libdm-iface.c:1795 dm message (253:35) OF
set_transaction_id 3 4 [16384] (*1)
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 4 status: 4
18/2560 0/160928 - rw discard_passdown.
#activate/dev_manager.c:3127 Creating CLEAN tree for thingroup.
#activate/dev_manager.c:1789 Getting device info for
testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
#ioctl/libdm-iface.c:1795 dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool
OF [16384] (*1)

Quoted 4 lines

#ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)

Looking above it is clear that discard_passdown _is_ enabled.

I'll have to defer to the lvm2 developers, I thought we added explicit logging when lvm2 issues discards (as part of lvremove, etc) -- Peter, and/or Alasdair?

vaLentin chernoZemski

author

2015-08-11 14:54 UTC #4 lore raw

Thanks for getting back to me again Mike.

Quoted 18 lines · Mike Snitzer

Quoted 15 lines · vaLentin chernoZemski
Quoted 4 lines
Plese verify your kernel has this commit:
19fa1a6756e ("dm thin: fix discard support to a previously shared block")
But it doesn't look like you're using snapshots so this may not matter.
The kernel we are using includes the changes listed in that commit.
Quoted 4 lines
If you do have the patch I referenced above then something else is going on. You should probably run with: lvremove -vvvv to see if lvm is actually issuing a discard. Or you could use blktrace to see if the thin device you're removing is actually receiving a discard.
lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and thingroup_tdata

Quoted 6 lines

Quoted 2 lines
However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta and tdata sections and all its child devices.
DISC-ZERO is discard_zeroes_data. DM thinp disables that. It doesn't mean discard aren't enabled. DISC-MAX and DISC-GRAN would need to be 0 for discards to be disabled.

Understood.

Quoted 11 lines

Quoted 5 lines
It appears to me that for some reason device mapper or kernel (not sure which should do that) are not advertising _tpool_ tmeta and tdata devices to support discards (as confirmed by lsblk). That's why during lvremove lvm skips issuing discards on those devices.
Nope, that isn't it. The pool and thin device are advertising discards. You should verify that the pool is configured to passdown discards to the underlying loop device. Run: dmsetup table You should see 'discard_passdown' -- which gets enabled by default if the thin-pool's underlying data device supports discards.

passdown was already set in lvm.conf. I forgot to mention that:

Here is a snip:

grep passdown /etc/lvm/lvm.conf
# Select one of "ignore", "nopassdown", "passdown" thin_pool_discards = "passdown"

dmsetup table yields the following for testgroup:

testgroup-thingroup: 0 20598784 linear 253:35 0
testgroup-thingroup-tpool: 0 20598784 thin-pool 253:33 253:34 128 0 1 skip_block_zeroing
testgroup-thingroup_tdata: 0 20598784 linear 7:2 24576
testgroup-thingroup_tmeta: 0 20480 linear 7:2 20623360
testgroup-testvol: 0 409600 thin 253:35 1

Quoted 24 lines

Quoted 19 lines
The only references in lvremove -f -vvvv that are stating discards are those
#libdm-deptree.c:2681 Suppressed testgroup-thingroup (253:36)
identical table reload.
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 3 status: 3
32/2560 1679/160928 - rw discard_passdown.
#ioctl/libdm-iface.c:1795 dm message (253:35) OF delete 1
[16384] (*1)
#ioctl/libdm-iface.c:1795 dm message (253:35) OF
set_transaction_id 3 4 [16384] (*1)
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 4 status: 4
18/2560 0/160928 - rw discard_passdown.
#activate/dev_manager.c:3127 Creating CLEAN tree for thingroup.
#activate/dev_manager.c:1789 Getting device info for
testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
#ioctl/libdm-iface.c:1795 dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool
OF [16384] (*1)
Quoted 4 lines
#ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)

Quoted 1 line

Looking above it is clear that discard_passdown _is_ enabled.

Got that.

Quoted 3 lines

I'll have to defer to the lvm2 developers, I thought we added explicit logging when lvm2 issues discards (as part of lvremove, etc) -- Peter, and/or Alasdair?

Once again I want to mention that discard issued by deleting file in a mount point reaches blockdev and sparse file shrinks in size just as expected.

However the problem is that if I am using lvremove even there are no parent snapshots, size of sparse file remains unchanged.

Let me know if there is anything else I can try or if I can pass you additional information.

Still did not have the chance to test all latest versions of kernel dm, dm libs and lvm.

vaLentin

Peter Rajnoha

2015-08-12 07:46 UTC #5 lore raw

In reply to Mike Snitzer

Quoted 4 lines · Mike Snitzer

I'll have to defer to the lvm2 developers, I thought we added explicit logging when lvm2 issues discards (as part of lvremove, etc) -- Peter, and/or Alasdair?

Yes, we log that like (visible in the verbose -vvv output):

"Discarding X extents offset Y sectors on /dev/Z."

Also, when removing a volume, we issue this message:

# lvremove vg1/lvol0
Do you really want to remove and DISCARD logical volume lvol0? [y/n]:

For this, the "devices/issue_discards=1" must be set - it's not by default.

Signature · 1 line

Peter

Peter Rajnoha

2015-08-12 07:57 UTC #6 lore raw

Quoted 10 lines · Peter Rajnoha

Quoted 4 lines · Mike Snitzer
I'll have to defer to the lvm2 developers, I thought we added explicit logging when lvm2 issues discards (as part of lvremove, etc) -- Peter, and/or Alasdair?
Yes, we log that like (visible in the verbose -vvv output):
"Discarding X extents offset Y sectors on /dev/Z."

Quoted 8 lines

Also, when removing a volume, we issue this message:
# lvremove vg1/lvol0
Do you really want to remove and DISCARD logical volume lvol0? [y/n]: 
For this, the "devices/issue_discards=1" must be set - it's not by default.

Well, sorry, this applies for usual PVs, not thin pools (I overlooked the $subject and just saw the last question...)

Signature · 1 line

Peter

vaLentin chernoZemski

author

2015-08-19 10:38 UTC #7 lore raw

In reply to Mike Snitzer

Hello Mike, Peter,

I just tested my case with LVM2.2.02.128 with the latest kernel 4.1.6 and it still does not work.

To avoid overloading this thread with logs information was added in the following paste:

http://pastebin.com/7qxmUCyb

Let me know if you need anything else that could help us debug this issue and fix it.

Thank you in advance.

Sincerely,

vaLentin

Quoted 66 lines · Mike Snitzer

Quoted 12 lines · vaLentin chernoZemski
Quoted 4 lines
Plese verify your kernel has this commit:
19fa1a6756e ("dm thin: fix discard support to a previously shared block")
But it doesn't look like you're using snapshots so this may not matter.
The kernel we are using includes the changes listed in that commit.
Quoted 4 lines
If you do have the patch I referenced above then something else is going on. You should probably run with: lvremove -vvvv to see if lvm is actually issuing a discard. Or you could use blktrace to see if the thin device you're removing is actually receiving a discard.
Quoted 5 lines
lsblk -D shows DISC-ZERO as 1 for loop dev, thingroup_tmeta and thingroup_tdata
However it shows DISC-ZERO as 0 for thingroup-tpool in both tmeta and tdata sections and all its child devices.
DISC-ZERO is discard_zeroes_data. DM thinp disables that. It doesn't mean discard aren't enabled. DISC-MAX and DISC-GRAN would need to be 0 for discards to be disabled.
Quoted 4 lines
It appears to me that for some reason device mapper or kernel (not sure which should do that) are not advertising _tpool_ tmeta and tdata devices to support discards (as confirmed by lsblk). That's why during lvremove lvm skips issuing discards on those devices.
Nope, that isn't it. The pool and thin device are advertising discards. You should verify that the pool is configured to passdown discards to the underlying loop device. Run: dmsetup table You should see 'discard_passdown' -- which gets enabled by default if the thin-pool's underlying data device supports discards.
Quoted 19 lines
The only references in lvremove -f -vvvv that are stating discards are those
#libdm-deptree.c:2681 Suppressed testgroup-thingroup (253:36)
identical table reload.
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 3 status: 3
32/2560 1679/160928 - rw discard_passdown.
#ioctl/libdm-iface.c:1795 dm message (253:35) OF delete 1
[16384] (*1)
#ioctl/libdm-iface.c:1795 dm message (253:35) OF
set_transaction_id 3 4 [16384] (*1)
#ioctl/libdm-iface.c:1795 dm status (253:35) ON [16384] (*1)
#libdm-deptree.c:1444 Thin pool transaction id: 4 status: 4
18/2560 0/160928 - rw discard_passdown.
#activate/dev_manager.c:3127 Creating CLEAN tree for thingroup.
#activate/dev_manager.c:1789 Getting device info for
testgroup-thingroup [LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool]
#ioctl/libdm-iface.c:1795 dm info LVM-qg1G3n02Kkjm0KKnhGhzP7JfoeGiiemlrsYfP0Ti5MCUiiPOWhTxoyRlvclhd3EH-pool
OF [16384] (*1)
Quoted 4 lines
#ioctl/libdm-iface.c:1795         dm deps   (253:36) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:35) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:34) OF   [16384] (*1)
#ioctl/libdm-iface.c:1795         dm deps   (253:33) OF   [16384] (*1)
Looking above it is clear that discard_passdown _is_ enabled.
I'll have to defer to the lvm2 developers, I thought we added explicit logging when lvm2 issues discards (as part of lvremove, etc) -- Peter, and/or Alasdair?

Tomas Janousek

2016-01-22 17:18 UTC #8 lore raw

Hello all,

Quoted 7 lines · vaLentin chernoZemski

I just tested my case with LVM2.2.02.128 with the latest kernel 4.1.6 and it still does not work.
[...]
Let me know if you need anything else that could help us debug this issue and fix it.

Has there been any progress here? I'm experiencing the same issue here with kernel 4.4.0 and lvm:

lvm> version
LVM version: 2.02.138(2) (2015-12-14)
Library version: 1.02.114 (2015-12-14)
Driver version: 4.34.0

Is there perhaps some way to force the discard manually? I thought that maybe creating a new sparse file/loop device/physical volume and doing pvmove might help, but what I got instead was a file that wasn't sparse at all, actually taking more space than the original one. Is this expected?
(The only explanation I can think of is that pvmove doesn't care nor understand the on-disk format of device mapper thin-pools so it has to copy everything byte by byte. Is that it?)

Maybe creating a sufficiently large volume in the thin pool, doing a dd if=/dev/zero of=/path/to/volume/tmp/blanks and then deleting /path/to/volume/tmp/blanks would help, but could be quite slow. :-(

Regards,

Signature · 1 line

Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/

Tomas Janousek

2016-01-26 17:03 UTC #9 lore raw

Hello all again,

I will answer one of my questions myself as I just found the answer:

Quoted 11 lines · Tomas Janousek

Is there perhaps some way to force the discard manually? I thought that maybe creating a new sparse file/loop device/physical volume and doing pvmove might help, but what I got instead was a file that wasn't sparse at all, actually taking more space than the original one. Is this expected?
(The only explanation I can think of is that pvmove doesn't care nor understand the on-disk format of device mapper thin-pools so it has to copy everything byte by byte. Is that it?)
Maybe creating a sufficiently large volume in the thin pool, doing a dd if=/dev/zero of=/path/to/volume/tmp/blanks and then deleting /path/to/volume/tmp/blanks would help, but could be quite slow. :-(

This hack with dd if=/dev/zero works indeed, but there's a tool called blkdiscard in util-linux which works much better. I just call it prior to lvremove (or lxc-destroy in my case) and the space is freed.

Signature · 1 line

Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/