Skip to contentSkip to conversation

OOM executing blkdiscardon a thin lv

linux-lvm Qu Wenruo 2 participants 3 messages Nov 26, 2014 – Nov 27, 2014

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

Hi,

[Bug]
When I try to discard the whole thin lv using blkdiscard, an OOM occurs and I lost my ssh connection. TTY login also fails to start, although kernel and network still looks alive but no way to login into the system. Have to reboot.

Not sure if this is a bug in blkdiscard or lvm.

[lvs output]
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
disk1 test_vg Vwi-a-tz-- 1.00t thinpool 0.77
thinpool test_vg twi-a-tz-- 30.00g 26.28

[Reproducer]
1. Make a thin pool with 30G space.
2. Create 1T thin lv from the thin pool created in 1.
3. Do some operation to occupy some space in thin lv
4. Exec blkdiscard on the thin lv

[Related package version]
kernel-3.17.3-200.fc20.x86_64
lvm2-2.02.106-1.fc20.x86_64
util-linux-2.24.2-1.fc20.x86_64

[How to discard thin lv]
BTW, what's the proper method to discard all allocated space for a thin lv? Lvremove and lvcreate?!

Thanks,
Qu

#1 lore raw

Re: [linux-lvm] [BUG?] OOM executing blkdiscard on a thin lv

Quoted 10 lines · Qu Wenruo

Hi,

[Bug]
When I try to discard the whole thin lv using blkdiscard, an OOM occurs and I lost my ssh connection.
TTY login also fails to start, although kernel and network still looks alive but no way to login into the system.
Have to reboot.

Not sure if this is a bug in blkdiscard or lvm.

It is a bug in the kernel's blkdev_issue_discard() for allowing too many inflight bios at once. We probably should add an upper bound for how many inflight discard bios blkdev_issue_discard() will allow at any one time.

Reason why so many discard bios are being issued is that the DM thin device currently only allows a discard as large as the DM thin-pool's configured blocksize. So if you're using something like a 64K or whatever for the thin-pool blocksize then divide your 1TB thin lv by that value and you can quickly see why it takes so may discard bios to cover the entire logical address space of the 1TB thin device.

Some good news is that Joe has been working on range (multi block) discard support to allow for very large discards to be issued to DM thin devices. He is focusing on getting this working on dm-cache at the moment but once that is complete bringing it to dm-thin is next. It is a stretch goal to have that land in time for 3.19 merge window (but it is quickly approaching so it may need to wait for 3.20).

Quoted 12 lines

[lvs output]
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
disk1 test_vg Vwi-a-tz-- 1.00t thinpool 0.77
thinpool test_vg twi-a-tz-- 30.00g 26.28

[Reproducer]
1. Make a thin pool with 30G space.
2. Create 1T thin lv from the thin pool created in 1.
3. Do some operation to occupy some space in thin lv
4. Exec blkdiscard on the thin lv

Quoted 8 lines

[Related package version]
kernel-3.17.3-200.fc20.x86_64
lvm2-2.02.106-1.fc20.x86_64
util-linux-2.24.2-1.fc20.x86_64

[How to discard thin lv]
BTW, what's the proper method to discard all allocated space for a thin lv? Lvremove and lvcreate?!

Moving forward it'll be a new tool that Joe wrote called thin_trim:
https://github.com/jthornber/thin-provisioning-tools/commit/8e921580554ed91e84bb68ea32a8c2ea47ad6ff3

But that code hasn't been fully tested and included in the device-mapper-persistent-data package yet.

author
#2 lore raw

Re: [linux-lvm] [BUG?] OOM executing blkdiscard on a thin lv

-------- Original Message --------
Subject: Re: [BUG?] OOM executing blkdiscard on a thin lv
From: Mike Snitzer <[email protected]>
To: Qu Wenruo <[email protected]>
Date: 2014年11月26日 23:06

Quoted 25 lines
Quoted 10 lines · Qu Wenruo

Hi,

[Bug]
When I try to discard the whole thin lv using blkdiscard, an OOM occurs and I lost my ssh connection.
TTY login also fails to start, although kernel and network still looks alive but no way to login into the system.
Have to reboot.

Not sure if this is a bug in blkdiscard or lvm.

It is a bug in the kernel's blkdev_issue_discard() for allowing too many inflight bios at once. We probably should add an upper bound for how many inflight discard bios blkdev_issue_discard() will allow at any one time.

Reason why so many discard bios are being issued is that the DM thin device currently only allows a discard as large as the DM thin-pool's configured blocksize. So if you're using something like a 64K or whatever for the thin-pool blocksize then divide your 1TB thin lv by that value and you can quickly see why it takes so may discard bios to cover the entire logical address space of the 1TB thin device.

Quoted 6 lines

Some good news is that Joe has been working on range (multi block) discard support to allow for very large discards to be issued to DM thin devices. He is focusing on getting this working on dm-cache at the moment but once that is complete bringing it to dm-thin is next. It is a stretch goal to have that land in time for 3.19 merge window (but it is quickly approaching so it may need to wait for 3.20).

Thanks for the explain, got it.
I'll wait several kernel releases and try it again.

Quoted 17 lines
Quoted 17 lines

[lvs output]
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
disk1 test_vg Vwi-a-tz-- 1.00t thinpool 0.77
thinpool test_vg twi-a-tz-- 30.00g 26.28

[Reproducer]
1. Make a thin pool with 30G space.
2. Create 1T thin lv from the thin pool created in 1.
3. Do some operation to occupy some space in thin lv
4. Exec blkdiscard on the thin lv

[Related package version]
kernel-3.17.3-200.fc20.x86_64
lvm2-2.02.106-1.fc20.x86_64
util-linux-2.24.2-1.fc20.x86_64

Quoted 8 lines
Quoted 3 lines

[How to discard thin lv]
BTW, what's the proper method to discard all allocated space for a thin lv? Lvremove and lvcreate?!

Moving forward it'll be a new tool that Joe wrote called thin_trim:
https://github.com/jthornber/thin-provisioning-tools/commit/8e921580554ed91e84bb68ea32a8c2ea47ad6ff3

But that code hasn't been fully tested and included in the device-mapper-persistent-data package yet.

OK, so I'll use the lvremove and lvcreate until the tool gets mature.

Thanks,
Qu