Skip to contentSkip to conversation

Fast thin volume preallocation?

linux-lvm Gionatan Danti 5 participants 6 messages May 31, 2019 – Jun 5, 2019

lore
Share & tools
lore
#0 lore raw

Hi all,
doing some tests on a 4-bays, entry-level NAS/SAN system, I discovered it is entirely based on lvm thin volumes.

On configuring what it calls "thick volumes" it create a new thin logical volume and pre-allocates all space inside the new volume.

What surprised me is the speed at which this allocation happens: a 2 TB volume was allocated (ie: all data chunks were touched) in about 2 min. This immediately exclude any simple zeroing of the volume, which will require much more time.

I tried on a regular CentOS box + lvmthin with lvm zeroing disabled and, indeed, allocating all blocks inside a thin volume requires much more time. I tried both a very simple "dd if=/dev/zero of=/dev/test/thinvol bs=1M oflag=direct" and "blkdiscard -z /dev/test/thinvol".

Being curious, I found that the NAS use a binary [1] that seems to create a pattern of null writes with extremely high queue depth [2].

So, my questions:
- for what you know, are commercial NAS using some patched/custom lvmthin version which enables fast volume preallocation, or early zero rejection?
- does standard lvmthin support something similar? If not, how do you see a zero coalesce/compression/trim/whatever feature?
- can I obtain something similar by simply touching (maybe with a 512B only write) once each thin chunk?

Thanks.

[1] I am not calling it because I don't know if discovering the binary, and so the NAS vendor, is contrary to this mailing list policy.

[2] iostat -x 1 produces the following example output. Please see how *no* writes are passed on backing devices:

extended device statistics

device mgr/s mgw/s    r/s    w/s    kr/s    kw/s   size queue   wait svc_t  %b
sda        0     0  105.2    0.0   841.3     0.0    8.0   0.0    0.2 0.2   2
sdb        0     0   62.9    0.0   503.2     0.0    8.0   0.0    0.2 0.2   1
sdd        0     0  124.8    0.0   998.5     0.0    8.0   0.0    0.0 0.0   0
sdc        0     0  119.9    0.0   959.2     0.0    8.0   0.0    0.2 0.2   2

mtdblock0 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0
mtdblock1 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0
mtdblock2 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0
mtdblock3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0
mtdblock4 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0
mtdblock5 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0
mtdblock6 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0

md9        0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
md13       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
md256      0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
md322      0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
md1        0     0  412.8    0.0  3302.2     0.0    8.0   0.0    0.0 0.0   0
dm-1       0     0  412.8    0.0  3302.2     0.0    8.0   0.0    0.1 0.1   5
dm-2       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
dm-3       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
dm-4       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
dm-5       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
dm-6       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
dm-7       0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0 0.0   0
dm-8       0     0    0.0    0.0     0.0     0.0    0.0   1.0    0.0 0.0  99
dm-0       0     0    0.0    0.0     0.0     0.0    0.0   1.0    0.0 0.0  99
dm-9       0     0    0.0    0.0     0.0     0.0    0.0 345966.5    0.0 

0.0 99
dm-10 0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0

Signature · 5 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8
#1 lore raw
Quoted 2 lines · Gionatan Danti

- does standard lvmthin support something similar? If not, how do you see a zero coalesce/compression/trim/whatever feature?

There isn't such a feature as yet. With the next iteration of thin I'd like to get away from the fixed allocation block sizes that we're using which should greatly reduce the number of mappings that we need to create to provision a thick volume and so speed it up a lot.

Presumably you want a thick volume but inside a thin pool so that you can used snapshots? If so have you considered the 'external snapshot' feature?

Quoted 2 lines

- can I obtain something similar by simply touching (maybe with a 512B only write) once each thin chunk?

Yes.

- Joe

#2 lore raw

Il 03-06-2019 15:23 Joe Thornber ha scritto:

Quoted 7 lines · Gionatan Danti
Quoted 4 lines

- does standard lvmthin support something similar? If not, how do you see a
zero coalesce/compression/trim/whatever feature?

There isn't such a feature as yet.

Ok, so the NAS vendor did something custom/proprietary. This is a reason why I do *not* like such devices: how much testing and verification are done before and after such customization? Not a lot, I would say.

Quoted 6 lines

With the next iteration of thin I'd like to
get away from the fixed allocation block sizes that we're using which should greatly
reduce the number of mappings that we need to create to provision a thick volume and
so speed it up a lot.

Interesting. Do you plan to have an dynamic chunk size bound to some minumum/maximum? If so, how do you plan to tackle situations where a single big chunk should be split due to a small CoW write happening inside the big chunk? Example: a 128MB chunk-sized thin LV which is snapshotted and a small 4K writes happening at offset 32M from chunk start.

Quoted 3 lines

Presumably you want a thick volume but inside a thin pool so that you can used snapshots?
If so have you considered the 'external snapshot' feature?

Yes, in some cases they are quite useful. Still, a fast volume allocation can be an handy addition.

Signature · 5 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8
#3 lore raw
Quoted 8 lines
Quoted 4 lines

Presumably you want a thick volume but inside a thin pool so that you can used snapshots?
If so have you considered the 'external snapshot' feature?

Yes, in some cases they are quite useful. Still, a fast volume allocation can be an handy addition.

Hello.
Can I use external snapshot for fast zero allocation?
"thpool" - is lvmthin with lvm zeroing disabled

# lvcreate -n ext2T -V 2TiB --thinpool thpool VG
# lvchange --permission r VG/ext2T
# lvcreate -n zeroed_lve -s VG/ext2T --thinpool VG/thpool

Or it will be the same as zeroing enabled?
Thanks.

#4 lore raw

Dne 03. 06. 19 v 23:12 Ilia Zykov napsal(a):

Quoted 13 lines
Quoted 8 lines
Quoted 4 lines

Presumably you want a thick volume but inside a thin pool so that you can used snapshots?
If so have you considered the 'external snapshot' feature?

Yes, in some cases they are quite useful. Still, a fast volume allocation can be an handy addition.

Hello.
Can I use external snapshot for fast zero allocation?
"thpool" - is lvmthin with lvm zeroing disabled

Quoted 6 lines
# lvcreate -n ext2T -V 2TiB --thinpool thpool VG
# lvchange --permission r VG/ext2T
# lvcreate -n zeroed_lve -s VG/ext2T --thinpool VG/thpool

Or it will be the same as zeroing enabled?

Hi

It's not clear what do you want to achieve.

External origin for thin-volume is always a read-only LV.

External origin can be of any type - i.e. even thinLV from another thin-pool and single origin can be used for number of thin volumes with external origin.

For now you CANNOT 'merge' such thinLV with external origin - so the only way to get 'writable' origin back is to drop all thins that are using it as external origin.

Now - how the 'zeroing' plays the role here ?

Zeroing is property of thin-pool that serves thinLV. Unprovisioned blocks always do return 'zeroes'.

When block is provisioned - unwritten portions needs to be zeroed (depending on zeroing setting) - the bigger the block the bigger the chance the zeroing will take more time - lvm2 does not recommend using thin-pool with zeroing and chunksize bigger then 512KB.

Most modern filesystem keep track of written pieces of a block device themselves, so zeroing has almost no practical use.

Regards

Zdenek

#5 lore raw
In reply to Gionatan Danti
Quoted 7 lines
Quoted 3 lines

Presumably you want a thick volume but inside a thin pool so that you can used snapshots?
If so have you considered the 'external snapshot' feature?

Yes, in some cases they are quite useful. Still, a fast volume allocation can be an handy addition.

Hello.
Can I use external snapshot for fast zero allocation?
"thpool" - is lvmthin with lvm zeroing disabled

# lvcreate -n ext2T -V 2TiB --thinpool thpool VG
# lvchange --permission r VG/ext2T
# lvcreate -n zeroed_lve -s VG/ext2T --thinpool VG/thpool

Or it will be the same as zeroing enabled?
Thanks.

----