thinpool metadata size

While researching thinpool provisioning, it seems one of the issues is that the size of the metadata is fixed as of creation, and that if the metadata allocation fills up, your pool is corrupted? In many of the places that concern was mentioned, it was also said that extending the size of the metadata lv was a feature coming soon, but I didn't find anything confirming whether or not that functionality had been released. Is the size of the metadata lv still fixed?

My intention is to have a 4TB PV (4 x 2TB RAID10), allocated completely to a thin pool, with the metadata stored separately on a 256G RAID1 of a couple SSD's (the rest of the SSD mirror will eventually be used for dm-cache when lvm support for that is released). This storage will be used for virtualization, with fairly heavy snapshots, where there will be half a dozen or so template volumes which will be snapshotted when a new vm is created, then each of those will have some number of snapshots for backup purposes (although those snapshots will never be written to). Given such a usage pattern, is there a best practice recommendation for sizing the metadata lv? It looks like going with the defaults would result in approximately 3.6G allocated for metadata.

Thanks.

No, metadata resize is now available. But you definitely want to be using the latest kernel (there have been various fixes for this feature).

Completely exhausting all space in the metadata device will expose you to a corner case that still needs work... so best to avoid that by sizing your metadata device conservatively (larger).

We'll soon be assessing whether a fix is needed for metadata resize once all metadata space is exhausted (but last I knew we have a bug lurking in dm-persistent-data for this case).

The largest the metadata volume can be is just under 16GB. The size of the metadata device will depend on the blocksize and number of expected snapshots.

The thin_metadata_size utility should be able to provide you with an approximation for the total metadata size needed.

Oh, cool; that makes the initial allocation decision a little less critical :).

I thought I saw a thin pool metadata corruption issue fly by recently with a fix destined for 3.14, I was tentatively thinking of waiting for the 3.14 release before migrating my box to thin provisioning. I'm currently running 3.12, it looks like that was designated a long-term support kernel? Are thin provisioning (and dm-cache, as I'm going to add that to the mix as soon as lvm supports it) patches going to be backported to that, or would it be better to track mainline stable kernels as they are released?

On the grand scale of things it doesn't look like it wants that much space, so over allocation sounds like a good idea.

Interesting; for some reason I thought metadata usage was also dependent on changes between origin and snapshots. So, if you had one origin lv and 100 snapshots of it that were all identical, it would use less metadata than if you had 100 snapshots that had been written to and were all wildly divergent from each other. Evidently not though?

In regards to blocksize, from what I read the recommendation was that if you're only looking for thin provisioning, but not planning to have lots of snapshots, it's better to have a larger blocksize, whereas if you're going to have a lot of snapshots a smaller blocksize is better? I think I'm just going to stick with the default 64k for now.

A short tangent; typically when you distinguish between gigabytes and gibibytes, the former are powers of 10 and the latter powers of 2, no? 1 gigabyte = 1000000000 bytes, 1 gibibyte = 1073741824 bytes? It looks like the thin_metadata_size utility has those reversed?

# thin_metadata_size -b 64k -s 4t -m 100000 -u gigabytes
thin_metadata_size - 2.41 gigabytes estimated metadata area size

# thin_metadata_size -b 64k -s 4t -m 100000 -u gibibytes                  
thin_metadata_size - 2.59 gibibytes estimated metadata area size

# thin_metadata_size -b 64k -s 4t -m 100000 -u bytes
thin_metadata_size - 2591174656 bytes estimated metadata area size

Back on subject, I guess there's some fixed overhead, as the metadata size difference between one 1 lv and 10000 lv's is pretty tiny:

# thin_metadata_size -b 64k -s 4t -m 1 -u g
thin_metadata_size - 2.03 gigabytes estimated metadata area size

# thin_metadata_size -b 64k -s 4t -m 10000 -u g
thin_metadata_size - 2.07 gigabytes estimated metadata area size

Another power of 10 increase in volumes still only adds a bit more:

# thin_metadata_size -b 64k -s 4t -m 100000 -u g
thin_metadata_size - 2.41 gigabytes estimated metadata area size

I think I'll be pretty safe allocating 2.5G, particularly given you can now resize it later if you start getting short.

Thanks much for the info.

The important fixes for long-standing issues will be marked for stable, e.g.: http://git.kernel.org/linus/cebc2de44d3bce53 (and yes I already sent a note to stable@ to have them pull this in to 3.12-stable too)

But significant improvements will not be. The biggest recent example of this are all the improvements made in 3.14 for "out-of-data-space" mode and all the associated error handling improvements.

So if I were relegated to using upstream kernels, I'd track latest stable kernel if I could. Otherwise, I'd do my own backports -- but wouldn't expect others to support my backports.

I'm not sure if the tool tracks the rate of change. It may account for worst case of _every_ block for the provided number of thin devices _not_ being shared.

I can, and I guess I will :), it just adds a little extra volatility and work. Maybe by the time the next long-term stable after 3.12 gets picked thin provisioning and cache will have settled down enough to go with it.

Interesting, then after I added an extra order of magnitude padding for the number of snapshots, it's probably quite over allocated. But still, again 2.5G isn't very much in the scale of things.

Thanks.

The enterprise boys allocate as much as 30% of usable space to housekeeping so I'd say 2.5GB is a damn trifle.

Yes, we're still unable to resize after metadata has been completely exhausted:

(NOTE: this is with a hacked 3.14-rc6 kernel that comments out the dm_pool_metadata_set_needs_check() block in drivers/md/dm-thin.c:abort_transaction().. otherwise this test would short-circuit on the 'needs_check' flag being set and would never attempt the resize)

# dmtest run --suite thin-provisioning -n /resize_metadata_after_exhaust/
Loaded suite thin-provisioning
Started
test_resize_metadata_after_exhaust(MetadataResizeTests): metadata_size = 512k
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: reached low water mark for metadata device: sending event.
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: space map metadata: unable to allocate new metadata block
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: metadata operation 'dm_pool_alloc_data_block' failed: error = -28
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: aborting current metadata transaction
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: switching pool to read-only mode
wipe_device failed as expected
resizing...
Mar 13 12:57:44 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186096
Mar 13 12:57:44 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186096
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: switching pool to write mode
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 128 to 192 blocks
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: space map metadata: unable to allocate new metadata block
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: metadata operation 'dm_pool_commit_metadata' failed: error = -28
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: aborting current metadata transaction
Mar 13 12:57:44 rhel-storage-02 kernel: device-mapper: thin: 251:5: switching pool to read-only mode
Mar 13 12:57:44 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:44 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:44 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:44 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:45 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:45 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:45 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186111
Mar 13 12:57:45 rhel-storage-02 kernel: Buffer I/O error on device dm-6, logical block 4186104
metadata_size = 768k
wipe_device failed as expected
resizing...
Mar 13 12:57:46 rhel-storage-02 kernel: device-mapper: thin: 251:5: switching pool to write mode
Mar 13 12:57:46 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 128 to 256 blocks
Mar 13 12:57:46 rhel-storage-02 kernel: device-mapper: space map metadata: unable to allocate new metadata block
Mar 13 12:57:46 rhel-storage-02 kernel: device-mapper: thin: 251:5: metadata operation 'dm_pool_commit_metadata' failed: error = -28
Mar 13 12:57:46 rhel-storage-02 kernel: device-mapper: thin: 251:5: aborting current metadata transaction
Mar 13 12:57:46 rhel-storage-02 kernel: device-mapper: thin: 251:5: switching pool to read-only mode

Good news is online metadata resize works fine as long as metadata space hasn't been exhausted:

# dmtest run --suite thin-provisioning -n /resize_metadata_with_io/
Loaded suite thin-provisioning
Started
test_resize_metadata_with_io(MetadataResizeTests):
Mar 13 13:19:10 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 256 to 512 blocks
Mar 13 13:19:11 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 512 to 1024 blocks
Mar 13 13:19:12 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 1024 to 1536 blocks
Mar 13 13:19:14 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 1536 to 2048 blocks
Mar 13 13:19:15 rhel-storage-02 kernel: device-mapper: thin: 251:5: growing the metadata device from 2048 to 2560 blocks
.

Finished in 11.653548028 seconds.

[...]

Hmm, then it seems it would be wise to keep an eye on utilization ;). Looks like there are some munin plug-ins for lvm, hopefully one of them already has metadata usage support or can be easily extended.

Thanks.

lvm2 should/could monitor for metadata low water mark threshold events, which the kernel will trigger, much like it does for data low water mark.

But since the kernel internalizes the metadata low water mark I'm not sure if lvm2 has taken this on. Zdenek?