Skip to contentSkip to conversation

lvconvert --uncache takes hours

linux-lvm Roy Sigurd Karlsbakk 4 participants 8 messages Mar 1, 2023 – Mar 2, 2023

lore
Share & tools
lore
#0 lore raw

Hi all

Working with a friend's machine, it has lvmcache turned on with writeback. This has worked well, but now it's uncaching and it takes *hours*. The amount of cache was chosen to 100GB on an SSD not used for much else and the dataset that is being cached, is a RAID-6 set of 10x2TB with XFS on top. The system mainly works with file serving, but also has some VMs that benefit from the caching quite a bit. But then - I wonder - how can it spend hours emptying the cache like this? Most write caching I know of last only seconds or perhaps in really worst case scenarios, minutes. Since this is taking hours, it looks to me something should have been flushed ages ago.

Have I (or we) done something very stupid here or is this really how it's supposed to work?

Vennlig hilsen

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#1 lore raw
Quoted 5 lines · Roy Sigurd Karlsbakk

Hi all

Working with a friend's machine, it has lvmcache turned on with writeback. This has worked well, but now it's uncaching and it takes *hours*. The amount of cache was chosen to 100GB on an SSD not used for much else and the dataset that is being cached, is a RAID-6 set of 10x2TB with XFS on top. The system mainly works with file serving, but also has some VMs that benefit from the caching quite a bit. But then - I wonder - how can it spend hours emptying the cache like this? Most write caching I know of last only seconds or perhaps in really worst case scenarios, minutes. Since this is taking hours, it looks to me something should have been flushed ages ago.

Have I (or we) done something very stupid here or is this really how it's supposed to work?

It’s likely normal. HDDs stink at small random writes and RAID-6 makes this even worse. That said, I *strongly* recommend using three-disk RAID-1 for the cache, to match the redundancy of the RAID-6. With write-back caching, a failed cache will result in a corrupt and unrecoverable filesystem.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 10 lines · Roy Sigurd Karlsbakk

Hi all

Working with a friend's machine, it has lvmcache turned on with writeback. This has worked well, but now it's uncaching and it takes *hours*. The amount of cache was chosen to 100GB on an SSD not used for much else and the dataset that is being cached, is a RAID-6 set of 10x2TB with XFS on top. The system mainly works with file serving, but also has some VMs that benefit from the caching quite a bit. But then - I wonder - how can it spend hours emptying the cache like this? Most write caching I know of last only seconds or perhaps in really worst case scenarios, minutes. Since this is taking hours, it looks to me something should have been flushed ages ago.

Have I (or we) done something very stupid here or is this really how it's supposed to work?

Vennlig hilsen

roy

A spinning raid6 array is slow on writes (see raid6 write penalty). Because of that the array can only do about 100 write operattions/sec.

If the disk is doing other work then it only has the extra capacity so it could destage slower.

A lot depends on how big each chunk is. The lvmcache indicates the
smallest chunksize is 32k.

100G / 32k = 3 million, and at 100seeks/sec that comes to at least an hour.

Lvm bookkeeping has to also be written to the spinning disks I would think, so 2 hours if the array were idle.

Throw in a 50% baseload on the disks and you get 4 hours.

Hours is reasonable.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw

----- Original Message -----

Quoted 23 lines

From: "Roger Heflin" <[email protected]>
To: "linux-lvm" <[email protected]>
Cc: "Malin Bruland" <[email protected]>
Sent: Thursday, 2 March, 2023 01:51:08
Subject: Re: [linux-lvm] lvconvert --uncache takes hours

Quoted 16 lines · Roy Sigurd Karlsbakk

Hi all

Working with a friend's machine, it has lvmcache turned on with writeback. This has worked well, but now it's uncaching and it takes *hours*. The amount of cache was chosen to 100GB on an SSD not used for much else and the dataset that is being cached, is a RAID-6 set of 10x2TB with XFS on top. The system mainly works with file serving, but also has some VMs that benefit from the caching quite a bit. But then - I wonder - how can it spend hours emptying the cache like this? Most write caching I know of last only seconds or perhaps in really worst case scenarios, minutes. Since this is taking hours, it looks to me something should have been flushed ages ago.

Have I (or we) done something very stupid here or is this really how it's supposed to work?

Quoted 6 lines
Quoted 3 lines

Vennlig hilsen

roy

A spinning raid6 array is slow on writes (see raid6 write penalty). Because of that the array can only do about 100 write operattions/sec.

About 100 writes/second per data drive, that is. md parallilses I/O well.

Quoted 2 lines

If the disk is doing other work then it only has the extra capacity so it could destage slower.

The system was mostly idle.

Quoted 4 lines

A lot depends on how big each chunk is. The lvmcache indicates the
smallest chunksize is 32k.

100G / 32k = 3 million, and at 100seeks/sec that comes to at least an hour.

Those 100GB was on SSD, not spinning rust. Last I checked, that was the whole point with caching.

Quoted 2 lines

Lvm bookkeeping has to also be written to the spinning disks I would think, so 2 hours if the array were idle.

erm - why on earth would you do writes to hdd if you're caching it?

Quoted 3 lines

Throw in a 50% baseload on the disks and you get 4 hours.

Hours is reasonable.

As I said, the system was idle.

Vennlig hilsen

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 26 lines · Roy Sigurd Karlsbakk

----- Original Message -----

Quoted 23 lines

From: "Roger Heflin" <[email protected]>
To: "linux-lvm" <[email protected]>
Cc: "Malin Bruland" <[email protected]>
Sent: Thursday, 2 March, 2023 01:51:08
Subject: Re: [linux-lvm] lvconvert --uncache takes hours

Quoted 16 lines · Roy Sigurd Karlsbakk

Hi all

Working with a friend's machine, it has lvmcache turned on with writeback. This has worked well, but now it's uncaching and it takes *hours*. The amount of cache was chosen to 100GB on an SSD not used for much else and the dataset that is being cached, is a RAID-6 set of 10x2TB with XFS on top. The system mainly works with file serving, but also has some VMs that benefit from the caching quite a bit. But then - I wonder - how can it spend hours emptying the cache like this? Most write caching I know of last only seconds or perhaps in really worst case scenarios, minutes. Since this is taking hours, it looks to me something should have been flushed ages ago.

Have I (or we) done something very stupid here or is this really how it's supposed to work?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Vennlig hilsen

roy

A spinning raid6 array is slow on writes (see raid6 write penalty). Because of that the array can only do about 100 write operattions/sec.

About 100 writes/second per data drive, that is. md parallilses I/O well.

No. On writes you get 100 writes to the raid6 total. With reads you get 100 iops/disk. The writes by their very raid6 nature cannot be parallalized.

Each write to md requires a lot of work. At min, you have to re-read
the sector you are writing, read the parity you need to update, calculate the parity changes, and , adjust the parity and re-write any
parities that you need to change. Your other option is you might be
able to write an entire stripe, but that requires writes to all disks
+ parity calc + writes to parity. All options of writing data to
raid5/6 breakdown to iops/disk == total write iops.
The raid5/6 format requires the multiple reads and writes, and really makes it slow on writes.

Quoted 11 lines
Quoted 2 lines

If the disk is doing other work then it only has the extra capacity so it could destage slower.

The system was mostly idle.

Quoted 4 lines

A lot depends on how big each chunk is. The lvmcache indicates the
smallest chunksize is 32k.

100G / 32k = 3 million, and at 100seeks/sec that comes to at least an hour.

Those 100GB was on SSD, not spinning rust. Last I checked, that was the whole point with caching.

You are de-staging the SSD cache to spinning disks. correct? The writes to spinning disks are slow.

Quoted 5 lines
Quoted 3 lines

Lvm bookkeeping has to also be written to the spinning disks I would think, so 2 hours if the array were idle.

erm - why on earth would you do writes to hdd if you're caching it?

Once the cache is gone all LVM should be on the spinning disks.

Quoted 9 lines
Quoted 4 lines

Throw in a 50% baseload on the disks and you get 4 hours.

Hours is reasonable.

As I said, the system was idle.

Vennlig hilsen

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw
In reply to Roger Heflin

Il 2023-03-02 01:51 Roger Heflin ha scritto:

Quoted 2 lines

A spinning raid6 array is slow on writes (see raid6 write penalty). Because of that the array can only do about 100 write operattions/sec.

True. But does flushing cached data really proceed in random LBA order (as seen by HDDs), rather than trying to coalesce writes in linear fashion?

Quoted 7 lines

If the disk is doing other work then it only has the extra capacity so it could destage slower.

A lot depends on how big each chunk is. The lvmcache indicates the
smallest chunksize is 32k.

100G / 32k = 3 million, and at 100seeks/sec that comes to at least an hour.

You are off an order of magnitude: 3 millions IOP at 100 IOPs means ~30000s, so about 9 hours.

Quoted 6 lines

Lvm bookkeeping has to also be written to the spinning disks I would think, so 2 hours if the array were idle.

Throw in a 50% baseload on the disks and you get 4 hours.

Hours is reasonable.

If flushing happens in random disk order, than yes, you are bound to wait several hours indeed.

Regards.

Signature · 11 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
#6 lore raw
Quoted 9 lines · Gionatan Danti

Il 2023-03-02 01:51 Roger Heflin ha scritto:

Quoted 2 lines

A spinning raid6 array is slow on writes (see raid6 write penalty). Because of that the array can only do about 100 write operattions/sec.

True. But does flushing cached data really proceed in random LBA order (as seen by HDDs), rather than trying to coalesce writes in linear fashion?

It is a 100G cache over 16TB, so even if it flushes in order the may not be that close to each other (1 in 160).

Also if pieces are decided and added to the cached then the cache is not in order on the ssd and proper coalescing would require reading the entire cache and sorting the 3,000,000 location entries before starting the de-stage. And that complication of a de-stage is likely not been coded yet if I was just guessing, the de-stage starts at the beginning and continues to the end of the cache.

Even coded though, if the you have enough blocks cached and if the blocks spread say one or 2 on each track it would break down to having to write a tiny bit on each track with seeks between mostly breaking down to the time required to simply read/write the HD end to end. At 150MB/sec (should be about the platter speed) that would take 3.5 hours.

Quoted 11 lines
Quoted 8 lines

If the disk is doing other work then it only has the extra capacity so it could destage slower.

A lot depends on how big each chunk is. The lvmcache indicates the
smallest chunksize is 32k.

100G / 32k = 3 million, and at 100seeks/sec that comes to at least an hour.

You are off an order of magnitude: 3 millions IOP at 100 IOPs means ~30000s, so about 9 hours.

Right, I did the calc in my head and screwed it up. I thought it should have been higher but did not re-check it.

Quoted 11 lines
Quoted 7 lines

Lvm bookkeeping has to also be written to the spinning disks I would think, so 2 hours if the array were idle.

Throw in a 50% baseload on the disks and you get 4 hours.

Hours is reasonable.

If flushing happens in random disk order, than yes, you are bound to wait several hours indeed.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw

Il 2023-03-02 19:33 Roger Heflin ha scritto:

Quoted 3 lines · Gionatan Danti

It is a 100G cache over 16TB, so even if it flushes in order the may not be that close to each other (1 in 160).

Yes, but destaging in LBA order (albeit far apart) is much better than in random order.

Quoted 6 lines

Also if pieces are decided and added to the cached then the cache is not in order on the ssd and proper coalescing would require reading the entire cache and sorting the 3,000,000 location entries before starting the de-stage. And that complication of a de-stage is likely not been coded yet if I was just guessing, the de-stage starts at the beginning and continues to the end of the cache.

I would expect reordering and coalescing to happen in reasonably sized window (ie: collect 64 MB of data, reorder and flush them). At the same time, considering how lvmcache works, you are probably right: cached chunks are going to be flushed as discovered (random order).

Quoted 6 lines

Even coded though, if the you have enough blocks cached and if the blocks spread say one or 2 on each track it would break down to having to write a tiny bit on each track with seeks between mostly breaking down to the time required to simply read/write the HD end to end. At 150MB/sec (should be about the platter speed) that would take 3.5 hours.

Which (apart being a totally worst outcome) would be way better than what required for totally random IO.

Regards.

Signature · 11 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/