Skip to contentSkip to conversation

Suggestions for building new 44TB Raid5 array

DISCUSSION Marc MERLIN 8 participants 24 messages Jun 11, 2022 – Jul 6, 2022 1 message not in the archive Btrfs file system

lore
Share & tools
lore
#0 lore raw

so, my apologies to all for the thread of death that is hopefully going to be over soon. I still want to help Josef fix the tools though, hopefully we'll get that filesystem back to a mountable state.

That said, it's been over 2 months now, and I do need to get this filesystem back up from backup, so I ended up buying new drives (5x 11TiB in raid5).

Given the pretty massive corruption that happened in ways that I still can't explain, I'll make sure to turn off all the drive write caches but I think I'm not sure I want to trust bcache anymore even though I had it in writethrough mode.

Here's the Email from March, questions still apply:

Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger files (100MB to 10GB).

1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none
3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
4) cryptsetup luksOpen /dev/bcache64 dshelf1
5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Any other btrfs options I should set for format to improve reliability first and performance second?
I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?

As for bcache, I'm really thinking about droppping it, unless I'm told it should be safe to use.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#1 lore raw
Quoted 6 lines · On Fri, 10 Jun 2022 21:51:20 -0700 Marc MERLIN <[email protected]> wrote:

Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger files (100MB to 10GB).

2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none

Maybe try LVM Cache this time?

Quoted 2 lines

3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache64
4) cryptsetup luksOpen /dev/bcache64 dshelf1

What's the threat scenario for LUKS on the array?

A major one for me would be not to be having to RMA a disk with all my data still on the platters. But with RAID5, a single disk by itself would not contain easily discernible or usable data. Or if you're protecting against unauthorized access to the entire array, then never mind.

Quoted 1 line

5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Personally I have switched from Btrfs on MD to individual disks and MergerFS.

The rationale for no RAID is the simplicity and resilience of the individual single-disk filesystems, and that anything important or not easily re-obtainable is backed up anyways; so the protection from single disk failures is not as important, compared to the introduced complexity and the chance of losing the entire huge FS (like you had).

Signature · 2 lines
With respect,
Roman
#2 lore raw
Quoted 6 lines · Andrea Gelmini

Il giorno sab 11 giu 2022 alle ore 09:16 Marc MERLIN <[email protected]> ha scritto:

Quoted 2 lines

As for bcache, I'm really thinking about droppping it, unless I'm told it should be safe to use.

https://lwn.net/Articles/895266/

Mmmh, bcachefs, I was not aware of this new one. Not sure if I want to add yet another layer, esepcially if it's new.

Quoted 5 lines · Roman Mamedov
Quoted 3 lines

2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none

Maybe try LVM Cache this time?

Hard to know either way, trading one layer for another, and LVM has always seemed heavier

Quoted 4 lines
Quoted 2 lines

3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache64
4) cryptsetup luksOpen /dev/bcache64 dshelf1

What's the threat scenario for LUKS on the array?

In case my computer gets stolen, and indeed being able to recycle old drives without having to worry, is a nice bonus.

Quoted 3 lines
Quoted 1 line

5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Personally I have switched from Btrfs on MD to individual disks and MergerFS.

That gives you no redundancy if a drive disk, correct?

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#3 lore raw
Quoted 2 lines · On Sat, 11 Jun 2022 07:52:59 -0700 Marc MERLIN <[email protected]> wrote:

1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal

One more thing I wanted to mention, did you have PPL on your previous array? Or it was not implemented yet back then? I know it is supposed to protect against the write hole, which could have caused your previous FS corruption.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Personally I have switched from Btrfs on MD to individual disks and MergerFS.

That gives you no redundancy if a drive disk, correct?

Yes, but in MergerFS each file is stored entirely within a single disk, there's no striping. So only files which happened to be on the failed disk are lost and need to be restored from backups. For this it helps to keep track of what was where, with something like "find /mnt/ > date.lst" in crontab.

Signature · 2 lines
With respect,
Roman
#17 lore raw
In reply to Marc MERLIN
Quoted 26 lines · Marc MERLIN

so, my apologies to all for the thread of death that is hopefully going to be over soon. I still want to help Josef fix the tools though, hopefully we'll get that filesystem back to a mountable state.

That said, it's been over 2 months now, and I do need to get this filesystem back up from backup, so I ended up buying new drives (5x 11TiB in raid5).

Given the pretty massive corruption that happened in ways that I still can't explain, I'll make sure to turn off all the drive write caches but I think I'm not sure I want to trust bcache anymore even though I had it in writethrough mode.

Here's the Email from March, questions still apply:

Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger files (100MB to 10GB).

1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none
3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
4) cryptsetup luksOpen /dev/bcache64 dshelf1
5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1
Quoted 4 lines

Any other btrfs options I should set for format to improve reliability first and performance second?

I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?

It's default with current btrfs-progs. I'm not sure what the cutoff version is, but it doesn't matter--you can convert to v2 on first mount, which will be fast on an empty filesystem.

Quoted 2 lines

As for bcache, I'm really thinking about droppping it, unless I'm told it should be safe to use.

I would not recommend the cache in this configuration for resilience because it doesn't keep device failures in separate failure domains. Common SSD failure modes (e.g. silent data corruption, dropped writes) can be detected but not repaired, and can affect any part of the filesystem when viewed through the cache.

Unfortunately cache is only resilient with btrfs raid1 using SSD+HDD cached device pairs so that a failure of any SSD or HDD affects at most one btrfs device. That configuration works reasonably well, but you'll need a pile more disks (both HDD and SSD) to match the capacity.

btrfs raid5 of SSD+HDD devices doesn't work--it will keep all IO accesses below the cache's sequential IO size cutoff, which will wear out the SSDs too fast (in addition to the other btrfs raid5 problems). Same problem with raid10 or raid0.

I've tested btrfs with both bcache and lvmcache. I mostly use lvmcache, and have had no problems with it. bcache had problems in testing, so I've never used bcache outside of test environments.

bcache has a few sharp edges when SSD devices fail that prevent recovery with the filesystem still online. It seems to trigger service-interrupting firmware bugs in some SSD models with its access patterns compared to lvmcache (failures that are common on one vendor/model/firmware that never happen on any other vendor/model/firmware, and that occur much more often, or at all, when bcache is in use compared to when bcache is not in use).

I have not lost data with bcache when SSD corruption is not present--it survived hundreds of power-fail crash test cycles and came back after all the SSD firmware crashes in testing--but the service interruptions from crashing firmware and the inability to recover from a failed drive while keeping the filesystem online were a problem. We worked around this by using lvmcache instead.

If your IO subsystem has problems with write dropping, then it's going to be much worse with any cache. Neither bcache nor lvmcache have any sort of hardening against SSD corruption or failure. They both fail badly on SSD corruption tests even in writethrough mode.

Quoted 7 lines

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#4 lore raw
In reply to Roman Mamedov
Quoted 9 lines · Roman Mamedov
Quoted 2 lines · On Sat, 11 Jun 2022 07:52:59 -0700 Marc MERLIN <[email protected]> wrote:

1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal

One more thing I wanted to mention, did you have PPL on your previous array? Or it was not implemented yet back then? I know it is supposed to protect against the write hole, which could have caused your previous FS corruption.

Looks like I had internal bitmap instead

gargamel:~# mdadm --query --detail  /dev/md7
/dev/md7:
           Version : 1.2
     Creation Time : Sun Feb 11 20:38:30 2018
        Raid Level : raid5
        Array Size : 23441561600 (22355.62 GiB 24004.16 GB)
     Used Dev Size : 5860390400 (5588.90 GiB 6001.04 GB)
      Raid Devices : 5
     Total Devices : 5
       Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Fri Jun 10 12:09:08 2022
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Consistency Policy : bitmap

I'll switch PPL instead, thanks for that. Actually I need to migrate my other raid5 arrays to that too. It looks like it can be done at runtime.

Quoted 4 lines

Yes, but in MergerFS each file is stored entirely within a single disk, there's no striping. So only files which happened to be on the failed disk are lost and need to be restored from backups. For this it helps to keep track of what was where, with something like "find /mnt/ > date.lst" in crontab.

Right, I figured. It's not bad, but I do want no data loss if I lose a drive, so I'll take raid5.
I realize that filesystem aware raid5, like the raid5 in btrfs which I'm not sure I can really trust, still? , could lay out the files to be one per disk without striping.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#5 lore raw
In reply to Marc MERLIN
Quoted 9 lines · On Sat, 11 Jun 2022 07:52:59 -0700 Marc MERLIN <[email protected]> wrote:
Quoted 5 lines · Roman Mamedov
Quoted 3 lines

2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none

Maybe try LVM Cache this time?

Hard to know either way, trading one layer for another, and LVM has always seemed heavier

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM cache can be added to an existing LV and then removed without a trace, all without having to displace 44 TB of data for that.

And plain no-cache LVM doesn't add much in terms of being a "layer".

Signature · 2 lines
With respect,
Roman
#6 lore raw
Quoted 10 lines · Roman Mamedov
Quoted 7 lines · On Sat, 11 Jun 2022 07:52:59 -0700 Marc MERLIN <[email protected]> wrote:
Quoted 5 lines · Roman Mamedov
Quoted 3 lines

2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none

Maybe try LVM Cache this time?

Quoted 7 lines
Quoted 2 lines

Hard to know either way, trading one layer for another, and LVM has always seemed heavier

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM cache can be added to an existing LV and then removed without a trace, all without having to displace 44 TB of data for that.

Thanks. I've always felt that LVM was heavyweight and required extra steps and tools, so I've been avoiding it, but maybe that wasn't rational.
bcache by the way, you can set it up without a backing device and then use it normally without the cache layer. I think it's actually pretty similar, but you have to set it up beforehand (just like LVM)

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#7 lore raw
Quoted 3 lines · On Mon, 13 Jun 2022 10:46:40 -0700 Marc MERLIN <[email protected]> wrote:

bcache by the way, you can set it up without a backing device and then use it normally without the cache layer. I think it's actually pretty similar, but you have to set it up beforehand (just like LVM)

What I mean is bcache in this way stays bcache-without-a-cache forever, which feels odd; it still goes through the bcache code, has the module loaded, keeps the device name, etc;

Whereas in LVM caching is a completely optional side-feature, and many people would just run LVM in any case, not even thinking about enabling cache. LVM is basically "the next generation" of disk partitions, with way more features, but not much more overhead.

Signature · 2 lines
With respect,
Roman
#8 lore raw
In reply to Marc MERLIN
Quoted 19 lines · Marc MERLIN
Quoted 10 lines · Roman Mamedov
Quoted 7 lines · On Sat, 11 Jun 2022 07:52:59 -0700 Marc MERLIN <[email protected]> wrote:
Quoted 5 lines · Roman Mamedov
Quoted 3 lines

2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none

Maybe try LVM Cache this time?

Quoted 7 lines
Quoted 2 lines

Hard to know either way, trading one layer for another, and LVM has always seemed heavier

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM cache can be added to an existing LV and then removed without a trace, all without having to displace 44 TB of data for that.

Quoted 6 lines

Thanks. I've always felt that LVM was heavyweight and required extra steps and tools, so I've been avoiding it, but maybe that wasn't rational.
bcache by the way, you can set it up without a backing device and then use it normally without the cache layer. I think it's actually pretty similar, but you have to set it up beforehand (just like LVM)

You can trivially convert from lvmcache to plain LV on the fly. It's a pretty essential capability for long-term maintenance, since you can't move or resize the LV while it's cached.

If you have a LV and you want it to be cached with bcache, you can hack up the LVM configuration after the fact with https://github.com/g2p/blocks

Quoted 6 lines

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#9 lore raw
In reply to Roman Mamedov
Quoted 2 lines · Roman Mamedov

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM

In case I decide to give that a shot, what would the actual LVM command(s) look like to create a null LVM? You'd just make a single PV using the cryptestup decrypted version of the mdadm raid5 and then an LV that takes all of it, but after the fact you can modify the LV and add a cache?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#10 lore raw
Quoted 7 lines · On Mon, 13 Jun 2022 11:13:22 -0700 Marc MERLIN <[email protected]> wrote:
Quoted 2 lines · Roman Mamedov

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM

In case I decide to give that a shot, what would the actual LVM command(s) look like to create a null LVM? You'd just make a single PV using the cryptestup decrypted version of the mdadm raid5

It is a question of whether you want to cache encrypted, or plain-text data. I guess the former should be preferable, for a complete peace-of-mind against data forensics vs the cache device, but with a toll on performance, due to the need to re-decrypt even the cache hits each time.

In case of caching encrypted, it's:

mdraid => PV => LV => LUKS

                |
             (cache)

Otherwise:

mdraid => LUKS => PV => LV

                        |
                     (cache)

For the actual commands see e.g.
https://tomlankhorst.nl/setup-lvm-raid-array-mdadm-linux#set-up-logical-volume-management-lvm

Quoted 2 lines

an LV that takes all of it, but after the fact you can modify the LV and add a cache?

Yes.

Signature · 2 lines
With respect,
Roman
#11 lore raw
In reply to Marc MERLIN
Quoted 9 lines · Marc MERLIN
Quoted 2 lines · Roman Mamedov

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM

In case I decide to give that a shot, what would the actual LVM command(s) look like to create a null LVM? You'd just make a single PV using the cryptestup decrypted version of the mdadm raid5 and then an LV that takes all of it, but after the fact you can modify the LV and add a cache?

Some variables:

vg=name of VG...
device=name of cache device (SSD) PV...
base=name of existing backing (HDD) LV...
meta=meta$base
pool=pool$base

Add a cache LV to an existing LV with:

	lvcreate $vg -n $meta -L 1G $device
	lvcreate $vg -n $pool -l 90%PVS $device
	lvconvert -f --type cache-pool --poolmetadata $vg/$meta $vg/$pool
	lvconvert -f --type cache --cachepool $vg/$pool $vg/$data --cachemode writethrough

Uncache with:

	lvconvert -f --uncache $vg/$data

Note that 'lvconvert' will flush the entire cache back to the backing store during uncache at minimum IO priority, so it will take some time and can be prolonged indefinitely by a continuous IO workload on top. Also, the uncache operation will propagate any corruption in the SSD cache back to the HDD LV, even in writethrough mode.

Quoted 6 lines

Mart

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#12 lore raw

Thanks to you both for your kind help. If I'm rebulding everything, might as well future-proof it as well as possible.

Quoted 3 lines · Roman Mamedov

What I mean is bcache in this way stays bcache-without-a-cache forever, which feels odd; it still goes through the bcache code, has the module loaded, keeps the device name, etc;

Fair point. I have done that, but I see what you're saying.

Quoted 4 lines

Whereas in LVM caching is a completely optional side-feature, and many people would just run LVM in any case, not even thinking about enabling cache. LVM is basically "the next generation" of disk partitions, with way more features, but not much more overhead.

Fair enough. I have used LVM for many years, since the now defunct lvm1, and I've run through a fair amount of issues, some reliability, some performance. that was many many years ago though, so I'll take your word for it that it's a lot more lightweight and safe now.

Actually I think I stopped using LVM the same time I started using btrfs, because effectively btrfs subvolumes were close enough to LVM LVs for my use, but yes I understand that different LVs are actually different filesystems and you can do extra stuff like caching.

Actually I have another array where there were so many files and snapshots that I split it into different LVs with dm-thin so that I didn't stress the btrfs code too much (which I'm told gets unhappy when you have hundreds of snapshots).

Quoted 6 lines · Zygo Blaxell

You can trivially convert from lvmcache to plain LV on the fly. It's a pretty essential capability for long-term maintenance, since you can't move or resize the LV while it's cached.

If you have a LV and you want it to be cached with bcache, you can hack up the LVM configuration after the fact with https://github.com/g2p/blocks

Got it, thanks much.

Quoted 4 lines · Roman Mamedov

It is a question of whether you want to cache encrypted, or plain-text data. I guess the former should be preferable, for a complete peace-of-mind against data forensics vs the cache device, but with a toll on performance, due to the need to re-decrypt even the cache hits each time.

Right, I know that tradeoff. Also, LUKS makes things a bit more complicated if you want to grow the FS.

Quoted 11 lines

In case of caching encrypted, it's:

mdraid => PV => LV => LUKS

                |
             (cache)

Otherwise:

mdraid => LUKS => PV => LV

                        |
                     (cache)

Right. I'll probably do that.

Quoted 9 lines · Zygo Blaxell

Add a cache LV to an existing LV with:

	lvcreate $vg -n $meta -L 1G $device
	lvcreate $vg -n $pool -l 90%PVS $device
	lvconvert -f --type cache-pool --poolmetadata $vg/$meta $vg/$pool
	lvconvert -f --type cache --cachepool $vg/$pool $vg/$data --cachemode writethrough

Uncache with:

Quoted 7 lines
	lvconvert -f --uncache $vg/$data

Note that 'lvconvert' will flush the entire cache back to the backing store during uncache at minimum IO priority, so it will take some time and can be prolonged indefinitely by a continuous IO workload on top. Also, the uncache operation will propagate any corruption in the SSD cache back to the HDD LV, even in writethrough mode.

Thanks much for the heads up.

Best,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#13 lore raw
In reply to Zygo Blaxell
Quoted 28 lines · Zygo Blaxell
Quoted 9 lines · Marc MERLIN
Quoted 2 lines · Roman Mamedov

I'd suggest to put the LUKS volume onto an LV still (in case you don't), so you can add and remove cache just to see how it works; unlike with bcache, an LVM

In case I decide to give that a shot, what would the actual LVM command(s) look like to create a null LVM? You'd just make a single PV using the cryptestup decrypted version of the mdadm raid5 and then an LV that takes all of it, but after the fact you can modify the LV and add a cache?

Some variables:

vg=name of VG...
device=name of cache device (SSD) PV...
base=name of existing backing (HDD) LV...
meta=meta$base
pool=pool$base

Add a cache LV to an existing LV with:

	lvcreate $vg -n $meta -L 1G $device
	lvcreate $vg -n $pool -l 90%PVS $device
	lvconvert -f --type cache-pool --poolmetadata $vg/$meta $vg/$pool
	lvconvert -f --type cache --cachepool $vg/$pool $vg/$data --cachemode writethrough

Uncache with:

Quoted 8 lines
	lvconvert -f --uncache $vg/$data

Note that 'lvconvert' will flush the entire cache back to the backing store during uncache at minimum IO priority, so it will take some time and can be prolonged indefinitely by a continuous IO workload on top. Also, the uncache operation will propagate any corruption in the SSD cache back to the HDD LV, even in writethrough mode.

Personally when i setup lvmcache i always use the "all in one" command to create it.
And if i forget the syntax because the man pages are a bit unclear on how to do it exactly then i got to: https://wiki.archlinux.org/title/LVM#Cache
for a refresher.

It is something like:
lvcreate --type cache --cachemode writethrough -l 100%FREE -n root_cachepool MyVolGroup/rootvol /dev/fastdisk

i usually change -l to -L and specify the size of the cache there and the name (-n) is not to important since the LV will still be named just the same as before the cache was enabled. so this name is "just" something that shows up in for example lvs output.

only type=cache allows you to create and remove the cache with a live filesystem, the other type writecache requires filesystem and probably also the lv to be deactivated. don't remmeber exactly but was more effort to turn on/off writecache vs normal cache

#18 lore raw
In reply to Marc MERLIN
Quoted 31 lines · Marc MERLIN

so, my apologies to all for the thread of death that is hopefully going to be over soon. I still want to help Josef fix the tools though, hopefully we'll get that filesystem back to a mountable state.

That said, it's been over 2 months now, and I do need to get this filesystem back up from backup, so I ended up buying new drives (5x 11TiB in raid5).

Given the pretty massive corruption that happened in ways that I still can't explain, I'll make sure to turn off all the drive write caches but I think I'm not sure I want to trust bcache anymore even though I had it in writethrough mode.

Here's the Email from March, questions still apply:

Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger files (100MB to 10GB).

1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach

   gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
   [writethrough] writeback writearound none
3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
4) cryptsetup luksOpen /dev/bcache64 dshelf1
5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Any other btrfs options I should set for format to improve reliability first and performance second?
I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?

Quoted 5 lines

As for bcache, I'm really thinking about droppping it, unless I'm told it should be safe to use.

Thanks,
Marc

My needs are much more basic.
I have a LOT of large files. ISO images and QEMU disk images. I also have hundreds of thousands of photos.

I used different multi-disk solutions but found them all too fragile so now, last 8 years, I have used a setup that is basically 5 disks each with their own EXT4 filesystem ontop of LUKS and two additional drives to have 2 disk parity in snapraid.
Now, snapraid does not do in-line raid updates so I carefully manage how I handle the data.
Audio, photos and QEMU base images are immutable so these files are not a problem.
For VM images I have for each machine an immutable 'base' image that snapraid takes care of and I have live images based on that that are not
handled by snapraid. (qemu-img create -b base.img cow.img) If a live image goes corrupt due to a poweroutage or similar I just re-create it ontop of the latest archived base image.
As I often do stuff to the VM images that cause kernel panics, this is a very convenient way to restore them quickly and with little effort to a known good state.

If one disk has a catastrophic failure, I only lose the files on that particular disk and just have to restore them but nothing else. Now I do export them as 5 different filesystems/shares but that is just because I am too lazy to set up some kind of "merge fs".

If your use case is mostly-read and mostly-archive this might work for you too and it is VERY reliable.
Ease of mind knowing that if a single disk dies I do not have a total dataloss scenario. If the Windows VM disk dies, I just restore those images while all the other disks are still online.

It is simple, primitive and 1980 type of technology but it works. And it is reliable.

Quoted 4 lines

--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.

Home page: http://marc.merlins.org/

#20 lore raw
In reply to a message not in the archive
Quoted 1 line

I have a stupid question to ask : Why use btrfs here ? Is not mdamd+xfs good enough ?

I use btrfs for historical snapshots and btrfs send/receive remote backups.

Quoted 1 line

If you want snapshot why not use ZFS then ? i try to use btrfs myself and meet a lot of issues with it that i did not had with mdadm+ext4. Perhaps btrfs is not suited to that use (here raid5) ?

ZFS is not GPL compatible and out of tree.

Quoted 1 line

ZFS has crypt, raid5 like array and snapshot and LARC cache allready so no need to add 4 layer on it. It seems a solution for you.

It has a few of its own issues, but yes, if it were actually GPL compatible and in the linux kernel source tree, I'd consider it.

It's also owned by a company (Oracle) that has tried to sue others for billions of dollars over software patents, or even an algorithm, i.e. not a company I'm willing to trust by any means.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#21 lore raw
Quoted 9 lines

It has a few of its own issues, but yes, if it were actually GPL compatible and in the linux kernel source tree, I'd consider it.

It's also owned by a company (Oracle) that has tried to sue others for billions of dollars over software patents, or even an algorithm, i.e. not a company I'm willing to trust by any means.

well i completly understand i use btrfs for the same reason but it seems on your side that this use case is a little far from the features provided. The more layer i use the more i fear a Pise tower syndrome :)

good luck for the setup !

Signature · 2 lines
cordialement,
Ghislain
#22 lore raw
Quoted 4 lines · Ghislain Adnet

well i completly understand i use btrfs for the same reason but it seems on your side that this use case is a little far from the features provided.
The more layer i use the more i fear a Pise tower syndrome :)

I share that worry, but using ZFS simply isn't an option to me for the reasons explained.
But indeed, I removed bcache as a layer.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#23 lore raw
In reply to Marc MERLIN
Quoted 7 lines · Marc MERLIN
Quoted 1 line

I have a stupid question to ask : Why use btrfs here ? Is not mdamd+xfs good enough ?

I use btrfs for historical snapshots and btrfs send/receive remote backups.

Quoted 1 line

If you want snapshot why not use ZFS then ? i try to use btrfs myself and meet a lot of issues with it that i did not had with mdadm+ext4. Perhaps btrfs is not suited to that use (here raid5) ?

Quoted 8 lines

ZFS is not GPL compatible and out of tree.

Quoted 1 line

ZFS has crypt, raid5 like array and snapshot and LARC cache allready so no need to add 4 layer on it. It seems a solution for you.

It has a few of its own issues, but yes, if it were actually GPL compatible and in the linux kernel source tree, I'd consider it.

It's also owned by a company (Oracle)

ZFS on Linux is not owned by Oracle to my best knowledge.

https://openzfs.github.io/openzfs-docs/License.html

that has tried to sue others for

Quoted 4 lines

billions of dollars over software patents, or even an algorithm, i.e. not a company I'm willing to trust by any means.

Marc

#24 lore raw
Quoted 3 lines · Andrei Borzenkov

ZFS on Linux is not owned by Oracle to my best knowledge.

https://openzfs.github.io/openzfs-docs/License.html

Oracle bought Sun and its patent portfolio in the process, including all claims to any patents in ZFS. I simply will never trust them given what they've already done.
I did give a full talk about this issue years ago.
https://marc.merlins.org/linux/talks/Btrfs-LC2014-JP/Btrfs.pdf
and go to page #5 and
https://www.theregister.com/2010/09/09/oracle_netapp_zfs_dismiss/
basically there likely are Netapp patents in ZFS too, but I'm less worried about Netapp suing others for patents, and they did settle with Sun back in the days.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#14 lore raw
In reply to Marc MERLIN

Il giorno sab 11 giu 2022 alle ore 16:53 Marc MERLIN <[email protected]> ha scritto:

Quoted 2 lines

Mmmh, bcachefs, I was not aware of this new one. Not sure if I want to add yet another layer, esepcially if it's new.

I share the link just to say: bcache author works in a great way. Bcachefs could be the idea to replace a few layers. Not to add new one.

Just for the record.
I'm using a 120TB array with BTRFS and bcache as caching over raid1 2TB ssd. I tried same setup with LVM, but - sadly - lvm tool complained about maximum cache size (16GB max, if I remember exactly, anyway no way to use the fully 2TB).
Sad, because nowhere they mentioned this.

Played a little bit with kernel source, but eventually didn't want to risk too much with a server I want to use in production at work.

On my side, in the end, I really like cryptsetup for each HD, with mergerfs and snapraid (for my home setup).
Very handy with replacing, playing, experimenting and so on. Each time I tried one big single volume setup, eventually I regret it.

Ciao,
Gelma

#15 lore raw
Quoted 9 lines · Andrea Gelmini

Il giorno sab 11 giu 2022 alle ore 16:53 Marc MERLIN <[email protected]> ha scritto:

Quoted 2 lines

Mmmh, bcachefs, I was not aware of this new one. Not sure if I want to add yet another layer, esepcially if it's new.

I share the link just to say: bcache author works in a great way. Bcachefs could be the idea to replace a few layers. Not to add new one.

Quoted 6 lines

Just for the record.
I'm using a 120TB array with BTRFS and bcache as caching over raid1 2TB ssd. I tried same setup with LVM, but - sadly - lvm tool complained about maximum cache size (16GB max, if I remember exactly, anyway no way to use the fully 2TB).
Sad, because nowhere they mentioned this.

How many years ago was this? There's no such limit today. Here is a 500TB LV with 4TB of cache:

	# lvs -o +cachemode
	  LV          VG         Attr       LSize   Pool                Origin        Data%  Meta%  Move Log Cpy%Sync Convert CacheMode
	  lvol0       tv         Cwi-aoC--- 500.00t [lvol0_cache0_cvol] [lvol0_corig] 0.19   27.78           34.14            writeback
	  lvol0_cache tv         -wi-a-----   4.00t

There is a limit on metadata size, but you can override it in lvm.conf. Presumably if you have a 100TB+ filesystem, you also have enough RAM lying around to make the metadata size larger (it's 2.8GB at chunk size 128, but that's not unreasonable for 4096GB of cache).

Quoted 10 lines

Played a little bit with kernel source, but eventually didn't want to risk too much with a server I want to use in production at work.

On my side, in the end, I really like cryptsetup for each HD, with mergerfs and snapraid (for my home setup).
Very handy with replacing, playing, experimenting and so on. Each time I tried one big single volume setup, eventually I regret it.

Ciao,
Gelma

#16 lore raw

Il giorno mar 21 giu 2022 alle ore 07:26 Zygo Blaxell <[email protected]> ha scritto:

Quoted 2 lines

How many years ago was this? There's no such limit today. Here is a 500TB LV with 4TB of cache:

Good to know, thanks!

Quoted 4 lines

There is a limit on metadata size, but you can override it in lvm.conf. Presumably if you have a 100TB+ filesystem, you also have enough RAM lying around to make the metadata size larger (it's 2.8GB at chunk size 128, but that's not unreasonable for 4096GB of cache).

Yeap. We have at least 48GB of RAM on each server. Well, I use them all to run beesd on night (tweaked the sources) Fun part, I found this project:
https://github.com/pkolaczk/fclones

It's a recent(!) file-based deduplicator in Rust, with parallelism on every stages, hard/soft liks and range clones.
Perfect for my needs and blazing fast.

Ciao,
Gelma