Bcache version 12
Subsystems (2)
Hi here.
Quoted 60 lines · ----- "Dan J Williams
Quoted 1 line · Kent Overstreet
Quoted 1 line
Quoted 1 line
Cache devices have a basically identical superblock as backing
devices
Quoted 1 line
Quoted 1 line
Quoted 1 line
though, and some of the registration code is shared, but cache
devices
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
don't correspond to any block devices.
Just like a raid0 is a virtual creation from two block devices?
Or
Quoted 7 lines
Quoted 1 line
some other meaning of "don't correspond"?
No.
Remember, you can hang multiple backing devices off a cache.
Each backing device shows up as as a new block device - i.e. if
you're
Quoted 3 lines
caching
/dev/sdb, you now use it as/dev/bcache0.But the SSD doesn't belong to any of those
/dev/bcacheNdevices.So to clarify I read that as "it belongs to all of them". The ssd (
/dev/sda, for example) can cache the contents of N block devices, and
to get to the cached version of each of those you go through /dev/bcache[0..N]. The problem you perceive is that an md device requires a 1:1 mapping of member devices to md devices. So if we had/dev/sdaand/dev/sdbin a cache configuration (/dev/md0) your concern
is that if we simultaneously wanted a/dev/md1that caches/dev/sdaand/dev/sdcthat md would not be able to handle it.Is that the right interpretation?
I assume
/dev/sdain the example would have some bcache-logical partitions to delineate the/dev/sdband/dev/sdccache data? Which sounds similar to the logical partitions md handles now for external metadata. I'm not proposing that cache-state metadata could be handled in userspace it's too integral to the i/o path, just pointing out that having/dev/sdabe a member of both/dev/md0and/dev/md1is possible.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
A cache set is a set of cache devices - i.e. SSDs. The primary motivitation for cache sets (as distinct from just caches) is to
have
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the ability to mirror only dirty data, and not clean data.
i.e. if you're doing writeback caching of a raid6, your ssd is
now a
Quoted 1 line
Quoted 1 line
Quoted 1 line
single point of failure. You could use raid1 SSDs, but most of
the data
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the cache is clean, so you don't need to mirror that... just
the
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
dirty data.
...but you only incur that "mirror clean data" penalty once, and
then
Quoted 2 lines
Quoted 1 line
it's just a normal raid1 mirroring writes, right?
Quoted 8 lines
Quoted 1 line
No idea what you mean...
/dev/md1is a slow raid5 and/dev/md0is a raid1 of two ssds. Once/dev/md0is synced the only mirror traffic is for incoming cache-dirtying writes and cache-clean read allocations. We agree about incoming dirty-data, but you are saying you don't want to mirror
read allocations?
Just one visualization of my understand of bcache set with mirroring only dirty data: http://147.175.167.212/~luvar/bcache/bcacheSSDset.png . If I am not wrong, read alocations are for example green and blue data. Dirty allocations is red one and it should be mirrored across all ssds in mirror set to provide ssd fail security.
On the other hand, greed, blue... data are backed up on raid6 and it is nod needed to mirror them across ssd set. They should be only on one ssd to provide read speedup.
Hmmm (sci-fi), if read allocations (not dirty data) will be mirrored in ssds set, they could be used to improve cache read speed, sacrificing some ssd space. It would be great if cache algorithm can mark really hot data to be mirrored for speed reading...
Quoted 119 lines
Quoted 2 lines
Quoted 2 lines
See, if these things were just md devices multiple cache device
would
Quoted 1 line
Quoted 1 line
already be "done", or at least on its way by just stacking md
devices.
Quoted 4 lines
Quoted 1 line
Where "done" is probably an oversimplification.
No, it really wouldn't save us anything. If all we wanted to do was mirror everything, there'd be no point in implementing multiple
cache
Quoted 4 lines
device support, and you'd just use bcache on top of md. We're implementing something completely new!
You read what I said about only mirroring dirty data... right?
I did but I guess I did not fully grok it.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
In any case it certainly could be modelled in md - and if the
modelling were
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
not elegant (e.g. even device numbers for backing devices, odd
device numbers
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
for cache devices) we could "fix" md to make it more elegant.
But we've no reason to create block devices for caches or have a
1:1
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
mapping - that'd be a serious step backwards in functionality.
I don't follow that... there's nothing that prevents having
multiple
Quoted 3 lines
Quoted 1 line
superblocks per cache array.
Multiple... superblocks? Do you mean partitioning up the cache, or
do
Quoted 1 line
you mean creating multiple block devices for a cache? Either way
it's a
Quoted 5 lines
silly hack.
Quoted 3 lines
A couple reasons I'm probing the md angle.
1/ Since the backing devices are md devices it would be nice if
all
Quoted 2 lines
Quoted 2 lines
the user space assembly logic that has seeped into udev and dracut could be re-used for assembling bcache devices. As it stands it
seems
Quoted 4 lines
Quoted 2 lines
bcache relies on in-kernel auto-assembly, which md has discouraged with the v1 superblock.
md was doing in kernel probing, which bcache does not do. What
bcache is
Quoted 2 lines
doing is centralizing all the code that touches the on disk superblock/metadata. You want to change something in the superblock
-
Quoted 1 line
you just have to tell the kernel to do it for you. Otherwise not
only
Quoted 3 lines
would there be duplication of code, it'd be impossible to do safely without races or the userspace code screwing something up; only the kernel knows and controls the state of everything.
Makes sense but there is a difference between the metadata that specifies the configuration and the metadata that tracks the state of the cache. If that distinction is made then userspace can tell the kernel to run a block cache of blockdevA and blockdevB and the kernel only needs to handle the cache state metadata.
Quoted 1 line
Or do you expect the ext4 superblock to be managed in normal
operation
Quoted 1 line
by userspace tools?
No.
Quoted 2 lines
Quoted 2 lines
We even have nascent GUI support in gnome-disk-utility it would be nice to harness some of that
enabling
Quoted 3 lines
Quoted 1 line
momentum for this.
I've got nothing against standardizing the userspace interfaces to
make
Quoted 1 line
life easier for things like gnome-disk-utility. Tell me what you
want
Quoted 1 line
and if it's sane I'll see about implementing it.
That's the point, userspace has some knowledge of how to interrogate and manage md devices. A bcache device is brand new... maybe for good
reason but that's what I'm trying to understand.Quoted 2 lines
Quoted 2 lines
2/ md supports multiple superblock formats and if you Google "ssd caching" you'll see that there may be other superblock formats
that
Quoted 2 lines
Quoted 2 lines
the Linux block-caching driver could be asked to support down the road. And wouldn't it be nice if bcache had at least the option
to
Quoted 3 lines
Quoted 1 line
support the on-disk format of whatever dm-cache is doing?
That's pure fantasy. That's like expecting the ext4 code to mount a
ntfs
Quoted 1 line
filesystem!
No, there's portions of what bcache does that are similar to what md does. Do we need to invent new multiple-device handling infrastructure for a block device driver? But we are quickly approaching the "show me the code" portion of this discussion, so I need to go do more reading of bcache.
Quoted 1 line
There's a lot more to bcache's metadata than a superblock, there's
a
Quoted 1 line
journal and a full b-tree. A cache is going to need an index of
some
Quoted 1 line
kind.
Yes, but that can be independent of the configuration metadata.
--
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html