Skip to contentSkip to conversation

bcache with iscsi, mpath and lvm

DISCUSSION infernix 2 participants 4 messages Jan 31, 2012 3 messages not in the archive bcache (block layer cache)

lore
Share & tools
lore

infernix

author
#0 lore raw

Hi,

I've got the following stack, bottom to top:

- iscsi sans with multiple NICs
- servers with multiple NICs
- open-iscsi cross-connecting to the san over those multiple nics
- multipath to aggregate and load-balance IO over said paths
- LVM on top of multiple mpathX devices
- qemu-kvm running with disks connecting to logical volumes in said LVM

I am wondering if bcache can fit in, and if so, where?

Ideally I would say it sits between multipath and LVM, e.g. it is active for the /dev/mapper/mpathX devices. But that doesn't work because there isn't really a filesystem on it and so it doesn't get assigned an UUID. Even if it did, that UUID would be the same for all sdX devices in the multipath device.

Is there any other way of activating bcache besides passing it an UUID? And can it even work on top of a dm-multipath device?

Thanks!

#2 lore raw
In reply to a message not in the archive

Hi,

I'm not a bcache expert by any means but I might hazard a few comments inline.

Quoted 4 lines · infernix

Hi,

I've got the following stack, bottom to top:

Quoted 8 lines

- iscsi sans with multiple NICs
- servers with multiple NICs
- open-iscsi cross-connecting to the san over those multiple nics
- multipath to aggregate and load-balance IO over said paths
- LVM on top of multiple mpathX devices
- qemu-kvm running with disks connecting to logical volumes in said LVM

I am wondering if bcache can fit in, and if so, where?

The primary purpose of bcache is to serve as a writeback cache as this provides the most benefit to I/O.
However where you place this cache depends on your usage patterns and bottlenecks...
If you are bottlenecking purely on random I/O then the easiest and most logical place for bcache is infront of your iSCSI backing store on your SAN.

Looking something like this:

SAN NICs > IET or SCST > bcache > Backing store

This is super simple and means you make the most possible use out of the cache because all of your clients random I/O is gauranteed to make its way through this path.

If however you are bottlenecked on the iSCSI interconnect you could feasibly place bcache ontop of the multipath devices (they are just standard dm targets) right below the VMs.
This adds the highest possible amount of performance to the VMs at the cost of increased maintaince, complexity of multiple caches and ofcourse cost.

Quoted 9 lines

Ideally I would say it sits between multipath and LVM, e.g. it is active for the /dev/mapper/mpathX devices. But that doesn't work because there isn't really a filesystem on it and so it doesn't get assigned an UUID. Even if it did, that UUID would be the same for all sdX devices in the multipath device.

Is there any other way of activating bcache besides passing it an UUID? And can it even work on top of a dm-multipath device?

It works on LVM so it should work fine on multipath targets too.

Quoted 6 lines

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hope that was able to help abit. :)
Keep the list in the loop on your progress.

Joseph.

Signature · 3 lines
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846
author
#4 lore raw
In reply to a message not in the archive

Op 31-1-2012 16:50, Joseph Glanville schreef:

Quoted 11 lines
Quoted 7 lines

I've got the following stack, bottom to top:
- iscsi sans with multiple NICs
- servers with multiple NICs
- open-iscsi cross-connecting to the san over those multiple nics
- multipath to aggregate and load-balance IO over said paths
- LVM on top of multiple mpathX devices
- qemu-kvm running with disks connecting to logical volumes in said LVM

If you are bottlenecking purely on random I/O then the easiest and most logical place for bcache is infront of your iSCSI backing store on your SAN.

Agreed. But since the SAN is proprietary, there's zero chance of implementing bcache on the SAN :)

Quoted 6 lines

If however you are bottlenecked on the iSCSI interconnect you could feasibly place bcache ontop of the multipath devices (they are just standard dm targets) right below the VMs.
This adds the highest possible amount of performance to the VMs at the cost of increased maintaince, complexity of multiple caches and ofcourse cost.

Yes. So that is why I think the tradeoff of putting bcache on the mpath devices is the best. There's no shared storage use between servers; each just uses its own LVs. As long as I'm using writethrough, this is safe. Doing writeback is simply impossible with multiple caches like this.

Setting up bcache for each and every logical volume is not really going to work because they are quite dynamic.

Quoted 4 lines
Quoted 2 lines

Is there any other way of activating bcache besides passing it an UUID? And can it even work on top of a dm-multipath device?

It works on LVM so it should work fine on multipath targets too.

Except that I can't get an UUID from the LVM PVs:

blkid /dev/mpath/mpath2 -s UUID -o value

Returns nothing. So I'm a little confused as to how to register the device for caching.

#6 lore raw
In reply to a message not in the archive
Quoted 11 lines · infernix

Op 31-1-2012 16:50, Joseph Glanville schreef:

Quoted 7 lines
Quoted 7 lines

I've got the following stack, bottom to top:
- iscsi sans with multiple NICs
- servers with multiple NICs
- open-iscsi cross-connecting to the san over those multiple nics
- multipath to aggregate and load-balance IO over said paths
- LVM on top of multiple mpathX devices
- qemu-kvm running with disks connecting to logical volumes in said LVM

Quoted 7 lines
Quoted 3 lines

If you are bottlenecking purely on random I/O then the easiest and most logical place for bcache is infront of your iSCSI backing store on your SAN.

Agreed. But since the SAN is proprietary, there's zero chance of implementing bcache on the SAN :)

Ahh the wonders of proprietary appliances. :P

Quoted 10 lines
Quoted 8 lines

If however you are bottlenecked on the iSCSI interconnect you could feasibly place bcache ontop of the multipath devices (they are just standard dm targets) right below the VMs.
This adds the highest possible amount of performance to the VMs at the cost of increased maintaince, complexity of multiple caches and ofcourse cost.

Quoted 4 lines

Yes. So that is why I think the tradeoff of putting bcache on the mpath devices is the best. There's no shared storage use between servers; each just uses its own LVs. As long as I'm using writethrough, this is safe. Doing writeback is simply impossible with multiple caches like this.

It's actually quite easy to script, you can use SCSI ID or similar to map the LUNs or LVs (not sure how you have things setup) to block paths, which you can then use with multipath -ll to get the multipath WWID
I'm playing with something similar atm.

Quoted 10 lines

Setting up bcache for each and every logical volume is not really going to work because they are quite dynamic.

Quoted 5 lines
Quoted 3 lines

Is there any other way of activating bcache besides passing it an UUID? And
can it even work on top of a dm-multipath device?

Quoted 6 lines
Quoted 1 line

It works on LVM so it should work fine on multipath targets too.

Except that I can't get an UUID from the LVM PVs:

blkid /dev/mpath/mpath2 -s UUID -o value

That is a very valid point.. bcache should probably index by something other than UUID because most block devices don't have them.. only filesystems..

Quoted 8 lines

Returns nothing. So I'm a little confused as to how to register the device for caching.

--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846