bcache: Use bcache without formatting existing device
PATCH 1/1 bcache: Use bcache without formatting existing device
Introducing a bcache control device (/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a struct cache_sb and uses it to register the backing device without any need of formatting them.
---
Hi all,
At Devo we started to think of using bcache in our production systems to boost performance. But, at the very beginning, we were faced with one annoying issue, at least for our use-case: bcache needs the backing devices to be formatted before being able to use them. What's really needed is just to wipe any FS signature out of them. This is definitely not an option we will consider in our production systems because we would need to move hundreds of terabytes of data.
To circumvent the "formatting" problem, in the past weeks I worked on some modifications to the actual bcache module. In particular, I added a bcache control device (exported to /dev/bcache_ctrl) that allows communicating to the driver from userspace via IOCTL. One of the IOCTL commands that I implemented receives a struct cache_sb and uses it to register the backing device. The modifications are really small and retro compatible. To then re-create the same configuration after every boot (because the backing devices now do not present the bcache super block anymore) I created an udev rule that invokes a python script that will re-create the same scenario based on a yaml configuration file.
I'm re-sending this patch without any confidentiality footer, sorry for that.
drivers/md/bcache/Makefile | 2 +- drivers/md/bcache/control.c | 117 ++++++++++++++++++++++++++++++++ drivers/md/bcache/control.h | 12 ++++ drivers/md/bcache/ioctl_codes.h | 19 ++++++ drivers/md/bcache/super.c | 50 +++++++++++--- 5 files changed, 189 insertions(+), 11 deletions(-) create mode 100644 drivers/md/bcache/control.c create mode 100644 drivers/md/bcache/control.h create mode 100644 drivers/md/bcache/ioctl_codes.h
drivers/md/bcache/Makefile +1 -1
@@ -4,4 +4,4 @@ obj-$(CONFIG_BCACHE) += bcache.o bcache-y := alloc.o bset.o btree.o closure.o debug.o extents.o\ io.o journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ - util.o writeback.o features.o + util.o writeback.o features.o control.o
drivers/md/bcache/control.c new file +117 -0
117 lines not on this page · raw message
drivers/md/bcache/control.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BCACHE_CONTROL_H__ +#define __BCACHE_CONTROL_H__ + +#include "ioctl_codes.h" + +int __init bch_ctrl_device_init (void ); +void bch_ctrl_device_deinit (void ); + +ssize_t register_bcache_ioctl (struct bch_register_device *brd); + +#endif
drivers/md/bcache/ioctl_codes.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BCACHE_IOCTL_CODES_H__ +#define __BCACHE_IOCTL_CODES_H__ + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct bch_register_device { +const char *dev_name; +size_t size; +struct cache_sb *sb; +}; + +#define BCH_IOCTL_MAGIC (0xBC) + +/* Register a new backing device */ +#define BCH_IOCTL_REGISTER_DEVICE _IOWR(BCH_IOCTL_MAGIC, 1, struct bch_register_device) + +#endif
drivers/md/bcache/super.c +40 -10
121 lines not on this page · raw message
--
2.37.0
Quoted 32 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道:
Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Signed-off-by Andrea Tomassetti <[email protected]>---
Hi all,
At Devo we started to think of using bcache in our production systems to boost performance. But, at the very beginning, we were faced with one annoying issue, at least for our use-case: bcache needs the backing devices to be formatted before being able to use them. What's really needed is just to wipe any FS signature out of them. This is definitely not an option we will consider in our production systems because we would need to move hundreds of terabytes of data.To circumvent the "formatting" problem, in the past weeks I worked on some modifications to the actual bcache module. In particular, I added a bcache control device (exported to
/dev/bcache_ctrl) that allows communicating to the driver from userspace via IOCTL. One of the IOCTL commands that I implemented receives a structcache_sband uses it to register the backing device. The modifications are really small and retro compatible. To then re-create the same configuration after every boot (because the backing devices now do not present the bcache super block anymore) I created an udev rule that invokes a python script that will re-create the same scenario based on a yaml configuration file.
Quoted 1 line
I'm re-sending this patch without any confidentiality footer, sorry for that.
Thanks for removing that confidential and legal statement, that’s the reason I was not able to reply your email.
Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Thanks.
Coly Li
Quoted 326 lines
drivers/md/bcache/Makefile | 2 +- drivers/md/bcache/control.c | 117 ++++++++++++++++++++++++++++++++ drivers/md/bcache/control.h | 12 ++++ drivers/md/bcache/ioctl_codes.h | 19 ++++++ drivers/md/bcache/super.c | 50 +++++++++++--- 5 files changed, 189 insertions(+), 11 deletions(-)create mode 100644
drivers/md/bcache/control.ccreate mode 100644drivers/md/bcache/control.hcreate mode 100644drivers/md/bcache/ioctl_codes.hdrivers/md/bcache/Makefile +1
-1@@ -4,4 +4,4 @@ obj-$(CONFIG_BCACHE) += bcache.o bcache-y := alloc.o bset.o btree.o closure.o debug.o extents.o\ io.o journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ - util.o writeback.o features.o + util.o writeback.o features.o control.odrivers/md/bcache/control.c new file +117
-0@@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/cdev.h> +#include <linux/fs.h> +#include <linux/vmalloc.h> + +#include "control.h" + +struct bch_ctrl_device { +struct cdev cdev; +struct class *class; +dev_t dev; +}; + +static struct bch_ctrl_device _control_device; + +/* this handles IOCTL for /dev/bcache_ctrl */ +/*********************************************/ +static long bch_service_ioctl_ctrl (struct file *filp,unsigned int cmd, +unsigned long arg) +{ +int retval =0 ; + +if (_IOC_TYPE (cmd) !=BCH_IOCTL_MAGIC ) +return -EINVAL ; + +if (!capable (CAP_SYS_ADMIN )) { +/* Must be root to issue ioctls */ +return -EPERM ; + } + +switch (cmd) { +case BCH_IOCTL_REGISTER_DEVICE : { +struct bch_register_device *cmd_info; + + cmd_info =vmalloc (sizeof (struct bch_register_device )); +if (!cmd_info) +return -ENOMEM ; + +if (copy_from_user (cmd_info, (void __user *)arg, +sizeof (struct bch_register_device ))) { +pr_err ("Cannot copy cmd info from user space\n" ); +vfree (cmd_info); +return -EINVAL ; + } + + retval =register_bcache_ioctl (cmd_info); + +vfree (cmd_info); +return retval; + } + +default : +return -EINVAL ; + } +} + +static const struct file_operations _ctrl_dev_fops = { + .owner =THIS_MODULE , + .unlocked_ioctl = bch_service_ioctl_ctrl +}; + +int __init bch_ctrl_device_init (void ) +{ +struct bch_ctrl_device *ctrl = &_control_device; +struct device *device; +int result =0 ; + + result =alloc_chrdev_region (&ctrl->dev,0 ,1 ,"bcache" ); +if (result) { +pr_err ("Cannot allocate control chrdev number.\n" ); + goto error_alloc_chrdev_region; + } + +cdev_init (&ctrl->cdev, &_ctrl_dev_fops); + + result =cdev_add (&ctrl->cdev, ctrl->dev,1 ); +if (result) { +pr_err ("Cannot add control chrdev.\n" ); + goto error_cdev_add; + } + + ctrl->class =class_create (THIS_MODULE ,"bcache" ); +if (IS_ERR (ctrl->class)) { +pr_err ("Cannot create control chrdev class.\n" ); + result =PTR_ERR (ctrl->class); + goto error_class_create; + } + + device =device_create (ctrl->class,NULL , ctrl->dev,NULL , +"bcache_ctrl" ); +if (IS_ERR (device)) { +pr_err ("Cannot create control chrdev.\n" ); + result =PTR_ERR (device); + goto error_device_create; + } + +return result; + +error_device_create: +class_destroy (ctrl->class); +error_class_create: +cdev_del (&ctrl->cdev); +error_cdev_add: +unregister_chrdev_region (ctrl->dev,1 ); +error_alloc_chrdev_region: +return result; +} + +void bch_ctrl_device_deinit (void ) +{ +struct bch_ctrl_device *ctrl = &_control_device; + +device_destroy (ctrl->class, ctrl->dev); +class_destroy (ctrl->class); +cdev_del (&ctrl->cdev); +unregister_chrdev_region (ctrl->dev,1 ); +}drivers/md/bcache/control.h new file +12
-0@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BCACHE_CONTROL_H__ +#define __BCACHE_CONTROL_H__ + +#include "ioctl_codes.h" + +int __init bch_ctrl_device_init (void ); +void bch_ctrl_device_deinit (void ); + +ssize_t register_bcache_ioctl (struct bch_register_device *brd); + +#endif drivers/md/bcache/ioctl_codes.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BCACHE_IOCTL_CODES_H__ +#define __BCACHE_IOCTL_CODES_H__ + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct bch_register_device { +const char *dev_name; +size_t size; +struct cache_sb *sb; +}; + +#define BCH_IOCTL_MAGIC (0xBC) + +/* Register a new backing device */ +#define BCH_IOCTL_REGISTER_DEVICE _IOWR(BCH_IOCTL_MAGIC, 1, struct bch_register_device) + +#endif drivers/md/bcache/super.c +20
-9@@ -14,6 +14,7 @@#include "request.h" #include "writeback.h" #include "features.h" +#include "control.h" #include <linux/blkdev.h> #include <linux/pagemap.h> @@ -340,6 +341,9 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent)struct closure *cl = &dc->sb_write;struct bio *bio = &dc->sb_bio; +if (!dc->sb_disk) +return ; +down (&dc->sb_write_mutex);closure_init (cl, parent);@@ -2403,14 +2407,14 @@ static int register_cache(struct cache_sb *sb, struct cache_sb_disk *sb_disk,/* Global interfaces/init */ -static ssize_t register_bcache (struct kobject *k,struct kobj_attribute *attr, +static ssize_t register_bcache_sysfs (struct kobject *k,struct kobj_attribute *attr,const char *buffer,size_t size);static ssize_t bch_pending_bdevs_cleanup (struct kobject *k,struct kobj_attribute *attr,const char *buffer,size_t size); -kobj_attribute_write (register, register_bcache); -kobj_attribute_write (register_quiet, register_bcache); +kobj_attribute_write (register, register_bcache_sysfs); +kobj_attribute_write (register_quiet, register_bcache_sysfs);kobj_attribute_write (pendings_cleanup, bch_pending_bdevs_cleanup);static bool bch_is_open_backing (dev_t dev)@@ -2465,7 +2469,8 @@ static void register_bdev_worker(struct work_struct *work) dc =kzalloc (sizeof (*dc),GFP_KERNEL );if (!dc) { fail = true; -put_page (virt_to_page (args->sb_disk)); +if (args->sb_disk) +put_page (virt_to_page (args->sb_disk));blkdev_put (args->bdev,FMODE_READ |FMODE_WRITE |FMODE_EXCL ); goto out; }@@ -2495,7 +2500,8 @@ static void register_cache_worker(struct work_struct *work) ca =kzalloc (sizeof (*ca),GFP_KERNEL );if (!ca) { fail = true; -put_page (virt_to_page (args->sb_disk)); +if (args->sb_disk) +put_page (virt_to_page (args->sb_disk));blkdev_put (args->bdev,FMODE_READ |FMODE_WRITE |FMODE_EXCL ); goto out; }@@ -2525,7 +2531,7 @@ static void register_device_async(struct async_reg_args *args)queue_delayed_work (system_wq, &args->reg_work,10 ); } -static ssize_t register_bcache (struct kobject *k,struct kobj_attribute *attr, +static ssize_t register_bcache_common (void *k,struct kobj_attribute *attr,const char *buffer,size_t size) {const char *err;@@ -2587,9 +2593,14 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,if (set_blocksize (bdev,4096 )) goto out_blkdev_put; - err =read_super (sb, bdev, &sb_disk); -if (err) - goto out_blkdev_put; +if (!k) { + err =read_super (sb, bdev, &sb_disk); +if (err) + goto out_blkdev_put; + }else { + sb_disk =NULL ; +memcpy (sb, (struct cache_sb *)k,sizeof (struct cache_sb )); + } err ="failed to register device" ;@@ -2651,7 +2662,8 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,return size;
out_put_sb_page:drivers/md/bcache/super.c +2
-1-put_page (virt_to_page (sb_disk)); +if (!k) +put_page (virt_to_page (sb_disk));
out_blkdev_put:blkdev_put(bdev, FMODE_READ | FMODE_WRITE | FMODE_EXCL);
out_free_sb:drivers/md/bcache/super.c +18
-1@@ -2666,6 +2678,18 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,return ret; } +static ssize_t register_bcache_sysfs (struct kobject *k,struct kobj_attribute *attr, +const char *buffer,size_t size) +{ +return register_bcache_common (NULL , attr, buffer, size); +} + +ssize_t register_bcache_ioctl (struct bch_register_device *brd) +{ +return register_bcache_common ((void *)brd->sb,NULL , brd->dev_name, brd->size); +} + +struct pdev {struct list_head list;@@ -2819,6 +2843,7 @@ static void bcache_exit(void) {bch_debug_exit ();bch_request_exit (); +bch_ctrl_device_deinit ();if (bcache_kobj)kobject_put (bcache_kobj);if (bcache_wq)@@ -2918,6 +2943,11 @@ static int __init bcache_init(void)bch_debug_init ();closure_debug_init (); +if (bch_ctrl_device_init ()) { +pr_err ("Cannot initialize control device\n" ); + goto err; + } + bcache_is_reboot = false;return 0 ; --2.37.0
Quoted 33 lines · Coly Li
Quoted 33 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道:
Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Signed-off-by Andrea Tomassetti <[email protected]>---
Hi all,
At Devo we started to think of using bcache in our production systems to boost performance. But, at the very beginning, we were faced with one annoying issue, at least for our use-case: bcache needs the backing devices to be formatted before being able to use them. What's really needed is just to wipe any FS signature out of them. This is definitely not an option we will consider in our production systems because we would need to move hundreds of terabytes of data.To circumvent the "formatting" problem, in the past weeks I worked on some modifications to the actual bcache module. In particular, I added a bcache control device (exported to
/dev/bcache_ctrl) that allows communicating to the driver from userspace via IOCTL. One of the IOCTL commands that I implemented receives a structcache_sband uses it to register the backing device. The modifications are really small and retro compatible. To then re-create the same configuration after every boot (because the backing devices now do not present the bcache super block anymore) I created an udev rule that invokes a python script that will re-create the same scenario based on a yaml configuration file.
Quoted 4 lines
Quoted 1 line
I'm re-sending this patch without any confidentiality footer, sorry for that.
Thanks for removing that confidential and legal statement, that’s the reason I was not able to reply your email.
Thank you for your patience and sorry for the newbie mistake.
Quoted 4 lines
Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
For the operational drawbacks: from time to time, we exploit the online filesystem resize capability of XFS to increase the volume size. This would be at least tedious, if not nearly impossible, if the volume is mapped inside a dm-linear.
Quoted 2 lines
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Which potential security risks concern you?
Thank you,
Andrea
Quoted 331 lines
Thanks.
Coly Li
Quoted 326 lines
drivers/md/bcache/Makefile | 2 +- drivers/md/bcache/control.c | 117 ++++++++++++++++++++++++++++++++ drivers/md/bcache/control.h | 12 ++++ drivers/md/bcache/ioctl_codes.h | 19 ++++++ drivers/md/bcache/super.c | 50 +++++++++++--- 5 files changed, 189 insertions(+), 11 deletions(-)create mode 100644
drivers/md/bcache/control.ccreate mode 100644drivers/md/bcache/control.hcreate mode 100644drivers/md/bcache/ioctl_codes.hdrivers/md/bcache/Makefile +1
-1@@ -4,4 +4,4 @@ obj-$(CONFIG_BCACHE) += bcache.o bcache-y := alloc.o bset.o btree.o closure.o debug.o extents.o\ io.o journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ - util.o writeback.o features.o + util.o writeback.o features.o control.odrivers/md/bcache/control.c new file +117
-0@@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/cdev.h> +#include <linux/fs.h> +#include <linux/vmalloc.h> + +#include "control.h" + +struct bch_ctrl_device { +struct cdev cdev; +struct class *class; +dev_t dev; +}; + +static struct bch_ctrl_device _control_device; + +/* this handles IOCTL for /dev/bcache_ctrl */ +/*********************************************/ +static long bch_service_ioctl_ctrl (struct file *filp,unsigned int cmd, +unsigned long arg) +{ +int retval =0 ; + +if (_IOC_TYPE (cmd) !=BCH_IOCTL_MAGIC ) +return -EINVAL ; + +if (!capable (CAP_SYS_ADMIN )) { +/* Must be root to issue ioctls */ +return -EPERM ; + } + +switch (cmd) { +case BCH_IOCTL_REGISTER_DEVICE : { +struct bch_register_device *cmd_info; + + cmd_info =vmalloc (sizeof (struct bch_register_device )); +if (!cmd_info) +return -ENOMEM ; + +if (copy_from_user (cmd_info, (void __user *)arg, +sizeof (struct bch_register_device ))) { +pr_err ("Cannot copy cmd info from user space\n" ); +vfree (cmd_info); +return -EINVAL ; + } + + retval =register_bcache_ioctl (cmd_info); + +vfree (cmd_info); +return retval; + } + +default : +return -EINVAL ; + } +} + +static const struct file_operations _ctrl_dev_fops = { + .owner =THIS_MODULE , + .unlocked_ioctl = bch_service_ioctl_ctrl +}; + +int __init bch_ctrl_device_init (void ) +{ +struct bch_ctrl_device *ctrl = &_control_device; +struct device *device; +int result =0 ; + + result =alloc_chrdev_region (&ctrl->dev,0 ,1 ,"bcache" ); +if (result) { +pr_err ("Cannot allocate control chrdev number.\n" ); + goto error_alloc_chrdev_region; + } + +cdev_init (&ctrl->cdev, &_ctrl_dev_fops); + + result =cdev_add (&ctrl->cdev, ctrl->dev,1 ); +if (result) { +pr_err ("Cannot add control chrdev.\n" ); + goto error_cdev_add; + } + + ctrl->class =class_create (THIS_MODULE ,"bcache" ); +if (IS_ERR (ctrl->class)) { +pr_err ("Cannot create control chrdev class.\n" ); + result =PTR_ERR (ctrl->class); + goto error_class_create; + } + + device =device_create (ctrl->class,NULL , ctrl->dev,NULL , +"bcache_ctrl" ); +if (IS_ERR (device)) { +pr_err ("Cannot create control chrdev.\n" ); + result =PTR_ERR (device); + goto error_device_create; + } + +return result; + +error_device_create: +class_destroy (ctrl->class); +error_class_create: +cdev_del (&ctrl->cdev); +error_cdev_add: +unregister_chrdev_region (ctrl->dev,1 ); +error_alloc_chrdev_region: +return result; +} + +void bch_ctrl_device_deinit (void ) +{ +struct bch_ctrl_device *ctrl = &_control_device; + +device_destroy (ctrl->class, ctrl->dev); +class_destroy (ctrl->class); +cdev_del (&ctrl->cdev); +unregister_chrdev_region (ctrl->dev,1 ); +}drivers/md/bcache/control.h new file +12
-0@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BCACHE_CONTROL_H__ +#define __BCACHE_CONTROL_H__ + +#include "ioctl_codes.h" + +int __init bch_ctrl_device_init (void ); +void bch_ctrl_device_deinit (void ); + +ssize_t register_bcache_ioctl (struct bch_register_device *brd); + +#endif drivers/md/bcache/ioctl_codes.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BCACHE_IOCTL_CODES_H__ +#define __BCACHE_IOCTL_CODES_H__ + +#include <linux/ioctl.h> +#include <linux/types.h> + +struct bch_register_device { +const char *dev_name; +size_t size; +struct cache_sb *sb; +}; + +#define BCH_IOCTL_MAGIC (0xBC) + +/* Register a new backing device */ +#define BCH_IOCTL_REGISTER_DEVICE _IOWR(BCH_IOCTL_MAGIC, 1, struct bch_register_device) + +#endif drivers/md/bcache/super.c +20
-9@@ -14,6 +14,7 @@#include "request.h" #include "writeback.h" #include "features.h" +#include "control.h" #include <linux/blkdev.h> #include <linux/pagemap.h> @@ -340,6 +341,9 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent)struct closure *cl = &dc->sb_write;struct bio *bio = &dc->sb_bio; +if (!dc->sb_disk) +return ; +down (&dc->sb_write_mutex);closure_init (cl, parent);@@ -2403,14 +2407,14 @@ static int register_cache(struct cache_sb *sb, struct cache_sb_disk *sb_disk,/* Global interfaces/init */ -static ssize_t register_bcache (struct kobject *k,struct kobj_attribute *attr, +static ssize_t register_bcache_sysfs (struct kobject *k,struct kobj_attribute *attr,const char *buffer,size_t size);static ssize_t bch_pending_bdevs_cleanup (struct kobject *k,struct kobj_attribute *attr,const char *buffer,size_t size); -kobj_attribute_write (register, register_bcache); -kobj_attribute_write (register_quiet, register_bcache); +kobj_attribute_write (register, register_bcache_sysfs); +kobj_attribute_write (register_quiet, register_bcache_sysfs);kobj_attribute_write (pendings_cleanup, bch_pending_bdevs_cleanup);static bool bch_is_open_backing (dev_t dev)@@ -2465,7 +2469,8 @@ static void register_bdev_worker(struct work_struct *work) dc =kzalloc (sizeof (*dc),GFP_KERNEL );if (!dc) { fail = true; -put_page (virt_to_page (args->sb_disk)); +if (args->sb_disk) +put_page (virt_to_page (args->sb_disk));blkdev_put (args->bdev,FMODE_READ |FMODE_WRITE |FMODE_EXCL ); goto out; }@@ -2495,7 +2500,8 @@ static void register_cache_worker(struct work_struct *work) ca =kzalloc (sizeof (*ca),GFP_KERNEL );if (!ca) { fail = true; -put_page (virt_to_page (args->sb_disk)); +if (args->sb_disk) +put_page (virt_to_page (args->sb_disk));blkdev_put (args->bdev,FMODE_READ |FMODE_WRITE |FMODE_EXCL ); goto out; }@@ -2525,7 +2531,7 @@ static void register_device_async(struct async_reg_args *args)queue_delayed_work (system_wq, &args->reg_work,10 ); } -static ssize_t register_bcache (struct kobject *k,struct kobj_attribute *attr, +static ssize_t register_bcache_common (void *k,struct kobj_attribute *attr,const char *buffer,size_t size) {const char *err;@@ -2587,9 +2593,14 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,if (set_blocksize (bdev,4096 )) goto out_blkdev_put; - err =read_super (sb, bdev, &sb_disk); -if (err) - goto out_blkdev_put; +if (!k) { + err =read_super (sb, bdev, &sb_disk); +if (err) + goto out_blkdev_put; + }else { + sb_disk =NULL ; +memcpy (sb, (struct cache_sb *)k,sizeof (struct cache_sb )); + } err ="failed to register device" ;@@ -2651,7 +2662,8 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,return size;
out_put_sb_page:drivers/md/bcache/super.c +2
-1-put_page (virt_to_page (sb_disk)); +if (!k) +put_page (virt_to_page (sb_disk));
out_blkdev_put:blkdev_put(bdev, FMODE_READ | FMODE_WRITE | FMODE_EXCL);
out_free_sb:drivers/md/bcache/super.c +18
-1@@ -2666,6 +2678,18 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,return ret; } +static ssize_t register_bcache_sysfs (struct kobject *k,struct kobj_attribute *attr, +const char *buffer,size_t size) +{ +return register_bcache_common (NULL , attr, buffer, size); +} + +ssize_t register_bcache_ioctl (struct bch_register_device *brd) +{ +return register_bcache_common ((void *)brd->sb,NULL , brd->dev_name, brd->size); +} + +struct pdev {struct list_head list;@@ -2819,6 +2843,7 @@ static void bcache_exit(void) {bch_debug_exit ();bch_request_exit (); +bch_ctrl_device_deinit ();if (bcache_kobj)kobject_put (bcache_kobj);if (bcache_wq)@@ -2918,6 +2943,11 @@ static int __init bcache_init(void)bch_debug_init ();closure_debug_init (); +if (bch_ctrl_device_init ()) { +pr_err ("Cannot initialize control device\n" ); + goto err; + } + bcache_is_reboot = false;return 0 ; --2.37.0
Quoted 47 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 33 lines · Coly Li
Quoted 33 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道:
Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Signed-off-by Andrea Tomassetti <[email protected]>---
Hi all,
At Devo we started to think of using bcache in our production systems to boost performance. But, at the very beginning, we were faced with one annoying issue, at least for our use-case: bcache needs the backing devices to be formatted before being able to use them. What's really needed is just to wipe any FS signature out of them. This is definitely not an option we will consider in our production systems because we would need to move hundreds of terabytes of data.To circumvent the "formatting" problem, in the past weeks I worked on some modifications to the actual bcache module. In particular, I added a bcache control device (exported to
/dev/bcache_ctrl) that allows communicating to the driver from userspace via IOCTL. One of the IOCTL commands that I implemented receives a structcache_sband uses it to register the backing device. The modifications are really small and retro compatible. To then re-create the same configuration after every boot (because the backing devices now do not present the bcache super block anymore) I created an udev rule that invokes a python script that will re-create the same scenario based on a yaml configuration file.Quoted 4 lines
Quoted 1 line
I'm re-sending this patch without any confidentiality footer, sorry for that.
Thanks for removing that confidential and legal statement, that’s the reason I was not able to reply your email.
Thank you for your patience and sorry for the newbie mistake.
Quoted 4 lines
Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
- A new IOCTL interface added, even all rested interface are sysfs based.
- Do things in kernel space while there is solution in user space.
All the above opinions are quite personal to myself, I don’t say you are wrong or I am correct. If the patch works, that’s cool and you can use it as you want, but I don’t support it to be in upstream.
Quoted 5 lines
For the operational drawbacks: from time to time, we exploit the online filesystem resize capability of XFS to increase the volume size. This would be at least tedious, if not nearly impossible, if the volume is mapped inside a dm-linear.
Currently bcache doesn’t support cache or backing device resize. I don’t see the connection between above statement and feeding an in-memory super block via IOCTL command.
Quoted 4 lines
Quoted 2 lines
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Which potential security risks concern you?
Currently we don’t check all members of struct cache_sb_disk, what we do is to simply trust bcache-tools. Create a new IOCTL interface to send a user space memory into kernel space as superblock, that needs quite a lot of checking code to make sure it won’t panic the kernel. It is possible, but it is not worthy to add so many code for the purpose, especially adding them into upstream code.
I am not able to provide an explicit example for security risk, just the new adding interface makes me not so confident.
Thanks.
Coly Li
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to dd ‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.
Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for bch_write_bdev_super() there does not appear to be a way to handle attaching caches to the running bcache. For example:
1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1
In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
I could imagine something like /etc/bcachetab (like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.
You want to make sure that when a dirty writeback-cached bdev is registered that it does not activate until either:
1. The cache is attached
2. echo 1 > /sys/block/bcache0/bcache/running
Quoted 1 line
- A new IOCTL interface added, even all rested interface are sysfs based.
True. Perhaps something like this would be better, and it would avoid sharing a userspace page for blindly populating struct cache_sb, too:
echo '/dev/bdev [bdev_uuid] [/dev/cachedev|cset_uuid] [options]' > ¥
/sys/fs/bcache/register_raw
Because of the writeback issue above, the cache and bdev either need to be registered simultaneously or the cset uuid need to be specified.
Quoted 5 lines
- Do things in kernel space while there is solution in user space.
All the above opinions are quite personal to myself, I don’t say you are wrong or I am correct. If the patch works, that’s cool and you can use it as you want, but I don’t support it to be in upstream.
An alternate implementation might be to create a dm-bcache target. The core bcache code could be left alone except a few EXPORT_SYMBOL()'s so dm-bcache can reach the bcache registration bits.
This would:
* Minimally impact the bcache code base
* Solve the blind-populating of struct cache_sb issue in the same way as register_raw could work above.
* Create a nicely segmented codebase (dm target) to upstream separately through the DM team.
* Could be maintained cleanly out-of-tree because the bcache registration interfaces change very infrequently.
* bdev resize could be done with a dmsetup replace but you'll need to add resize support as below.
Quoted 8 lines
Quoted 4 lines
For the operational drawbacks: from time to time, we exploit the online filesystem resize capability of XFS to increase the volume size. This would be at least tedious, if not nearly impossible, if the volume is mapped inside a dm-linear.
Currently bcache doesn’t support cache or backing device resize. I don’t see the connection between above statement and feeding an in-memory super block via IOCTL command.
Resize is perhaps unrelated, so if you decide to tackle bdev or cachedev resize then please start a new list thread. Briefly: supporting bdev resize is probably easy. I've looked through the code a few times with this in mind but haven't had time.
Here's the summary, not sure if there are any other considerations:
1. Create a sysfs file like /sys/block/bcache0/bcache/resize to trigger resize on echo 1 >.
2. Refactor the set_capacity() bits from bcache_device_init() into its own function.
3. Put locks around bcache_device.full_dirty_stripes and bcache_device.stripe_sectors_dirty. Re-alloc+copy+free and zero the new bytes at the end.
The cachedev's don't know anything about the bdev size, so (according to Kent) they will "just work" by referencing new offsets that didn't exist before when IOs come through. (This is basically the same as resizing the bdev and then unregister/re-register which is how we resize bdevs now.)
As for resizing a cachedev, I've not looked at all---not sure about that one. We always detach, resize, make-bcache and re-attach the new cache. Maybe it is similarly simple, but haven't looked.
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Which potential security risks concern you?
Currently we don’t check all members of struct
cache_sb_disk, what we do is to simply trust bcache-tools. Create a new IOCTL interface to send a user space memory into kernel space as superblock, that needs quite a lot of checking code to make sure it won’t panic the kernel. It is possible, but it is not worthy to add so many code for the purpose, especially adding them into upstream code.
Quoted 2 lines
I am not able to provide an explicit example for security risk, just the new adding interface makes me not so confident.
Maintaining a blind structure population from a userspace page would be difficult as well because even if the kernel validates _everything_ in cache_sb today, anyone extending struct cache_sb needs to remember to add checks for that. Failing to enforce validation inside the kernel could lead to kernel crashes or data corruption from userspace which is of course never good.
We always assume that, somehow, someone could leverage an insecure IOCTL call and crash the OS when they shouldn't be allowed to (even if they are not root, like from sudo). This is a security issue from an assurance and integrity point of view, even if there might be neither an obvious privelege escalation nor privacy concern.
-Eric
Quoted 7 lines
Thanks.
Coly Li
Hi Eric and Coly,
thank you very much for the interesting discussion.
Let me give you a little bit more context that maybe will help you better understand the specific use case that this patch is trying to tackle.
As I stated in a previous email, we're currently using ephemeral disks as cache devices so, for us, using bcache in writeback mode is out of discussion because it could lead us to data loss.
Moreover, I fully understand the dangerous implications of registering a backing device without formatting it and using it in wb mode. That's why this patch is meant to register backing devices *only* (no need to register a cache device without formatting it) and in wt mode *only*. I'm not saying it cannot be used to register backing devices in wb mode, I'm just saying that it needs further and deeper analysis on the implications.
Sorry if this context wasn't clear enough.
Quoted 44 lines · Eric Wheeler
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
Quoted 5 lines
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
That's exactly what was preventing us from using bcache in production prior to this patch.
Quoted 25 lines
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to
dd‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for
bch_write_bdev_super()there does not appear to be a way to handle attaching caches to the running bcache. For example:1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
Quoted 8 lines
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
I could imagine something like
/etc/bcachetab(like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.
Yes, in conjunction with this patch, I developed a python udev script that manages device registration base on a YAML configuration file. I even patched bcache-tools to support the new IOCTL registration. You will find a link to the Github project at the end of this message.
Quoted 7 lines
You want to make sure that when a dirty writeback-cached bdev is registered that it does not activate until either:
1. The cache is attached
2. echo 1 >/sys/block/bcache0/bcache/running
Quoted 8 lines
Quoted 1 line
- A new IOCTL interface added, even all rested interface are sysfs based.
True. Perhaps something like this would be better, and it would avoid sharing a userspace page for blindly populating
struct cache_sb, too:echo '
/dev/bdev[bdev_uuid] [/dev/cachedev|cset_uuid] [options]' > ¥
/sys/fs/bcache/register_raw
I thought that introducing an IOCTL interface could be a stopper for this patch, so I'm more than welcome to refactor it using sysfs entries only. But, I will do so only if there's even the slightest chance that this patch can be merged.
Quoted 74 lines
Because of the writeback issue above, the cache and bdev either need to be registered simultaneously or the cset uuid need to be specified.
Quoted 5 lines
- Do things in kernel space while there is solution in user space.
All the above opinions are quite personal to myself, I don’t say you are wrong or I am correct. If the patch works, that’s cool and you can use it as you want, but I don’t support it to be in upstream.
An alternate implementation might be to create a dm-bcache target. The core bcache code could be left alone except a few EXPORT_SYMBOL()'s so dm-bcache can reach the bcache registration bits.
This would:
* Minimally impact the bcache code base
* Solve the blind-populating ofstruct cache_sbissue in the same way asregister_rawcould work above.
* Create a nicely segmented codebase (dm target) to upstream separately through the DM team.
* Could be maintained cleanly out-of-tree because the bcache registration interfaces change very infrequently.
* bdev resize could be done with admsetup replacebut you'll need to add resize support as below.Quoted 8 lines
Quoted 4 lines
For the operational drawbacks: from time to time, we exploit the online filesystem resize capability of XFS to increase the volume size. This would be at least tedious, if not nearly impossible, if the volume is mapped inside a dm-linear.
Currently bcache doesn’t support cache or backing device resize. I don’t see the connection between above statement and feeding an in-memory super block via IOCTL command.
Resize is perhaps unrelated, so if you decide to tackle bdev or cachedev resize then please start a new list thread. Briefly: supporting bdev resize is probably easy. I've looked through the code a few times with this in mind but haven't had time.
Here's the summary, not sure if there are any other considerations:
1. Create a sysfs file like
/sys/block/bcache0/bcache/resizeto trigger resize on echo 1 >.
2. Refactor theset_capacity()bits frombcache_device_init()into its own function.
3. Put locks around bcache_device.full_dirty_stripes and bcache_device.stripe_sectors_dirty. Re-alloc+copy+free and zero the new bytes at the end.The cachedev's don't know anything about the bdev size, so (according to Kent) they will "just work" by referencing new offsets that didn't exist before when IOs come through. (This is basically the same as resizing the bdev and then unregister/re-register which is how we resize bdevs now.)
As for resizing a cachedev, I've not looked at all---not sure about that one. We always detach, resize, make-bcache and re-attach the new cache. Maybe it is similarly simple, but haven't looked.
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Which potential security risks concern you?
Currently we don’t check all members of struct
cache_sb_disk, what we do is to simply trust bcache-tools. Create a new IOCTL interface to send a user space memory into kernel space as superblock, that needs quite a lot of checking code to make sure it won’t panic the kernel. It is possible, but it is not worthy to add so many code for the purpose, especially adding them into upstream code.Quoted 2 lines
I am not able to provide an explicit example for security risk, just the new adding interface makes me not so confident.
Quoted 7 lines
Maintaining a blind structure population from a userspace page would be difficult as well because even if the kernel validates _everything_ in
cache_sbtoday, anyone extendingstruct cache_sbneeds to remember to add checks for that. Failing to enforce validation inside the kernel could lead to kernel crashes or data corruption from userspace which is of course never good.
Can I solve this by refactoring the patch and using sysfs based registration instead?
For anyone that is interested to try this solution, and as a reference, I leave here below the links to my public Github repositories:
- bcache-tools: https://github.com/andreatomassetti/bcache-tools
- bcache (standalone, patched): https://github.com/andreatomassetti/bcache
Thank you very much,
Andrea
Quoted 16 lines
We always assume that, somehow, someone could leverage an insecure IOCTL call and crash the OS when they shouldn't be allowed to (even if they are not root, like from sudo). This is a security issue from an assurance and integrity point of view, even if there might be neither an obvious privelege escalation nor privacy concern.
-Eric
Quoted 7 lines
Thanks.
Coly Li
Quoted 44 lines · Andrea Tomassetti
Quoted 43 lines · Eric Wheeler
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
Quoted 7 lines
Quoted 4 lines
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
That's exactly what was preventing us from using bcache in production prior to this patch.
Ok, but we always use writeback...and others may wish to, too. I think any patch that introduces a feature needs to support existing features without introducing limitations on the behavior.
Quoted 29 lines
Quoted 29 lines
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to
dd‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for
bch_write_bdev_super()there does not appear to be a way to handle attaching caches to the running bcache. For example:1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
Quoted 8 lines
Quoted 4 lines
I could imagine something like
/etc/bcachetab(like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.Yes, in conjunction with this patch, I developed a python udev script that manages device registration base on a YAML configuration file. I even patched bcache-tools to support the new IOCTL registration. You will find a link to the Github project at the end of this message.
Fewer dependencies are better: There are still python2 vs python3 conflicts out there---and loading python and its dependencies into an initrd/initramfs for early bcache loading could be very messy, indeed!
You've already put some work into make-bcache so creating a bcache_udev function and a bcache-udev.c file (like make-bcache.c) is probably easy enough. IMHO, a single-line grepable format (instead of YAML) could be used to minimize dependencies so that it is clean in an initramfs in the same way that mdadm.conf and crypttab already do. You could then parse it with C or bash pretty easily...
Quoted 11 lines
Quoted 11 lines
You want to make sure that when a dirty writeback-cached bdev is registered that it does not activate until either:
1. The cache is attached
2. echo 1 >/sys/block/bcache0/bcache/runningQuoted 1 line
- A new IOCTL interface added, even all rested interface are sysfs based.
True. Perhaps something like this would be better, and it would avoid sharing a userspace page for blindly populating
struct cache_sb, too:
Quoted 7 lines
Quoted 2 lines
echo '
/dev/bdev[bdev_uuid] [/dev/cachedev|cset_uuid] [options]' > ¥
/sys/fs/bcache/register_rawI thought that introducing an IOCTL interface could be a stopper for this patch, so I'm more than welcome to refactor it using sysfs entries only. But, I will do so only if there's even the slightest chance that this patch can be merged.
I'm for the patch with sysfs, but I'm not the decision maker either.
Coly: what would your requirements be to accept this upstream?
Quoted 82 lines
Quoted 74 lines
Because of the writeback issue above, the cache and bdev either need to be registered simultaneously or the cset uuid need to be specified.
Quoted 5 lines
- Do things in kernel space while there is solution in user space.
All the above opinions are quite personal to myself, I don’t say you are wrong or I am correct. If the patch works, that’s cool and you can use it as you want, but I don’t support it to be in upstream.
An alternate implementation might be to create a dm-bcache target. The core bcache code could be left alone except a few EXPORT_SYMBOL()'s so dm-bcache can reach the bcache registration bits.
This would:
* Minimally impact the bcache code base
* Solve the blind-populating ofstruct cache_sbissue in the same way asregister_rawcould work above.
* Create a nicely segmented codebase (dm target) to upstream separately through the DM team.
* Could be maintained cleanly out-of-tree because the bcache registration interfaces change very infrequently.
* bdev resize could be done with admsetup replacebut you'll need to add resize support as below.Quoted 8 lines
Quoted 4 lines
For the operational drawbacks: from time to time, we exploit the online filesystem resize capability of XFS to increase the volume size. This would be at least tedious, if not nearly impossible, if the volume is mapped inside a dm-linear.
Currently bcache doesn’t support cache or backing device resize. I don’t see the connection between above statement and feeding an in-memory super block via IOCTL command.
Resize is perhaps unrelated, so if you decide to tackle bdev or cachedev resize then please start a new list thread. Briefly: supporting bdev resize is probably easy. I've looked through the code a few times with this in mind but haven't had time.
Here's the summary, not sure if there are any other considerations:
1. Create a sysfs file like
/sys/block/bcache0/bcache/resizeto trigger resize on echo 1 >.
2. Refactor theset_capacity()bits frombcache_device_init()into its own function.
3. Put locks around bcache_device.full_dirty_stripes and bcache_device.stripe_sectors_dirty. Re-alloc+copy+free and zero the new bytes at the end.The cachedev's don't know anything about the bdev size, so (according to Kent) they will "just work" by referencing new offsets that didn't exist before when IOs come through. (This is basically the same as resizing the bdev and then unregister/re-register which is how we resize bdevs now.)
As for resizing a cachedev, I've not looked at all---not sure about that one. We always detach, resize, make-bcache and re-attach the new cache. Maybe it is similarly simple, but haven't looked.
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Which potential security risks concern you?
Currently we don’t check all members of struct
cache_sb_disk, what we do is to simply trust bcache-tools. Create a new IOCTL interface to send a user space memory into kernel space as superblock, that needs quite a lot of checking code to make sure it won’t panic the kernel. It is possible, but it is not worthy to add so many code for the purpose, especially adding them into upstream code.Quoted 2 lines
I am not able to provide an explicit example for security risk, just the new adding interface makes me not so confident.
Quoted 7 lines
Maintaining a blind structure population from a userspace page would be difficult as well because even if the kernel validates _everything_ in
cache_sbtoday, anyone extendingstruct cache_sbneeds to remember to add checks for that. Failing to enforce validation inside the kernel could lead to kernel crashes or data corruption from userspace which is of course never good.
Quoted 2 lines
Can I solve this by refactoring the patch and using sysfs based registration instead?
Sure, just make the sysfs interface configurable enough that:
- all configuration is text (not binary)
- bcache.txt documentation is updated
-Eric
Quoted 26 lines
For anyone that is interested to try this solution, and as a reference, I leave here below the links to my public Github repositories:
- bcache-tools: https://github.com/andreatomassetti/bcache-tools
- bcache (standalone, patched): https://github.com/andreatomassetti/bcacheThank you very much,
AndreaQuoted 16 lines
We always assume that, somehow, someone could leverage an insecure IOCTL call and crash the OS when they shouldn't be allowed to (even if they are not root, like from sudo). This is a security issue from an assurance and integrity point of view, even if there might be neither an obvious privelege escalation nor privacy concern.
-Eric
Quoted 7 lines
Thanks.
Coly Li
Quoted 51 lines · Eric Wheeler
Quoted 49 lines · Andrea Tomassetti
Quoted 47 lines · Eric Wheeler
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
Quoted 7 lines
Quoted 2 lines
That's exactly what was preventing us from using bcache in production prior to this patch.
Ok, but we always use writeback...and others may wish to, too. I think any patch that introduces a feature needs to support existing features without introducing limitations on the behavior.
Totally agree. My only point was that I extensively tested this patch with wt mode. It works in wb mode as well, for sure because the backing device's header is almost never used. The only issue I can foresee in wb mode is in case of a machine reboot: the backing device will lose the virtual header and, at boot time, another one will be generated. It will get attached again to its cache device with a new UID and I'm not sure if this will imply the loss of the data that was not previously written to it, but was only present on the cache device. But I think that losing data it's a well-known risk of wb mode.
Quoted 42 lines
Quoted 29 lines
Quoted 29 lines
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to
dd‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for
bch_write_bdev_super()there does not appear to be a way to handle attaching caches to the running bcache. For example:1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
Quoted 8 lines
Quoted 4 lines
I could imagine something like
/etc/bcachetab(like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.Yes, in conjunction with this patch, I developed a python udev script that manages device registration base on a YAML configuration file. I even patched bcache-tools to support the new IOCTL registration. You will find a link to the Github project at the end of this message.
Fewer dependencies are better: There are still python2 vs python3 conflicts out there---and loading python and its dependencies into an initrd/initramfs for early bcache loading could be very messy, indeed!
Quoted 7 lines
You've already put some work into make-bcache so creating a
bcache_udevfunction and abcache-udev.cfile (likemake-bcache.c) is probably easy enough. IMHO, a single-line grepable format (instead of YAML) could be used to minimize dependencies so that it is clean in an initramfs in the same way that mdadm.conf and crypttab already do. You could then parse it with C or bash pretty easily...
I will be really glad to rework the patch if we can agree on some modifications that will make it suitable to be merged upstream.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
You want to make sure that when a dirty writeback-cached bdev is registered that it does not activate until either:
1. The cache is attached
2. echo 1 >/sys/block/bcache0/bcache/runningQuoted 1 line
- A new IOCTL interface added, even all rested interface are sysfs based.
True. Perhaps something like this would be better, and it would avoid sharing a userspace page for blindly populating
struct cache_sb, too:echo '
/dev/bdev[bdev_uuid] [/dev/cachedev|cset_uuid] [options]' > ¥
/sys/fs/bcache/register_raw
Quoted 7 lines
Quoted 4 lines
I thought that introducing an IOCTL interface could be a stopper for this patch, so I'm more than welcome to refactor it using sysfs entries only. But, I will do so only if there's even the slightest chance that this patch can be merged.
I'm for the patch with sysfs, but I'm not the decision maker either.
As Eric already asked,
Quoted 2 lines
Coly: what would your requirements be to accept this upstream?
Andrea
Quoted 117 lines
Quoted 82 lines
Quoted 74 lines
Because of the writeback issue above, the cache and bdev either need to be registered simultaneously or the cset uuid need to be specified.
Quoted 5 lines
- Do things in kernel space while there is solution in user space.
All the above opinions are quite personal to myself, I don’t say you are wrong or I am correct. If the patch works, that’s cool and you can use it as you want, but I don’t support it to be in upstream.
An alternate implementation might be to create a dm-bcache target. The core bcache code could be left alone except a few EXPORT_SYMBOL()'s so dm-bcache can reach the bcache registration bits.
This would:
* Minimally impact the bcache code base
* Solve the blind-populating ofstruct cache_sbissue in the same way asregister_rawcould work above.
* Create a nicely segmented codebase (dm target) to upstream separately through the DM team.
* Could be maintained cleanly out-of-tree because the bcache registration interfaces change very infrequently.
* bdev resize could be done with admsetup replacebut you'll need to add resize support as below.Quoted 8 lines
Quoted 4 lines
For the operational drawbacks: from time to time, we exploit the online filesystem resize capability of XFS to increase the volume size. This would be at least tedious, if not nearly impossible, if the volume is mapped inside a dm-linear.
Currently bcache doesn’t support cache or backing device resize. I don’t see the connection between above statement and feeding an in-memory super block via IOCTL command.
Resize is perhaps unrelated, so if you decide to tackle bdev or cachedev resize then please start a new list thread. Briefly: supporting bdev resize is probably easy. I've looked through the code a few times with this in mind but haven't had time.
Here's the summary, not sure if there are any other considerations:
1. Create a sysfs file like
/sys/block/bcache0/bcache/resizeto trigger resize on echo 1 >.
2. Refactor theset_capacity()bits frombcache_device_init()into its own function.
3. Put locks around bcache_device.full_dirty_stripes and bcache_device.stripe_sectors_dirty. Re-alloc+copy+free and zero the new bytes at the end.The cachedev's don't know anything about the bdev size, so (according to Kent) they will "just work" by referencing new offsets that didn't exist before when IOs come through. (This is basically the same as resizing the bdev and then unregister/re-register which is how we resize bdevs now.)
As for resizing a cachedev, I've not looked at all---not sure about that one. We always detach, resize, make-bcache and re-attach the new cache. Maybe it is similarly simple, but haven't looked.
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
It is unnecessary to create a new IOCTL interface, and I feel the way how it works is really unconvinced for potential security risk.
Which potential security risks concern you?
Currently we don’t check all members of struct
cache_sb_disk, what we do is to simply trust bcache-tools. Create a new IOCTL interface to send a user space memory into kernel space as superblock, that needs quite a lot of checking code to make sure it won’t panic the kernel. It is possible, but it is not worthy to add so many code for the purpose, especially adding them into upstream code.Quoted 2 lines
I am not able to provide an explicit example for security risk, just the new adding interface makes me not so confident.
Quoted 7 lines
Maintaining a blind structure population from a userspace page would be difficult as well because even if the kernel validates _everything_ in
cache_sbtoday, anyone extendingstruct cache_sbneeds to remember to add checks for that. Failing to enforce validation inside the kernel could lead to kernel crashes or data corruption from userspace which is of course never good.Quoted 2 lines
Can I solve this by refactoring the patch and using sysfs based registration instead?
Sure, just make the sysfs interface configurable enough that:
- all configuration is text (not binary)
- bcache.txt documentation is updated-Eric
Quoted 26 lines
For anyone that is interested to try this solution, and as a reference, I leave here below the links to my public Github repositories:
- bcache-tools: https://github.com/andreatomassetti/bcache-tools
- bcache (standalone, patched): https://github.com/andreatomassetti/bcacheThank you very much,
AndreaQuoted 16 lines
We always assume that, somehow, someone could leverage an insecure IOCTL call and crash the OS when they shouldn't be allowed to (even if they are not root, like from sudo). This is a security issue from an assurance and integrity point of view, even if there might be neither an obvious privelege escalation nor privacy concern.
-Eric
Quoted 7 lines
Thanks.
Coly Li
Quoted 63 lines
2022年7月20日 16:06,Andrea Tomassetti <[email protected]> 写道:
Quoted 51 lines · Eric Wheeler
Quoted 49 lines · Andrea Tomassetti
Quoted 47 lines · Eric Wheeler
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
Quoted 7 lines
Quoted 2 lines
That's exactly what was preventing us from using bcache in production prior to this patch.
Ok, but we always use writeback...and others may wish to, too. I think any patch that introduces a feature needs to support existing features without introducing limitations on the behavior.
Quoted 10 lines
Totally agree. My only point was that I extensively tested this patch with wt mode. It works in wb mode as well, for sure because the backing device's header is almost never used. The only issue I can foresee in wb mode is in case of a machine reboot: the backing device will lose the virtual header and, at boot time, another one will be generated. It will get attached again to its cache device with a new UID and I'm not sure if this will imply the loss of the data that was not previously written to it, but was only present on the cache device. But I think that losing data it's a well-known risk of wb mode.
NO, losing dirty data on cache device is unacceptable. If the previous attached cache device is not ready, the backing device will be suspended and its bcache device won’t show up in /dev/.
Quoted 50 lines
Quoted 43 lines
Quoted 30 lines
Quoted 30 lines
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to
dd‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for
bch_write_bdev_super()there does not appear to be a way to handle attaching caches to the running bcache. For example:1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
Quoted 8 lines
Quoted 4 lines
I could imagine something like
/etc/bcachetab(like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.Yes, in conjunction with this patch, I developed a python udev script that manages device registration base on a YAML configuration file. I even patched bcache-tools to support the new IOCTL registration. You will find a link to the Github project at the end of this message.
Fewer dependencies are better: There are still python2 vs python3 conflicts out there---and loading python and its dependencies into an initrd/initramfs for early bcache loading could be very messy, indeed!
Quoted 7 lines
You've already put some work into make-bcache so creating a
bcache_udevfunction and abcache-udev.cfile (likemake-bcache.c) is probably easy enough. IMHO, a single-line grepable format (instead of YAML) could be used to minimize dependencies so that it is clean in an initramfs in the same way that mdadm.conf and crypttab already do. You could then parse it with C or bash pretty easily...
Quoted 2 lines
I will be really glad to rework the patch if we can agree on some modifications that will make it suitable to be merged upstream.
I don’t support the idea to copy a block of memory from user space to kernel space and reference it as a super block, neither IOCTL nor sysfs interface. It is very easy to generate a corrupted super block memory and send it into kernel space and panic the whole system, I will not take this potential risk.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 7 lines
You want to make sure that when a dirty writeback-cached bdev is registered that it does not activate until either:
1. The cache is attached
2. echo 1 >/sys/block/bcache0/bcache/runningQuoted 7 lines
Quoted 1 line
- A new IOCTL interface added, even all rested interface are sysfs based.
True. Perhaps something like this would be better, and it would avoid sharing a userspace page for blindly populating
struct cache_sb, too:echo '
/dev/bdev[bdev_uuid] [/dev/cachedev|cset_uuid] [options]' > ¥/sys/fs/bcache/register_rawI thought that introducing an IOCTL interface could be a stopper for this patch, so I'm more than welcome to refactor it using sysfs entries only. But, I will do so only if there's even the slightest chance that this patch can be merged.
Quoted 6 lines
Quoted 2 lines
I'm for the patch with sysfs, but I'm not the decision maker either.
As Eric already asked,
Quoted 2 lines
Coly: what would your requirements be to accept this upstream?
Quoted 1 line
Quoted 1 line
Quoted 1 line
You may use it as you want, but I won’t accept it into upstream via my path.
Coly Li
Quoted 75 lines · Coly Li
Quoted 64 lines
2022年7月20日 16:06,Andrea Tomassetti <[email protected]> 写道:
Quoted 51 lines · Eric Wheeler
Quoted 49 lines · Andrea Tomassetti
Quoted 47 lines · Eric Wheeler
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
Quoted 7 lines
Quoted 2 lines
That's exactly what was preventing us from using bcache in production prior to this patch.
Ok, but we always use writeback...and others may wish to, too. I think any patch that introduces a feature needs to support existing features without introducing limitations on the behavior.
Quoted 10 lines
Totally agree. My only point was that I extensively tested this patch with wt mode. It works in wb mode as well, for sure because the backing device's header is almost never used. The only issue I can foresee in wb mode is in case of a machine reboot: the backing device will lose the virtual header and, at boot time, another one will be generated. It will get attached again to its cache device with a new UID and I'm not sure if this will imply the loss of the data that was not previously written to it, but was only present on the cache device. But I think that losing data it's a well-known risk of wb mode.
Quoted 56 lines
NO, losing dirty data on cache device is unacceptable. If the previous attached cache device is not ready, the backing device will be suspended and its bcache device won’t show up in
/dev/.Quoted 52 lines
Quoted 43 lines
Quoted 30 lines
Quoted 30 lines
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to
dd‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for
bch_write_bdev_super()there does not appear to be a way to handle attaching caches to the running bcache. For example:1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
Quoted 8 lines
Quoted 4 lines
I could imagine something like
/etc/bcachetab(like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.Yes, in conjunction with this patch, I developed a python udev script that manages device registration base on a YAML configuration file. I even patched bcache-tools to support the new IOCTL registration. You will find a link to the Github project at the end of this message.
Fewer dependencies are better: There are still python2 vs python3 conflicts out there---and loading python and its dependencies into an initrd/initramfs for early bcache loading could be very messy, indeed!
Quoted 7 lines
You've already put some work into make-bcache so creating a
bcache_udevfunction and abcache-udev.cfile (likemake-bcache.c) is probably easy enough. IMHO, a single-line grepable format (instead of YAML) could be used to minimize dependencies so that it is clean in an initramfs in the same way that mdadm.conf and crypttab already do. You could then parse it with C or bash pretty easily...I will be really glad to rework the patch if we can agree on some modifications that will make it suitable to be merged upstream.
Hi Coly,
thank you very much for your time.
Quoted 4 lines
I don’t support the idea to copy a block of memory from user space to kernel space and reference it as a super block, neither IOCTL nor sysfs interface. It is very easy to generate a corrupted super block memory and send it into kernel space and panic the whole system, I will not take this potential risk.
I think I'm missing something here because I cannot see the difference between passing the structure through the sysfs interface or reading it from the header of the block device. In both cases the source of such structure will be the same: the user via the make-bcache command. My understanding of the part involved is:
udev_rule -> bcache-register.c -> sysfs/register ->
bcache_module/register_bcache -> read_super
So, in read_super the bcache module will read what the userspace utility make-bcache wrote as a super block. Correct?
If I'm correct, a big if, I cannot see why it should be "easier" to generate a corrupted super block with this patch. Can you please elaborate on this?
Thank you in advance,
Andrea
Quoted 32 lines
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 7 lines
You want to make sure that when a dirty writeback-cached bdev is registered that it does not activate until either:
1. The cache is attached
2. echo 1 >/sys/block/bcache0/bcache/runningQuoted 7 lines
Quoted 1 line
- A new IOCTL interface added, even all rested interface are sysfs based.
True. Perhaps something like this would be better, and it would avoid sharing a userspace page for blindly populating
struct cache_sb, too:echo '
/dev/bdev[bdev_uuid] [/dev/cachedev|cset_uuid] [options]' > ¥/sys/fs/bcache/register_rawI thought that introducing an IOCTL interface could be a stopper for this patch, so I'm more than welcome to refactor it using sysfs entries only. But, I will do so only if there's even the slightest chance that this patch can be merged.
Quoted 6 lines
Quoted 2 lines
I'm for the patch with sysfs, but I'm not the decision maker either.
As Eric already asked,
Quoted 2 lines
Coly: what would your requirements be to accept this upstream?
Quoted 1 line
Quoted 1 line
Quoted 1 line
You may use it as you want, but I won’t accept it into upstream via my path.
Coly Li
Quoted 148 lines
2022年7月20日 22:23,Andrea Tomassetti <[email protected]> 写道:
Quoted 75 lines · Coly Li
Quoted 64 lines
2022年7月20日 16:06,Andrea Tomassetti <[email protected]> 写道:
Quoted 51 lines · Eric Wheeler
Quoted 49 lines · Andrea Tomassetti
Quoted 47 lines · Eric Wheeler
Quoted 30 lines · Coly Li
Quoted 19 lines
2022年7月5日 16:48,Andrea Tomassetti <[email protected]> 写道:
Quoted 11 lines · Coly Li
Quoted 7 lines
2022年7月4日 23:13,Andrea Tomassetti <[email protected]> 写道: Introducing a bcache control device (
/dev/bcache_ctrl) that allows communicating to the driver from user space via IOCTL. The only IOCTL commands currently implemented, receives a structcache_sband uses it to register the backing device without any need of formatting them.Back to the patch, I don’t support this idea. For the problem you are solving, indeed people uses device mapper linear target for many years, and it works perfectly without any code modification.
Quoted 6 lines
That is, create a 8K image and set it as a loop device, then write a dm table to combine it with the existing hard drive. Then you run “bcache make -B <combined dm target>”, you will get a bcache device whose first 8K in the loop device and existing super block of the hard driver located at expected offset.
Quoted 10 lines
We evaluated this option but we weren't satisfied by the outcomes for, at least, two main reasons: complexity and operational drawbacks. For the complexity side: in production we use more than 20 HD that need to be cached. It means we need to create 20+ header for them, 20+ loop devices and 20+ dm linear devices. So, basically, there's a 3x factor for each HD that we want to cache. Moreover, we're currently using ephemeral disks as cache devices. In case of a machine reboot, ephemeral devices can get lost; at this point, there will be some trouble to mount the dm-linear bcache backing device because there will be no cache device.
Quoted 4 lines
OK, I get your point. It might make sense for your situation, although I know some other people use the linear dm-table to solve similar situation but may be not perfect for you. This patch may work in your procedure, but there are following things make me uncomfortable,
Coly is right: there are some issues to address.
Coly, I do not wish to contradict you, only to add that we have had times where this would be useful. I like the idea, particularly avoiding placing dm-linear atop of the bdev which reduces the IO codepath. Maybe there is an implementation that would fit everyone's needs.
For us, such a superblock-less registration could resolve two issues:
1. Most commonly we wish to add bcache to an existing device without re-formatting and without adding the dm-linear complexity.
Quoted 7 lines
Quoted 2 lines
That's exactly what was preventing us from using bcache in production prior to this patch.
Ok, but we always use writeback...and others may wish to, too. I think any patch that introduces a feature needs to support existing features without introducing limitations on the behavior.
Quoted 10 lines
Totally agree. My only point was that I extensively tested this patch with wt mode. It works in wb mode as well, for sure because the backing device's header is almost never used. The only issue I can foresee in wb mode is in case of a machine reboot: the backing device will lose the virtual header and, at boot time, another one will be generated. It will get attached again to its cache device with a new UID and I'm not sure if this will imply the loss of the data that was not previously written to it, but was only present on the cache device. But I think that losing data it's a well-known risk of wb mode.
Quoted 56 lines
NO, losing dirty data on cache device is unacceptable. If the previous attached cache device is not ready, the backing device will be suspended and its bcache device won’t show up in
/dev/.Quoted 52 lines
Quoted 43 lines
Quoted 30 lines
Quoted 30 lines
2. Relatedly, IMHO, I've never liked the default location at 8k because we like to align our bdev's to the RAID stride width and rarely is the bdev array aligned at 8k (usually 64k or 256k for hardware RAID). If we forget to pass the --data-offset at make-bcache time then we are stuck with an 8k-misalignment for the life of the device.
In #2 we usually reformat the volume to avoid dm-linear complexity (and in both cases we have wanted writeback cache support). This process can take a while to
dd‾30TBs of bdev on spinning disks and we have to find temporary disk space to move that data out and back in.Quoted 1 line
- Copying a user space memory and directly using it as a complicated in-kernel memory object.
In the ioctl changes for
bch_write_bdev_super()there does not appear to be a way to handle attaching caches to the running bcache. For example:1. Add a cacheless bcache0 volume via ioctl
2. Attach a writeback cache, write some data.
3. Unregister everything
4. Re-register the _dirty_ bdev from #1In this scenario bcache will start "running" immediately because it cannot update its cset.uuid (as reported by bcache-super-show) which I believe is stored in cache_sb.set_uuid.
I suppose in step #2 you could update your userspace state with the cset.uuid so your userspace ioctl register tool would put the cset.uuid in place, but this is fragile without good userspace integration.
Quoted 8 lines
Quoted 4 lines
I could imagine something like
/etc/bcachetab(like crypttab or mdadm.conf) that maps cset UUID's to bdev UUIDs. Then your userspace ioctl tool could be included in a udev script to auto-load volumes as they become available.Yes, in conjunction with this patch, I developed a python udev script that manages device registration base on a YAML configuration file. I even patched bcache-tools to support the new IOCTL registration. You will find a link to the Github project at the end of this message.
Fewer dependencies are better: There are still python2 vs python3 conflicts out there---and loading python and its dependencies into an initrd/initramfs for early bcache loading could be very messy, indeed!
Quoted 7 lines
You've already put some work into make-bcache so creating a
bcache_udevfunction and abcache-udev.cfile (likemake-bcache.c) is probably easy enough. IMHO, a single-line grepable format (instead of YAML) could be used to minimize dependencies so that it is clean in an initramfs in the same way that mdadm.conf and crypttab already do. You could then parse it with C or bash pretty easily...I will be really glad to rework the patch if we can agree on some modifications that will make it suitable to be merged upstream.
Hi Coly,
thank you very much for your time.Quoted 4 lines
I don’t support the idea to copy a block of memory from user space to kernel space and reference it as a super block, neither IOCTL nor sysfs interface. It is very easy to generate a corrupted super block memory and send it into kernel space and panic the whole system, I will not take this potential risk.
I think I'm missing something here because I cannot see the difference between passing the structure through the sysfs interface or reading it from the header of the block device. In both cases the source of such structure will be the same: the user via the make-bcache command. My understanding of the part involved is:
Quoted 7 lines
udev_rule -> bcache-register.c -> sysfs/register ->bcache_module/register_bcache ->
read_super
So, inread_superthe bcache module will read what the userspace utility make-bcache wrote as a super block. Correct?
If I'm correct, a big if, I cannot see why it should be "easier" to generate a corrupted super block with this patch. Can you please elaborate on this?
If you do all things correctly, all things will work as you mentioned. But if some cracker tries to use this interface to send a special composed super block which has incorrect journal bucket LBAs and corrupts cached data, what should you do?
Now bcache kernel code fully trusts bcache-tools, if someone modifies bcache-tools and generate corrupted super block, he or she may pay more effort on it. But the memory copy interface makes similar risky things too much easier. I am not an expert on code attack, and am not able to provide an exact example. But creating an interface and permit user space to send arbitrary memory object into kernel space, this is really risky and I won’t take it for upstream kernel.
Coly Li