bcache: device failure handling improvement
Subsystems (2)
v4 cover bcache: device failure handling improvement
Hi maintainers and folks,
This patch set tries to improve bcache device failure handling, includes cache device and backing device failures.
The basic idea to handle failed cache device is,
- Unregister cache set
- Detach all backing devices which are attached to this cache set
- Stop all the detached bcache devices (configurable)
- Stop all flash only volume on the cache set
The above process is named 'cache set retire' by me. The result of cache set retire is, cache set and bcache devices are all removed, following I/O requests will get failed immediately to notift upper layer or user space coce that the cache device is failed or disconnected.
For failed backing device, there are two kinds of failures to handle,
- If device is disconnected, and kernel thread dc->status_update_thread finds it is offline for BACKING_DEV_OFFLINE_TIMEOUT (5) seconds, the kernel thread will set dc->io_disable and call bcache_device_stop() to stop and remove the bcache device from system.
- If device is alive but returns too many I/O errors, after errors number exceeds dc->error_limit, call bch_cached_dev_error() to set dc->io_disable and stop bcache device. Then the broken backing device and its bcache device will be removed from system.
The v4 patch set combines two v3 patches into one, and adds one more patch to permit users to explicitly avoid stopping attached bcache device from a retiring cache set. This is a configurable option suggested by Nix <[email protected]>.
Some patches of this patch set is already in bcache-for-next and not included here anymore. Most of the patches are reviewed by Hannes Reinecke and Junhui Tang. There are still severl patches need to be reviewed,
- [PATCH v4 05/13] bcache: stop dc->writeback_rate_update properly
- [PATCH v4 13/13] bcache: add stop_attached_devs_on_fail to struct cached_dev
Any comment, question and review are warmly welcome. Thanks in advance.
Changelog:
v4: add per-cached_dev option stop_attached_devs_on_fail to avoid stopping
attached bcache device from a retiring cache set.
v3: fix detach issue find in v2 patch set.
v2: fixes all problems found in v1 review.
add patches to handle backing device failure.
add one more patch to set writeback_rate_update_seconds range. include a patch from Junhui Tang.
v1: the initial version, only handles cache device failure.
Coly Li
---
Coly Li (12):
bcache: set writeback_rate_update_seconds in range [1, 60] seconds
bcache: properly set task state in bch_writeback_thread()
bcache: fix cached_dev->count usage for bch_cache_set_error()
bcache: quit dc->writeback_thread when BCACHE_DEV_DETACHING is set
bcache: stop dc->writeback_rate_update properly
bcache: set error_limit correctly
bcache: add CACHE_SET_IO_DISABLE to struct cache_set flags
bcache: stop all attached bcache devices for a retired cache set
bcache: add backing_request_endio() for bi_end_io of attached backing
device I/O
bcache: add io_disable to struct cached_dev
bcache: stop bcache device when backing device is offline
bcache: add stop_attached_devs_on_fail to struct cached_dev
Tang Junhui (1):
bcache: fix inaccurate io state for detached bcache devices
drivers/md/bcache/alloc.c | 5 +- drivers/md/bcache/bcache.h | 38 ++++++++- drivers/md/bcache/btree.c | 10 ++- drivers/md/bcache/io.c | 16 +++- drivers/md/bcache/journal.c | 4 +- drivers/md/bcache/request.c | 187 +++++++++++++++++++++++++++++++++++------- drivers/md/bcache/super.c | 181 ++++++++++++++++++++++++++++++++++++---- drivers/md/bcache/sysfs.c | 55 ++++++++++++- drivers/md/bcache/util.h | 6 -- drivers/md/bcache/writeback.c | 99 ++++++++++++++++++---- drivers/md/bcache/writeback.h | 5 +- 11 files changed, 522 insertions(+), 84 deletions(-)
Signature · 1 line
2.15.1
PATCH v4 1/13 bcache: set writeback_rate_update_seconds in range [1, 60] seconds
dc->writeback_rate_update_seconds can be set via sysfs and its value can be set to [1, ULONG_MAX]. It does not make sense to set such a large value, 60 seconds is long enough value considering the default 5 seconds works well for long time.
Because dc->writeback_rate_update is a special delayed work, it re-arms itself inside the delayed work routine update_writeback_rate(). When stopping it by cancel_delayed_work_sync(), there should be a timeout to wait and make sure the re-armed delayed work is stopped too. A small max value of dc->writeback_rate_update_seconds is also helpful to decide a reasonable small timeout.
This patch limits sysfs interface to set dc->writeback_rate_update_seconds in range of [1, 60] seconds, and replaces the hand-coded number by macros.
--- drivers/md/bcache/sysfs.c | 3 +++ drivers/md/bcache/writeback.c | 2 +- drivers/md/bcache/writeback.h | 3 +++ 3 files changed, 7 insertions(+), 1 deletion(-)
drivers/md/bcache/sysfs.c +3 -0
@@ -215,6 +215,9 @@ STORE(__cached_dev)sysfs_strtoul_clamp (writeback_rate, dc->writeback_rate.rate,1 ,INT_MAX ); +sysfs_strtoul_clamp (writeback_rate_update_seconds, + dc->writeback_rate_update_seconds, +1 ,WRITEBACK_RATE_UPDATE_SECS_MAX );d_strtoul_nonzero (writeback_rate_update_seconds);d_strtoul (writeback_rate_i_term_inverse);d_strtoul_nonzero (writeback_rate_p_term_inverse);
drivers/md/bcache/writeback.c +1 -1
@@ -652,7 +652,7 @@ void bch_cached_dev_writeback_init(struct cached_dev *dc) dc->writeback_rate.rate =1024 ; dc->writeback_rate_minimum =8 ; - dc->writeback_rate_update_seconds =5 ; + dc->writeback_rate_update_seconds =WRITEBACK_RATE_UPDATE_SECS_DEFAULT ; dc->writeback_rate_p_term_inverse =40 ; dc->writeback_rate_i_term_inverse =10000 ;
drivers/md/bcache/writeback.h +3 -0
@@ -8,6 +8,9 @@#define MAX_WRITEBACKS_IN_PASS 5#define MAX_WRITESIZE_IN_PASS 5000/* *512b */ +#define WRITEBACK_RATE_UPDATE_SECS_MAX 60 +#define WRITEBACK_RATE_UPDATE_SECS_DEFAULT 5 +/* * 14 (16384ths) is chosen here as something that each backing device * should be a reasonable fraction of the share, and not to blow up
Signature · 1 line
2.15.1
Re: [PATCH v4 01/13] bcache: set writeback_rate_update_seconds in range [1, 60] seconds
Quoted 17 lines · Coly Li
dc->writeback_rate_update_secondscan be set via sysfs and its value can be set to [1,ULONG_MAX]. It does not make sense to set such a large value, 60 seconds is long enough value considering the default 5 seconds works well for long time.Because
dc->writeback_rate_updateis a special delayed work, it re-arms itself inside the delayed work routineupdate_writeback_rate(). When stopping it bycancel_delayed_work_sync(), there should be a timeout to wait and make sure the re-armed delayed work is stopped too. A small max value ofdc->writeback_rate_update_secondsis also helpful to decide a reasonable small timeout.This patch limits sysfs interface to set
dc->writeback_rate_update_secondsin range of [1, 60] seconds, and replaces the hand-coded number by macros.Signed-off-by Coly Li <[email protected]>Reviewed-by Hannes Reinecke <[email protected]>
[snip]
Quoted 11 lines
--- drivers/md/bcache/sysfs.c | 3 +++ drivers/md/bcache/writeback.c | 2 +- drivers/md/bcache/writeback.h | 3 +++ 3 files changed, 7 insertions(+), 1 deletion(-)drivers/md/bcache/sysfs.c +0
-0
drivers/md/bcache/sysfs.c STORE(__cached_dev) · line 221
Quoted 7 lines
drivers/md/bcache/sysfs.c +3
-0@@ -215,6 +215,9 @@ STORE(__cached_dev)sysfs_strtoul_clamp (writeback_rate, dc->writeback_rate.rate,1 ,INT_MAX ); +sysfs_strtoul_clamp (writeback_rate_update_seconds, + dc->writeback_rate_update_seconds, +1 ,WRITEBACK_RATE_UPDATE_SECS_MAX );d_strtoul_nonzero (writeback_rate_update_seconds);
Doesn't the above line need to be removed for this to work?
Mike
PATCH v4 2/13 bcache: properly set task state in bch_writeback_thread()
In reply to Coly LiKernel thread routine bch_writeback_thread() has the following code block,
447 down_write(&dc->writeback_lock);
448~450 if (check conditions) {
451 up_write(&dc->writeback_lock);
452 set_current_state(TASK_INTERRUPTIBLE);
453
454 if (kthread_should_stop())
455 return 0;
456
457 schedule();
458 continue;
459 }
If condition check is true, its task state is set to TASK_INTERRUPTIBLE and call schedule() to wait for others to wake up it.
There are 2 issues in current code,
1, Task state is set to TASK_INTERRUPTIBLE after the condition checks, if
another process changes the condition and call wake_up_process(dc-> writeback_thread), then at line 452 task state is set back to TASK_INTERRUPTIBLE, the writeback kernel thread will lose a chance to be waken up.
2, At line 454 if kthread_should_stop() is true, writeback kernel thread
will return to kernel/kthread.c:kthread() with TASK_INTERRUPTIBLE and call do_exit(). It is not good to enter do_exit() with task state TASK_INTERRUPTIBLE, in following code path might_sleep() is called and a warning message is reported by __might_sleep(): "WARNING: do not call blocking ops when !TASK_RUNNING; state=1 set at [xxxx]".
For the first issue, task state should be set before condition checks. Ineed because dc->writeback_lock is required when modifying all the conditions, calling set_current_state() inside code block where dc-> writeback_lock is hold is safe. But this is quite implicit, so I still move set_current_state() before all the condition checks.
For the second issue, frankley speaking it does not hurt when kernel thread exits with TASK_INTERRUPTIBLE state, but this warning message scares users, makes them feel there might be something risky with bcache and hurt their data. Setting task state to TASK_RUNNING before returning fixes this problem.
In alloc.c:allocator_wait(), there is also a similar issue, and is also fixed in this patch.
Changelog:
v3: merge two similar fixes into one patch
v2: fix the race issue in v1 patch.
v1: initial buggy fix.
--- drivers/md/bcache/alloc.c | 4 +++- drivers/md/bcache/writeback.c | 7 +++++-- 2 files changed, 8 insertions(+), 3 deletions(-)
drivers/md/bcache/alloc.c +3 -1
@@ -287,8 +287,10 @@ do { \break ; \ \mutex_unlock (&(ca)->set->bucket_lock); \ -if (kthread_should_stop ()) \ +if (kthread_should_stop ()) { \ +set_current_state (TASK_RUNNING ); \return 0 ; \ + } \ \schedule (); \mutex_lock (&(ca)->set->bucket_lock); \
drivers/md/bcache/writeback.c +5 -2
@@ -564,18 +564,21 @@ static int bch_writeback_thread(void *arg)while (!kthread_should_stop ()) {down_write (&dc->writeback_lock); +set_current_state (TASK_INTERRUPTIBLE );if (!atomic_read (&dc->has_dirty) || (!test_bit (BCACHE_DEV_DETACHING , &dc->disk.flags) && !dc->writeback_running)) {up_write (&dc->writeback_lock); -set_current_state (TASK_INTERRUPTIBLE ); -if (kthread_should_stop ()) +if (kthread_should_stop ()) { +set_current_state (TASK_RUNNING );return 0 ; + }schedule ();continue ; } +set_current_state (TASK_RUNNING ); searched_full_index =refill_dirty (dc);
Signature · 1 line
2.15.1
Re: [PATCH v4 02/13] bcache: properly set task state in bch_writeback_thread()
Quoted 50 lines · Coly Li
Kernel thread routine
bch_writeback_thread()has the following code block,447
down_write(&dc->writeback_lock);
448~450 if (check conditions) {
451up_write(&dc->writeback_lock);
452set_current_state(TASK_INTERRUPTIBLE);
453
454 if (kthread_should_stop())
455 return 0;
456
457schedule();
458 continue;
459 }If condition check is true, its task state is set to
TASK_INTERRUPTIBLEand callschedule()to wait for others to wake up it.There are 2 issues in current code,
1, Task state is set toTASK_INTERRUPTIBLEafter the condition checks, if
another process changes the condition and callwake_up_process(dc-> writeback_thread), then at line 452 task state is set back toTASK_INTERRUPTIBLE, the writeback kernel thread will lose a chance to be waken up.
2, At line 454 ifkthread_should_stop()is true, writeback kernel thread
will return to kernel/kthread.c:kthread() withTASK_INTERRUPTIBLEand calldo_exit(). It is not good to enterdo_exit()with task stateTASK_INTERRUPTIBLE, in following code pathmight_sleep()is called and a warning message is reported by__might_sleep(): "WARNING: do not call blocking ops when !TASK_RUNNING; state=1 set at [xxxx]".For the first issue, task state should be set before condition checks. Ineed because
dc->writeback_lockis required when modifying all the conditions, callingset_current_state()inside code block where dc->writeback_lockis hold is safe. But this is quite implicit, so I still moveset_current_state()before all the condition checks.For the second issue, frankley speaking it does not hurt when kernel thread exits with
TASK_INTERRUPTIBLEstate, but this warning message scares users, makes them feel there might be something risky with bcache and hurt their data. Setting task state toTASK_RUNNINGbefore returning fixes this problem.In alloc.c:allocator_wait(), there is also a similar issue, and is also fixed in this patch.
Changelog:
v3: merge two similar fixes into one patch
v2: fix the race issue in v1 patch.
v1: initial buggy fix.
Quoted 4 lines
Signed-off-by Coly Li <[email protected]>Reviewed-by Hannes Reinecke <[email protected]>Cc Michael Lyle <[email protected]>Cc Junhui Tang <[email protected]>
This one LGTM-- applying to my staging branch
PATCH v4 3/13 bcache: fix cached_dev->count usage for bch_cache_set_error()
In reply to Coly LiWhen bcache metadata I/O fails, bcache will call bch_cache_set_error() to retire the whole cache set. The expected behavior to retire a cache set is to unregister the cache set, and unregister all backing device attached to this cache set, then remove sysfs entries of the cache set and all attached backing devices, finally release memory of structs cache_set, cache, cached_dev and bcache_device.
In my testing when journal I/O failure triggered by disconnected cache device, sometimes the cache set cannot be retired, and its sysfs entry /sys/fs/bcache/<uuid> still exits and the backing device also references it. This is not expected behavior.
When metadata I/O failes, the call senquence to retire whole cache set is,
bch_cache_set_error()
bch_cache_set_unregister()
bch_cache_set_stop()
__cache_set_unregister() <- called as callback by calling
clousre_queue(&c->caching)
cache_set_flush() <- called as a callback when refcount
of cache_set->caching is 0
cache_set_free() <- called as a callback when refcount
of catch_set->cl is 0
bch_cache_set_release() <- called as a callback when refcount
of catch_set->kobj is 0
I find if kernel thread bch_writeback_thread() quits while-loop when kthread_should_stop() is true and searched_full_index is false, clousre callback cache_set_flush() set by continue_at() will never be called. The result is, bcache fails to retire whole cache set.
cache_set_flush() will be called when refcount of closure c->caching is 0, and in function bcache_device_detach() refcount of closure c->caching is released to 0 by clousre_put(). In metadata error code path, function bcache_device_detach() is called by cached_dev_detach_finish(). This is a callback routine being called when cached_dev->count is 0. This refcount is decreased by cached_dev_put().
The above dependence indicates, cache_set_flush() will be called when refcount of cache_set->cl is 0, and refcount of cache_set->cl to be 0 when refcount of cache_dev->count is 0.
The reason why sometimes cache_dev->count is not 0 (when metadata I/O fails and bch_cache_set_error() called) is, in bch_writeback_thread(), refcount of cache_dev is not decreased properly.
In bch_writeback_thread(), cached_dev_put() is called only when searched_full_index is true and cached_dev->writeback_keys is empty, a.k.a there is no dirty data on cache. In most of run time it is correct, but when bch_writeback_thread() quits the while-loop while cache is still dirty, current code forget to call cached_dev_put() before this kernel thread exits. This is why sometimes cache_set_flush() is not executed and cache set fails to be retired.
The reason to call cached_dev_put() in bch_writeback_rate() is, when the cache device changes from clean to dirty, cached_dev_get() is called, to make sure during writeback operatiions both backing and cache devices won't be released.
Adding following code in bch_writeback_thread() does not work,
static int bch_writeback_thread(void *arg)
}
+ if (atomic_read(&dc->has_dirty))
+ cached_dev_put()
+
return 0;
}
because writeback kernel thread can be waken up and start via sysfs entry:
echo 1 > /sys/block/bcache<N>/bcache/writeback_running
It is difficult to check whether backing device is dirty without race and extra lock. So the above modification will introduce potential refcount underflow in some conditions.
The correct fix is, to take cached dev refcount when creating the kernel thread, and put it before the kernel thread exits. Then bcache does not need to take a cached dev refcount when cache turns from clean to dirty, or to put a cached dev refcount when cache turns from ditry to clean. The writeback kernel thread is alwasy safe to reference data structure from cache set, cache and cached device (because a refcount of cache device is taken for it already), and no matter the kernel thread is stopped by I/O errors or system reboot, cached_dev->count can always be used correctly.
The patch is simple, but understanding how it works is quite complicated.
Changelog:
v2: set dc->writeback_thread to NULL in this patch, as suggested by Hannes.
v1: initial version for review.
--- drivers/md/bcache/super.c | 1 - drivers/md/bcache/writeback.c | 11 ++++++++--- drivers/md/bcache/writeback.h | 2 -- 3 files changed, 8 insertions(+), 6 deletions(-)
drivers/md/bcache/super.c +0 -1
@@ -1052,7 +1052,6 @@ int bch_cached_dev_attach(struct cached_dev *dc, struct cache_set *c)if (BDEV_STATE (&dc->sb) ==BDEV_STATE_DIRTY ) {bch_sectors_dirty_init (&dc->disk);atomic_set (&dc->has_dirty,1 ); -refcount_inc (&dc->count);bch_writeback_queue (dc); }
drivers/md/bcache/writeback.c +8 -3
@@ -572,7 +572,7 @@ static int bch_writeback_thread(void *arg)if (kthread_should_stop ()) {set_current_state (TASK_RUNNING ); -return 0 ; +break ; }schedule ();
@@ -585,7 +585,6 @@ static int bch_writeback_thread(void *arg)if (searched_full_index &&RB_EMPTY_ROOT (&dc->writeback_keys.keys)) {atomic_set (&dc->has_dirty,0 ); -cached_dev_put (dc);SET_BDEV_STATE (&dc->sb,BDEV_STATE_CLEAN );bch_write_bdev_super (dc,NULL ); }
@@ -606,6 +605,9 @@ static int bch_writeback_thread(void *arg) } } + dc->writeback_thread =NULL ; +cached_dev_put (dc); +return 0 ; }
@@ -669,10 +671,13 @@ int bch_cached_dev_writeback_start(struct cached_dev *dc)if (!dc->writeback_write_wq)return -ENOMEM ; +cached_dev_get (dc); dc->writeback_thread =kthread_create (bch_writeback_thread, dc,"bcache_writeback" ); -if (IS_ERR (dc->writeback_thread)) +if (IS_ERR (dc->writeback_thread)) { +cached_dev_put (dc);return PTR_ERR (dc->writeback_thread); + }schedule_delayed_work (&dc->writeback_rate_update, dc->writeback_rate_update_seconds *HZ );
drivers/md/bcache/writeback.h +0 -2
@@ -105,8 +105,6 @@ static inline void bch_writeback_add(struct cached_dev *dc) {if (!atomic_read (&dc->has_dirty) && !atomic_xchg (&dc->has_dirty,1 )) { -refcount_inc (&dc->count); -if (BDEV_STATE (&dc->sb) !=BDEV_STATE_DIRTY ) {SET_BDEV_STATE (&dc->sb,BDEV_STATE_DIRTY );/* XXX: should do this synchronously */
Signature · 1 line
2.15.1
PATCH v4 4/13 bcache: quit dc->writeback_thread when BCACHE_DEV_DETACHING is set
In reply to Coly LiIn patch "bcache: fix cached_dev->count usage for bch_cache_set_error()", cached_dev_get() is called when creating dc->writeback_thread, and cached_dev_put() is called when exiting dc->writeback_thread. This modification works well unless people detach the bcache device manually by
'echo 1 > /sys/block/bcache<N>/bcache/detach'
Because this sysfs interface only calls bch_cached_dev_detach() which wakes up dc->writeback_thread but does not stop it. The reason is, before patch "bcache: fix cached_dev->count usage for bch_cache_set_error()", inside bch_writeback_thread(), if cache is not dirty after writeback, cached_dev_put() will be called here. And in cached_dev_make_request() when a new write request makes cache from clean to dirty, cached_dev_get() will be called there. Since we don't operate dc->count in these locations, refcount d->count cannot be dropped after cache becomes clean, and cached_dev_detach_finish() won't be called to detach bcache device.
This patch fixes the issue by checking whether BCACHE_DEV_DETACHING is set inside bch_writeback_thread(). If this bit is set and cache is clean (no existing writeback_keys), break the while-loop, call cached_dev_put() and quit the writeback thread.
Please note if cache is still dirty, even BCACHE_DEV_DETACHING is set the writeback thread should continue to perform writeback, this is the original design of manually detach.
I compose a separte patch because that patch "bcache: fix cached_dev->count usage for bch_cache_set_error()" already gets a "Reviewed-by:" from Hannes Reinecke. Also this fix is not trivial and good for a separate patch.
--- drivers/md/bcache/writeback.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
drivers/md/bcache/writeback.c +17 -3
@@ -565,9 +565,15 @@ static int bch_writeback_thread(void *arg)while (!kthread_should_stop ()) {down_write (&dc->writeback_lock);set_current_state (TASK_INTERRUPTIBLE ); -if (!atomic_read (&dc->has_dirty) || - (!test_bit (BCACHE_DEV_DETACHING , &dc->disk.flags) && - !dc->writeback_running)) { +/* +* If the bache device is detaching, skip here and continue +* to perform writeback. Otherwise, if no dirty data on cache, +* or there is dirty data on cache but writeback is disabled, +* the writeback thread should sleep here and wait for others +* to wake up it. +*/ +if (!test_bit (BCACHE_DEV_DETACHING , &dc->disk.flags) && + (!atomic_read (&dc->has_dirty) || !dc->writeback_running)) {up_write (&dc->writeback_lock);if (kthread_should_stop ()) {
@@ -587,6 +593,14 @@ static int bch_writeback_thread(void *arg)atomic_set (&dc->has_dirty,0 );SET_BDEV_STATE (&dc->sb,BDEV_STATE_CLEAN );bch_write_bdev_super (dc,NULL ); +/* +* If bcache device is detaching via sysfs interface, +* writeback thread should stop after there is no dirty +* data on cache. BCACHE_DEV_DETACHING flag is set in +* bch_cached_dev_detach(). +*/ +if (test_bit (BCACHE_DEV_DETACHING , &dc->disk.flags)) +break ; }up_write (&dc->writeback_lock);
Signature · 1 line
2.15.1
PATCH v4 5/13 bcache: stop dc->writeback_rate_update properly
In reply to Coly Listruct delayed_work writeback_rate_update in struct cache_dev is a delayed worker to call function update_writeback_rate() in period (the interval is defined by dc->writeback_rate_update_seconds).
When a metadate I/O error happens on cache device, bcache error handling routine bch_cache_set_error() will call bch_cache_set_unregister() to retire whole cache set. On the unregister code path, this delayed work is stopped by calling cancel_delayed_work_sync(&dc->writeback_rate_update).
dc->writeback_rate_update is a special delayed work from others in bcache. In its routine update_writeback_rate(), this delayed work is re-armed itself. That means when cancel_delayed_work_sync() returns, this delayed work can still be executed after several seconds defined by dc->writeback_rate_update_seconds.
The problem is, after cancel_delayed_work_sync() returns, the cache set unregister code path will continue and release memory of struct cache set. Then the delayed work is scheduled to run, __update_writeback_rate() will reference the already released cache_set memory, and trigger a NULL pointer deference fault.
This patch introduces two more bcache device flags,
- BCACHE_DEV_WB_RUNNING
bit set: bcache device is in writeback mode and running, it is OK for
dc->writeback_rate_update to re-arm itself.
bit clear:bcache device is trying to stop dc->writeback_rate_update,
this delayed work should not re-arm itself and quit.
- BCACHE_DEV_RATE_DW_RUNNING
bit set: routine update_writeback_rate() is executing. bit clear: routine update_writeback_rate() quits.
This patch also adds a function cancel_writeback_rate_update_dwork() to wait for dc->writeback_rate_update quits before cancel it by calling cancel_delayed_work_sync(). In order to avoid a deadlock by unexpected quit dc->writeback_rate_update, after time_out seconds this function will give up and continue to call cancel_delayed_work_sync().
And here I explain how this patch stops self re-armed delayed work properly with the above stuffs.
update_writeback_rate() sets BCACHE_DEV_RATE_DW_RUNNING at its beginning and clears BCACHE_DEV_RATE_DW_RUNNING at its end. Before calling cancel_writeback_rate_update_dwork() clear flag BCACHE_DEV_WB_RUNNING.
Before calling cancel_delayed_work_sync() wait utill flag BCACHE_DEV_RATE_DW_RUNNING is clear. So when calling cancel_delayed_work_sync(), dc->writeback_rate_update must be already re- armed, or quite by seeing BCACHE_DEV_WB_RUNNING cleared. In both cases delayed work routine update_writeback_rate() won't be executed after cancel_delayed_work_sync() returns.
Inside update_writeback_rate() before calling schedule_delayed_work(), flag BCACHE_DEV_WB_RUNNING is checked before. If this flag is cleared, it means someone is about to stop the delayed work. Because flag BCACHE_DEV_RATE_DW_RUNNING is set already and cancel_delayed_work_sync() has to wait for this flag to be cleared, we don't need to worry about race condition here.
If update_writeback_rate() is scheduled to run after checking BCACHE_DEV_RATE_DW_RUNNING and before calling cancel_delayed_work_sync() in cancel_writeback_rate_update_dwork(), it is also safe. Because at this moment BCACHE_DEV_WB_RUNNING is cleared with memory barrier. As I mentioned previously, update_writeback_rate() will see BCACHE_DEV_WB_RUNNING is clear and quit immediately.
Because there are more dependences inside update_writeback_rate() to struct cache_set memory, dc->writeback_rate_update is not a simple self re-arm delayed work. After trying many different methods (e.g. hold dc->count, or use locks), this is the only way I can find which works to properly stop dc->writeback_rate_update delayed work.
Changelog:
v2: Try to fix the race issue which is pointed out by Junhui.
v1: The initial version for review
--- drivers/md/bcache/bcache.h | 9 +++++---- drivers/md/bcache/super.c | 39 +++++++++++++++++++++++++++++++++++---- drivers/md/bcache/sysfs.c | 3 ++- drivers/md/bcache/writeback.c | 29 ++++++++++++++++++++++++++++- 4 files changed, 70 insertions(+), 10 deletions(-)
drivers/md/bcache/bcache.h +5 -4
@@ -258,10 +258,11 @@ struct bcache_device {struct gendisk *disk;unsigned long flags; -#define BCACHE_DEV_CLOSING 0 -#define BCACHE_DEV_DETACHING 1 -#define BCACHE_DEV_UNLINK_DONE 2 - +#define BCACHE_DEV_CLOSING 0 +#define BCACHE_DEV_DETACHING 1 +#define BCACHE_DEV_UNLINK_DONE 2 +#define BCACHE_DEV_WB_RUNNING 4 +#define BCACHE_DEV_RATE_DW_RUNNING 8unsigned nr_stripes;unsigned stripe_size;atomic_t *stripe_sectors_dirty;
drivers/md/bcache/super.c +35 -4
@@ -899,6 +899,32 @@ void bch_cached_dev_run(struct cached_dev *dc)pr_debug ("error creating sysfs link" ); } +/* +* If BCACHE_DEV_RATE_DW_RUNNING is set, it means routine of the delayed +* work dc->writeback_rate_update is running. Wait until the routine +* quits (BCACHE_DEV_RATE_DW_RUNNING is clear), then continue to +* cancel it. If BCACHE_DEV_RATE_DW_RUNNING is not clear after time_out +* seconds, give up waiting here and continue to cancel it too. +*/ +static void cancel_writeback_rate_update_dwork (struct cached_dev *dc) +{ +int time_out =WRITEBACK_RATE_UPDATE_SECS_MAX *HZ ; + +do { +if (!test_bit (BCACHE_DEV_RATE_DW_RUNNING , + &dc->disk.flags)) +break ; + time_out--; +schedule_timeout_interruptible (1 ); + }while (time_out >0 ); + +if (time_out ==0 ) +pr_warn ("bcache: give up waiting for " +"dc->writeback_write_update to quit" ); + +cancel_delayed_work_sync (&dc->writeback_rate_update); +} +static void cached_dev_detach_finish (struct work_struct *w) { struct cached_dev *dc = container_of(w, struct cached_dev, detach);
@@ -911,7 +937,9 @@ static void cached_dev_detach_finish(struct work_struct *w)mutex_lock (&bch_register_lock); -cancel_delayed_work_sync (&dc->writeback_rate_update); +if (test_and_clear_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) +cancel_writeback_rate_update_dwork (dc); +if (!IS_ERR_OR_NULL (dc->writeback_thread)) {kthread_stop (dc->writeback_thread); dc->writeback_thread =NULL ;
@@ -954,6 +982,7 @@ void bch_cached_dev_detach(struct cached_dev *dc)closure_get (&dc->disk.cl);bch_writeback_queue (dc); +cached_dev_put (dc); }
@@ -1079,14 +1108,16 @@ static void cached_dev_free(struct closure *cl) { struct cached_dev *dc = container_of(cl, struct cached_dev, disk.cl); -cancel_delayed_work_sync (&dc->writeback_rate_update); +mutex_lock (&bch_register_lock); + +if (test_and_clear_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) +cancel_writeback_rate_update_dwork (dc); +if (!IS_ERR_OR_NULL (dc->writeback_thread))kthread_stop (dc->writeback_thread);if (dc->writeback_write_wq)destroy_workqueue (dc->writeback_write_wq); -mutex_lock (&bch_register_lock); -if (atomic_read (&dc->running))bd_unlink_disk_holder (dc->bdev, dc->disk.disk);bcache_device_free (&dc->disk);
drivers/md/bcache/sysfs.c +2 -1
@@ -304,7 +304,8 @@ STORE(bch_cached_dev)bch_writeback_queue (dc);if (attr == &sysfs_writeback_percent) -schedule_delayed_work (&dc->writeback_rate_update, +if (!test_and_set_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) +schedule_delayed_work (&dc->writeback_rate_update, dc->writeback_rate_update_seconds *HZ );mutex_unlock (&bch_register_lock);
drivers/md/bcache/writeback.c +28 -1
@@ -115,6 +115,21 @@ static void update_writeback_rate(struct work_struct *work) struct cached_dev, writeback_rate_update); +/* +* should check BCACHE_DEV_RATE_DW_RUNNING before calling +* cancel_delayed_work_sync(). +*/ +set_bit (BCACHE_DEV_RATE_DW_RUNNING , &dc->disk.flags); +/* paired with where BCACHE_DEV_RATE_DW_RUNNING is tested */ +smp_mb (); + +if (!test_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) { +clear_bit (BCACHE_DEV_RATE_DW_RUNNING , &dc->disk.flags); +/* paired with where BCACHE_DEV_RATE_DW_RUNNING is tested */ +smp_mb (); +return ; + } +down_read (&dc->writeback_lock); if (atomic_read(&dc->has_dirty) &&
@@ -123,8 +138,18 @@ static void update_writeback_rate(struct work_struct *work)up_read (&dc->writeback_lock); -schedule_delayed_work (&dc->writeback_rate_update, +if (test_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) { +schedule_delayed_work (&dc->writeback_rate_update, dc->writeback_rate_update_seconds *HZ ); + } + +/* +* should check BCACHE_DEV_RATE_DW_RUNNING before calling +* cancel_delayed_work_sync(). +*/ +clear_bit (BCACHE_DEV_RATE_DW_RUNNING , &dc->disk.flags); +/* paired with where BCACHE_DEV_RATE_DW_RUNNING is tested */ +smp_mb (); }static unsigned writeback_delay (struct cached_dev *dc,unsigned sectors)
@@ -675,6 +700,7 @@ void bch_cached_dev_writeback_init(struct cached_dev *dc) dc->writeback_rate_p_term_inverse =40 ; dc->writeback_rate_i_term_inverse =10000 ; +WARN_ON (test_and_clear_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags));INIT_DELAYED_WORK (&dc->writeback_rate_update, update_writeback_rate); }
@@ -693,6 +719,7 @@ int bch_cached_dev_writeback_start(struct cached_dev *dc)return PTR_ERR (dc->writeback_thread); } +WARN_ON (test_and_set_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags));schedule_delayed_work (&dc->writeback_rate_update, dc->writeback_rate_update_seconds *HZ );
Signature · 1 line
2.15.1
PATCH v4 6/13 bcache: set error_limit correctly
In reply to Coly LiStruct cache uses io_errors for two purposes,
- Error decay: when cache set error_decay is set, io_errors is used to generate a small piece of delay when I/O error happens.
- I/O errors counter: in order to generate big enough value for error decay, I/O errors counter value is stored by left shifting 20 bits (a.k.a IO_ERROR_SHIFT).
In function bch_count_io_errors(), if I/O errors counter reaches cache set error limit, bch_cache_set_error() will be called to retire the whold cache set. But current code is problematic when checking the error limit, see the following code piece from bch_count_io_errors(),
90 if (error) {
91 char buf[BDEVNAME_SIZE];
92 unsigned errors = atomic_add_return(1 << IO_ERROR_SHIFT,
93 &ca->io_errors);
94 errors >>= IO_ERROR_SHIFT;
95
96 if (errors < ca->set->error_limit)
97 pr_err("%s: IO error on %s, recovering",
98 bdevname(ca->bdev, buf), m);
99 else
100 bch_cache_set_error(ca->set,
101 "%s: too many IO errors %s",
102 bdevname(ca->bdev, buf), m);
103 }
At line 94, errors is right shifting IO_ERROR_SHIFT bits, now it is real errors counter to compare at line 96. But ca->set->error_limit is initia- lized with an amplified value in bch_cache_set_alloc(),
1545 c->error_limit = 8 << IO_ERROR_SHIFT;
It means by default, in bch_count_io_errors(), before 8<<20 errors happened bch_cache_set_error() won't be called to retire the problematic cache device. If the average request size is 64KB, it means bcache won't handle failed device until 512GB data is requested. This is too large to be an I/O threashold. So I believe the correct error limit should be much less.
This patch sets default cache set error limit to 8, then in bch_count_io_errors() when errors counter reaches 8 (if it is default value), function bch_cache_set_error() will be called to retire the whole cache set. This patch also removes bits shifting when store or show io_error_limit value via sysfs interface.
Nowadays most of SSDs handle internal flash failure automatically by LBA address re-indirect mapping. If an I/O error can be observed by upper layer code, it will be a notable error because that SSD can not re-indirect map the problematic LBA address to an available flash block. This situation indicates the whole SSD will be failed very soon. Therefore setting 8 as the default io error limit value makes sense, it is enough for most of cache devices.
Changelog:
v2: add reviewed-by from Hannes.
v1: initial version for review.
--- drivers/md/bcache/bcache.h | 1 + drivers/md/bcache/super.c | 2 +- drivers/md/bcache/sysfs.c | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-)
drivers/md/bcache/bcache.h +1 -0
@@ -663,6 +663,7 @@ struct cache_set {ON_ERROR_UNREGISTER ,ON_ERROR_PANIC , } on_error; +#define DEFAULT_IO_ERROR_LIMIT 8unsigned error_limit;unsigned error_decay;
drivers/md/bcache/super.c +1 -1
@@ -1583,7 +1583,7 @@ struct cache_set *bch_cache_set_alloc(struct cache_sb *sb) c->congested_read_threshold_us =2000 ; c->congested_write_threshold_us =20000 ; - c->error_limit =8 <<IO_ERROR_SHIFT ; + c->error_limit =DEFAULT_IO_ERROR_LIMIT ;return c; err:
drivers/md/bcache/sysfs.c +2 -2
@@ -560,7 +560,7 @@ SHOW(__bch_cache_set)/* See count_io_errors for why 88 */ sysfs_print (io_error_halflife, c->error_decay *88 ); -sysfs_print (io_error_limit, c->error_limit >>IO_ERROR_SHIFT ); +sysfs_print (io_error_limit, c->error_limit);sysfs_hprint (congested, ((uint64_t )bch_get_congested (c)) <<9 );
@@ -660,7 +660,7 @@ STORE(__bch_cache_set) }if (attr == &sysfs_io_error_limit) - c->error_limit =strtoul_or_return (buf) <<IO_ERROR_SHIFT ; + c->error_limit =strtoul_or_return (buf);/* See count_io_errors() for why 88 */ if (attr == &sysfs_io_error_halflife)
Signature · 1 line
2.15.1
Re: [PATCH v4 06/13] bcache: set error_limit correctly
Quoted 6 lines · Coly Li
Struct cache uses
io_errorsfor two purposes,
- Error decay: when cache seterror_decayis set,io_errorsis used to generate a small piece of delay when I/O error happens.
- I/O errors counter: in order to generate big enough value for error decay, I/O errors counter value is stored by left shifting 20 bits (a.k.aIO_ERROR_SHIFT).
This LGTM
PATCH v4 7/13 bcache: add CACHE_SET_IO_DISABLE to struct cache_set flags
In reply to Coly LiWhen too many I/Os failed on cache device, bch_cache_set_error() is called in the error handling code path to retire whole problematic cache set. If new I/O requests continue to come and take refcount dc->count, the cache set won't be retired immediately, this is a problem.
Further more, there are several kernel thread and self-armed kernel work may still running after bch_cache_set_error() is called. It needs to wait quite a while for them to stop, or they won't stop at all. They also prevent the cache set from being retired.
The solution in this patch is, to add per cache set flag to disable I/O request on this cache and all attached backing devices. Then new coming I/O requests can be rejected in *_make_request() before taking refcount, kernel threads and self-armed kernel worker can stop very fast when flags bit CACHE_SET_IO_DISABLE is set.
Because bcache also do internal I/Os for writeback, garbage collection, bucket allocation, journaling, this kind of I/O should be disabled after bch_cache_set_error() is called. So closure_bio_submit() is modified to check whether CACHE_SET_IO_DISABLE is set on cache_set->flags. If set, closure_bio_submit() will set bio->bi_status to BLK_STS_IOERR and return, generic_make_request() won't be called.
A sysfs interface is also added to set or clear CACHE_SET_IO_DISABLE bit from cache_set->flags, to disable or enable cache set I/O for debugging. It is helpful to trigger more corner case issues for failed cache device.
Changelog
v2, more changes by previous review,
- Use CACHE_SET_IO_DISABLE of cache_set->flags, suggested by Junhui.
- Check CACHE_SET_IO_DISABLE in bch_btree_gc() to stop a while-loop, this is reported and inspired from origal patch of Pavel Vazharov.
v1, initial version.
--- drivers/md/bcache/alloc.c | 3 ++- drivers/md/bcache/bcache.h | 18 ++++++++++++++++++ drivers/md/bcache/btree.c | 10 +++++++--- drivers/md/bcache/io.c | 2 +- drivers/md/bcache/journal.c | 4 ++-- drivers/md/bcache/request.c | 26 +++++++++++++++++++------- drivers/md/bcache/super.c | 6 +++++- drivers/md/bcache/sysfs.c | 20 ++++++++++++++++++++ drivers/md/bcache/util.h | 6 ------ drivers/md/bcache/writeback.c | 35 +++++++++++++++++++++++++++-------- 10 files changed, 101 insertions(+), 29 deletions(-)
drivers/md/bcache/alloc.c +2 -1
@@ -287,7 +287,8 @@ do { \break ; \ \mutex_unlock (&(ca)->set->bucket_lock); \ -if (kthread_should_stop ()) { \ +if (kthread_should_stop () || \ +test_bit (CACHE_SET_IO_DISABLE , &ca->set->flags)) { \set_current_state (TASK_RUNNING ); \return 0 ; \ } \
drivers/md/bcache/bcache.h +18 -0
@@ -475,10 +475,15 @@ struct gc_stat {* * CACHE_SET_RUNNING means all cache devices have been registered and journal * replay is complete. +* +* CACHE_SET_IO_DISABLE is set when bcache is stopping the whold cache set, all +* external and internal I/O should be denied when this flag is set. +* */ #define CACHE_SET_UNREGISTERING 0#define CACHE_SET_STOPPING 1#define CACHE_SET_RUNNING 2 +#define CACHE_SET_IO_DISABLE 4struct cache_set {struct closure cl;
@@ -862,6 +867,19 @@ static inline void wake_up_allocators(struct cache_set *c)wake_up_process (ca->alloc_thread); } +static inline void closure_bio_submit (struct cache_set *c, +struct bio *bio, +struct closure *cl) +{ +closure_get (cl); +if (unlikely (test_bit (CACHE_SET_IO_DISABLE , &c->flags))) { + bio->bi_status =BLK_STS_IOERR ; +bio_endio (bio); +return ; + } +generic_make_request (bio); +} +/* Forward declarations */ void bch_count_io_errors (struct cache *,blk_status_t ,int ,const char *);
drivers/md/bcache/btree.c +7 -3
@@ -1744,6 +1744,7 @@ static void bch_btree_gc(struct cache_set *c)btree_gc_start (c); +/* if CACHE_SET_IO_DISABLE set, gc thread should stop too */ do { ret = btree_root(gc_root, c, &op, &writes, &stats); closure_sync(&writes);
@@ -1751,7 +1752,7 @@ static void bch_btree_gc(struct cache_set *c)if (ret && ret != -EAGAIN )pr_warn ("gc failed!" ); - }while (ret); + }while (ret && !test_bit (CACHE_SET_IO_DISABLE , &c->flags));bch_btree_gc_finish (c);wake_up_allocators (c);
@@ -1789,9 +1790,12 @@ static int bch_gc_thread(void *arg)while (1 ) {wait_event_interruptible (c->gc_wait, -kthread_should_stop () ||gc_should_run (c)); +kthread_should_stop () || +test_bit (CACHE_SET_IO_DISABLE , &c->flags) || +gc_should_run (c)); -if (kthread_should_stop ()) +if (kthread_should_stop () || +test_bit (CACHE_SET_IO_DISABLE , &c->flags))break ;set_gc_sectors (c);
drivers/md/bcache/io.c +1 -1
@@ -38,7 +38,7 @@ void __bch_submit_bbio(struct bio *bio, struct cache_set *c)bio_set_dev (bio,PTR_CACHE (c, &b->key,0 )->bdev); b->submit_time_us =local_clock_us (); -closure_bio_submit (bio, bio->bi_private); +closure_bio_submit (c, bio, bio->bi_private); } void bch_submit_bbio(struct bio *bio, struct cache_set *c,
drivers/md/bcache/journal.c +2 -2
@@ -62,7 +62,7 @@ reread: left = ca->sb.bucket_size - offset;bio_set_op_attrs (bio,REQ_OP_READ ,0 );bch_bio_map (bio, data); -closure_bio_submit (bio, &cl); +closure_bio_submit (ca->set, bio, &cl);closure_sync (&cl);/* This function could be simpler now since we no longer write
@@ -653,7 +653,7 @@ static void journal_write_unlocked(struct closure *cl)spin_unlock (&c->journal.lock);while ((bio =bio_list_pop (&list))) -closure_bio_submit (bio, cl); +closure_bio_submit (c, bio, cl);continue_at (cl, journal_write_done,NULL ); }
drivers/md/bcache/request.c +19 -7
@@ -747,7 +747,7 @@ static void cached_dev_read_error(struct closure *cl)/* XXX: invalidate cache */ -closure_bio_submit (bio, cl); +closure_bio_submit (s->iop.c, bio, cl); }continue_at (cl, cached_dev_cache_miss_done,NULL );
@@ -872,7 +872,7 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s, s->cache_miss = miss; s->iop.bio = cache_bio;bio_get (cache_bio); -closure_bio_submit (cache_bio, &s->cl); +closure_bio_submit (s->iop.c, cache_bio, &s->cl);return ret; out_put:
@@ -880,7 +880,7 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s, out_submit: miss->bi_end_io = request_endio; miss->bi_private = &s->cl; -closure_bio_submit (miss, &s->cl); +closure_bio_submit (s->iop.c, miss, &s->cl);return ret; }
@@ -945,7 +945,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s)if ((bio_op (bio) !=REQ_OP_DISCARD ) ||blk_queue_discard (bdev_get_queue (dc->bdev))) -closure_bio_submit (bio, cl); +closure_bio_submit (s->iop.c, bio, cl); }else if (s->iop.writeback) {bch_writeback_add (dc); s->iop.bio = bio;
@@ -960,12 +960,12 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) flush->bi_private = cl; flush->bi_opf =REQ_OP_WRITE |REQ_PREFLUSH ; -closure_bio_submit (flush, cl); +closure_bio_submit (s->iop.c, flush, cl); } }else { s->iop.bio =bio_clone_fast (bio,GFP_NOIO , dc->disk.bio_split); -closure_bio_submit (bio, cl); +closure_bio_submit (s->iop.c, bio, cl); }closure_call (&s->iop.cl, bch_data_insert,NULL , cl);
@@ -981,7 +981,7 @@ static void cached_dev_nodata(struct closure *cl)bch_journal_meta (s->iop.c, cl);/* If it's a flush, we send the flush to the backing device too */ -closure_bio_submit (bio, cl); +closure_bio_submit (s->iop.c, bio, cl);continue_at (cl, cached_dev_bio_complete,NULL ); }
@@ -996,6 +996,12 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q, struct cached_dev *dc = container_of(d, struct cached_dev, disk);int rw =bio_data_dir (bio); +if (unlikely (d->c &&test_bit (CACHE_SET_IO_DISABLE , &d->c->flags))) { + bio->bi_status =BLK_STS_IOERR ; +bio_endio (bio); +return BLK_QC_T_NONE ; + } +atomic_set (&dc->backing_idle,0 );generic_start_io_acct (q, rw,bio_sectors (bio), &d->disk->part0);
@@ -1112,6 +1118,12 @@ static blk_qc_t flash_dev_make_request(struct request_queue *q,struct bcache_device *d = bio->bi_disk->private_data;int rw =bio_data_dir (bio); +if (unlikely (d->c &&test_bit (CACHE_SET_IO_DISABLE , &d->c->flags))) { + bio->bi_status =BLK_STS_IOERR ; +bio_endio (bio); +return BLK_QC_T_NONE ; + } +generic_start_io_acct (q, rw,bio_sectors (bio), &d->disk->part0); s =search_alloc (bio, d);
drivers/md/bcache/super.c +5 -1
@@ -521,7 +521,7 @@ static void prio_io(struct cache *ca, uint64_t bucket, int op,bio_set_op_attrs (bio, op,REQ_SYNC |REQ_META |op_flags);bch_bio_map (bio, ca->disk_buckets); -closure_bio_submit (bio, &ca->prio); +closure_bio_submit (ca->set, bio, &ca->prio);closure_sync (cl); }
@@ -1349,6 +1349,9 @@ bool bch_cache_set_error(struct cache_set *c, const char *fmt, ...) test_bit(CACHE_SET_STOPPING, &c->flags))return false; +if (test_and_set_bit (CACHE_SET_IO_DISABLE , &c->flags)) +pr_warn ("bcache: CACHE_SET_IO_DISABLE already set" ); +/* XXX: we can be called from atomic context acquire_console_sem(); */
@@ -1584,6 +1587,7 @@ struct cache_set *bch_cache_set_alloc(struct cache_sb *sb) c->congested_read_threshold_us =2000 ; c->congested_write_threshold_us =20000 ; c->error_limit =DEFAULT_IO_ERROR_LIMIT ; +WARN_ON (test_and_clear_bit (CACHE_SET_IO_DISABLE , &c->flags));return c; err:
drivers/md/bcache/sysfs.c +20 -0
@@ -92,6 +92,7 @@ read_attribute(partial_stripes_expensive);rw_attribute (synchronous);rw_attribute (journal_delay_ms); +rw_attribute (io_disable);rw_attribute (discard);rw_attribute (running);rw_attribute (label);
@@ -577,6 +578,8 @@ SHOW(__bch_cache_set)sysfs_printf (gc_always_rewrite,"%i" , c->gc_always_rewrite);sysfs_printf (btree_shrinker_disabled,"%i" , c->shrinker_disabled);sysfs_printf (copy_gc_enabled,"%i" , c->copy_gc_enabled); +sysfs_printf (io_disable,"%i" , +test_bit (CACHE_SET_IO_DISABLE , &c->flags));if (attr == &sysfs_bset_tree_stats)return bch_bset_print_stats (c, buf);
@@ -666,6 +669,22 @@ STORE(__bch_cache_set)if (attr == &sysfs_io_error_halflife) c->error_decay =strtoul_or_return (buf) /88 ; +if (attr == &sysfs_io_disable) { +int v =strtoul_or_return (buf); + +if (v) { +if (test_and_set_bit (CACHE_SET_IO_DISABLE , + &c->flags)) +pr_warn ("bcache: CACHE_SET_IO_DISABLE" +" already set" ); + }else { +if (!test_and_clear_bit (CACHE_SET_IO_DISABLE , + &c->flags)) +pr_warn ("bcache: CACHE_SET_IO_DISABLE" +" already cleared" ); + } + } +sysfs_strtoul (journal_delay_ms, c->journal_delay_ms);sysfs_strtoul (verify, c->verify);sysfs_strtoul (key_merging_disabled, c->key_merging_disabled);
@@ -748,6 +767,7 @@ static struct attribute *bch_cache_set_internal_files[] = { &sysfs_gc_always_rewrite, &sysfs_btree_shrinker_disabled, &sysfs_copy_gc_enabled, + &sysfs_io_disable, NULL };KTYPE (bch_cache_set_internal);
drivers/md/bcache/util.h +0 -6
@@ -565,12 +565,6 @@ static inline sector_t bdev_sectors(struct block_device *bdev)return bdev->bd_inode->i_size >>9 ; } -#define closure_bio_submit (bio, cl) \ -do { \ - closure_get(cl); \ - generic_make_request(bio); \ -} while (0) -uint64_t bch_crc64_update (uint64_t ,const void *,size_t );uint64_t bch_crc64 (const void *,size_t );
drivers/md/bcache/writeback.c +27 -8
@@ -114,6 +114,7 @@ static void update_writeback_rate(struct work_struct *work)struct cached_dev *dc =container_of (to_delayed_work (work), struct cached_dev, writeback_rate_update); +struct cache_set *c = dc->disk.c;/* * should check BCACHE_DEV_RATE_DW_RUNNING before calling
@@ -123,7 +124,12 @@ static void update_writeback_rate(struct work_struct *work)/* paired with where BCACHE_DEV_RATE_DW_RUNNING is tested */ smp_mb (); -if (!test_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) { +/* +* CACHE_SET_IO_DISABLE might be set via sysfs interface, +* check it here too. +*/ +if (!test_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags) || +test_bit (CACHE_SET_IO_DISABLE , &c->flags)) {clear_bit (BCACHE_DEV_RATE_DW_RUNNING , &dc->disk.flags);/* paired with where BCACHE_DEV_RATE_DW_RUNNING is tested */ smp_mb ();
@@ -138,7 +144,12 @@ static void update_writeback_rate(struct work_struct *work)up_read (&dc->writeback_lock); -if (test_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags)) { +/* +* CACHE_SET_IO_DISABLE might be set via sysfs interface, +* check it here too. +*/ +if (test_bit (BCACHE_DEV_WB_RUNNING , &dc->disk.flags) && + !test_bit (CACHE_SET_IO_DISABLE , &c->flags)) {schedule_delayed_work (&dc->writeback_rate_update, dc->writeback_rate_update_seconds *HZ ); }
@@ -278,7 +289,7 @@ static void write_dirty(struct closure *cl)bio_set_dev (&io->bio, io->dc->bdev); io->bio.bi_end_io = dirty_endio; -closure_bio_submit (&io->bio, cl); +closure_bio_submit (io->dc->disk.c, &io->bio, cl); }atomic_set (&dc->writeback_sequence_next, next_sequence);
@@ -304,7 +315,7 @@ static void read_dirty_submit(struct closure *cl) { struct dirty_io *io = container_of(cl, struct dirty_io, cl); -closure_bio_submit (&io->bio, cl); +closure_bio_submit (io->dc->disk.c, &io->bio, cl);continue_at (cl, write_dirty, io->dc->writeback_write_wq); }
@@ -330,7 +341,9 @@ static void read_dirty(struct cached_dev *dc) next =bch_keybuf_next (&dc->writeback_keys); -while (!kthread_should_stop () && next) { +while (!kthread_should_stop () && + !test_bit (CACHE_SET_IO_DISABLE , &dc->disk.c->flags) && + next) { size =0 ; nk =0 ;
@@ -427,7 +440,9 @@ static void read_dirty(struct cached_dev *dc) } } -while (!kthread_should_stop () && delay) { +while (!kthread_should_stop () && + !test_bit (CACHE_SET_IO_DISABLE , &dc->disk.c->flags) && + delay) {schedule_timeout_interruptible (delay); delay =writeback_delay (dc,0 ); }
@@ -583,11 +598,13 @@ static bool refill_dirty(struct cached_dev *dc)static int bch_writeback_thread (void *arg) {struct cached_dev *dc = arg; +struct cache_set *c = dc->disk.c;bool searched_full_index;bch_ratelimit_reset (&dc->writeback_rate); -while (!kthread_should_stop ()) { +while (!kthread_should_stop () && + !test_bit (CACHE_SET_IO_DISABLE , &c->flags)) {down_write (&dc->writeback_lock);set_current_state (TASK_INTERRUPTIBLE );/*
@@ -601,7 +618,8 @@ static int bch_writeback_thread(void *arg) (!atomic_read(&dc->has_dirty) || !dc->writeback_running)) {up_write (&dc->writeback_lock); -if (kthread_should_stop ()) { +if (kthread_should_stop () || +test_bit (CACHE_SET_IO_DISABLE , &c->flags)) {set_current_state (TASK_RUNNING );break ; }
@@ -637,6 +655,7 @@ static int bch_writeback_thread(void *arg)while (delay && !kthread_should_stop () && + !test_bit (CACHE_SET_IO_DISABLE , &c->flags) && !test_bit (BCACHE_DEV_DETACHING , &dc->disk.flags)) delay =schedule_timeout_interruptible (delay);
Signature · 1 line
2.15.1
PATCH v4 8/13 bcache: stop all attached bcache devices for a retired cache set
In reply to Coly LiWhen there are too many I/O errors on cache device, current bcache code will retire the whole cache set, and detach all bcache devices. But the detached bcache devices are not stopped, which is problematic when bcache is in writeback mode.
If the retired cache set has dirty data of backing devices, continue writing to bcache device will write to backing device directly. If the LBA of write request has a dirty version cached on cache device, next time when the cache device is re-registered and backing device re-attached to it again, the stale dirty data on cache device will be written to backing device, and overwrite latest directly written data. This situation causes a quite data corruption.
This patch checkes whether cache_set->io_disable is true in __cache_set_unregister(). If cache_set->io_disable is true, it means cache set is unregistering by too many I/O errors, then all attached bcache devices will be stopped as well. If cache_set->io_disable is not true, it means __cache_set_unregister() is triggered by writing 1 to sysfs file /sys/fs/bcache/<UUID>/bcache/stop. This is an exception because users do it explicitly, this patch keeps existing behavior and does not stop any bcache device.
Even the failed cache device has no dirty data, stopping bcache device is still a desired behavior by many Ceph and data base users. Then their application will report I/O errors due to disappeared bcache device, and operation people will know the cache device is broken or disconnected.
Changelog:
v2: add reviewed-by from Hannes.
v1: initial version for review.
--- drivers/md/bcache/super.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/md/bcache/super.c +8 -0
@@ -1478,6 +1478,14 @@ static void __cache_set_unregister(struct closure *cl) dc =container_of (c->devices[i], struct cached_dev, disk);bch_cached_dev_detach (dc); +/* +* If we come here by too many I/O errors, +* bcache device should be stopped too, to +* keep data consistency on cache and +* backing devices. +*/ +if (test_bit (CACHE_SET_IO_DISABLE , &c->flags)) +bcache_device_stop (c->devices[i]); }else {bcache_device_stop (c->devices[i]); }
Signature · 1 line
2.15.1
PATCH v4 9/13 bcache: fix inaccurate io state for detached bcache devices
In reply to Coly LiFrom: Tang Junhui <[email protected]>
When we run IO in a detached device, and run iostat to shows IO status, normally it will show like bellow (Omitted some fields):
Device: ... avgrq-sz avgqu-sz await r_await w_await svctm %util sdd ... 15.89 0.53 1.82 0.20 2.23 1.81 52.30 bcache0 ... 15.89 115.42 0.00 0.00 0.00 2.40 69.60
but after IO stopped, there are still very big avgqu-sz and %util values as bellow:
Device: ... avgrq-sz avgqu-sz await r_await w_await svctm %util
bcache0 ... 0 5326.32 0.00 0.00 0.00 0.00 100.10
The reason for this issue is that, only generic_start_io_acct() called and no generic_end_io_acct() called for detached device in cached_dev_make_request(). See the code:
//start generic_start_io_acct()generic_start_io_acct(q, rw, bio_sectors(bio), &d->disk->part0);
if (cached_dev_get(dc)) {
//will callback generic_end_io_acct()
}
else {
//will not call generic_end_io_acct()
}This patch calls generic_end_io_acct() in the end of IO for detached devices, so we can show IO state correctly.
(Modified to use GFP_NOIO in kzalloc() by Coly Li)
--- drivers/md/bcache/request.c | 58 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 51 insertions(+), 7 deletions(-)
drivers/md/bcache/request.c +51 -7
@@ -986,6 +986,55 @@ static void cached_dev_nodata(struct closure *cl)continue_at (cl, cached_dev_bio_complete,NULL ); } +struct detached_dev_io_private { +struct bcache_device *d; +unsigned long start_time; +bio_end_io_t *bi_end_io; +void *bi_private; +}; + +static void detatched_dev_end_io (struct bio *bio) +{ +struct detached_dev_io_private *ddip; + + ddip = bio->bi_private; + bio->bi_end_io = ddip->bi_end_io; + bio->bi_private = ddip->bi_private; + +generic_end_io_acct (ddip->d->disk->queue, +bio_data_dir (bio), + &ddip->d->disk->part0, ddip->start_time); + +kfree (ddip); + + bio->bi_end_io (bio); +} + +static void detached_dev_do_request (struct bcache_device *d,struct bio *bio) +{ +struct detached_dev_io_private *ddip; + struct cached_dev *dc = container_of(d, struct cached_dev, disk); + +/* +* no need to call closure_get(&dc->disk.cl), +* because upper layer had already opened bcache device, +* which would call closure_get(&dc->disk.cl) +*/ + ddip =kzalloc (sizeof (struct detached_dev_io_private ),GFP_NOIO ); + ddip->d = d; + ddip->start_time = jiffies; + ddip->bi_end_io = bio->bi_end_io; + ddip->bi_private = bio->bi_private; + bio->bi_end_io = detatched_dev_end_io; + bio->bi_private = ddip; + +if ((bio_op (bio) ==REQ_OP_DISCARD ) && + !blk_queue_discard (bdev_get_queue (dc->bdev))) + bio->bi_end_io (bio); +else +generic_make_request (bio); +} +/* Cached devices - read & write stuff */ static blk_qc_t cached_dev_make_request(struct request_queue *q,
@@ -1028,13 +1077,8 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q,else cached_dev_read (dc, s); } - }else { -if ((bio_op (bio) ==REQ_OP_DISCARD ) && - !blk_queue_discard (bdev_get_queue (dc->bdev))) -bio_endio (bio); -else -generic_make_request (bio); - } + }else +detached_dev_do_request (d, bio);return BLK_QC_T_NONE ; }
Signature · 1 line
2.15.1
PATCH v4 10/13 bcache: add backing_request_endio() for bi_end_io of attached backing device I/O
In reply to Coly LiIn order to catch I/O error of backing device, a separate bi_end_io call back is required. Then a per backing device counter can record I/O errors number and retire the backing device if the counter reaches a per backing device I/O error limit.
This patch adds backing_request_endio() to bcache backing device I/O code path, this is a preparation for further complicated backing device failure handling. So far there is no real code logic change, I make this change a separate patch to make sure it is stable and reliable for further work.
Changelog:
v2: indeed this is new added in this patch set.
--- drivers/md/bcache/request.c | 95 +++++++++++++++++++++++++++++++++++-------- drivers/md/bcache/super.c | 1 + drivers/md/bcache/writeback.c | 1 + 3 files changed, 81 insertions(+), 16 deletions(-)
drivers/md/bcache/request.c +79 -16
@@ -139,6 +139,7 @@ static void bch_data_invalidate(struct closure *cl) } op->insert_data_done = true; +/* get in bch_data_insert() */ bio_put (bio); out:continue_at (cl, bch_data_insert_keys, op->wq);
@@ -630,6 +631,38 @@ static void request_endio(struct bio *bio)closure_put (cl); } +static void backing_request_endio (struct bio *bio) +{ +struct closure *cl = bio->bi_private; + +if (bio->bi_status) { + struct search *s = container_of(cl, struct search, cl); +/* +* If a bio has REQ_PREFLUSH for writeback mode, it is +* speically assembled in cached_dev_write() for a non-zero +* write request which has REQ_PREFLUSH. we don't set +* s->iop.status by this failure, the status will be decided +* by result of bch_data_insert() operation. +*/ +if (unlikely (s->iop.writeback && + bio->bi_opf &REQ_PREFLUSH )) { +char buf[BDEVNAME_SIZE ]; + +bio_devname (bio, buf); +pr_err ("Can't flush %s: returned bi_status %i" , + buf, bio->bi_status); + }else { +/* set to orig_bio->bi_status in bio_complete() */ + s->iop.status = bio->bi_status; + } + s->recoverable = false; +/* should count I/O error for backing device here */ + } + +bio_put (bio); +closure_put (cl); +} +static void bio_complete (struct search *s) {if (s->orig_bio) {
@@ -644,13 +677,21 @@ static void bio_complete(struct search *s) } } -static void do_bio_hook (struct search *s,struct bio *orig_bio) +static void do_bio_hook (struct search *s, +struct bio *orig_bio, +bio_end_io_t *end_io_fn) {struct bio *bio = &s->bio.bio;bio_init (bio,NULL ,0 );__bio_clone_fast (bio, orig_bio); - bio->bi_end_io = request_endio; +/* +* bi_end_io can be set separately somewhere else, e.g. the +* variants in, +* - cache_bio->bi_end_io from cached_dev_cache_miss() +* - n->bi_end_io from cache_lookup_fn() +*/ + bio->bi_end_io = end_io_fn; bio->bi_private = &s->cl;bio_cnt_set (bio,3 );
@@ -676,7 +717,7 @@ static inline struct search *search_alloc(struct bio *bio, s =mempool_alloc (d->c->search,GFP_NOIO );closure_init (&s->cl,NULL ); -do_bio_hook (s, bio); +do_bio_hook (s, bio, request_endio); s->orig_bio = bio; s->cache_miss =NULL ;
@@ -743,10 +784,11 @@ static void cached_dev_read_error(struct closure *cl)trace_bcache_read_retry (s->orig_bio); s->iop.status =0 ; -do_bio_hook (s, s->orig_bio); +do_bio_hook (s, s->orig_bio, backing_request_endio);/* XXX: invalidate cache */ +/* I/O request sent to backing device */ closure_bio_submit (s->iop.c, bio, cl); }
@@ -859,7 +901,7 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,bio_copy_dev (cache_bio, miss); cache_bio->bi_iter.bi_size = s->insert_bio_sectors <<9 ; - cache_bio->bi_end_io = request_endio; + cache_bio->bi_end_io = backing_request_endio; cache_bio->bi_private = &s->cl;bch_bio_map (cache_bio,NULL );
@@ -872,14 +914,16 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s, s->cache_miss = miss; s->iop.bio = cache_bio;bio_get (cache_bio); +/* I/O request sent to backing device */ closure_bio_submit (s->iop.c, cache_bio, &s->cl);return ret; out_put:bio_put (cache_bio); out_submit: - miss->bi_end_io = request_endio; + miss->bi_end_io = backing_request_endio; miss->bi_private = &s->cl; +/* I/O request sent to backing device */ closure_bio_submit (s->iop.c, miss, &s->cl);return ret; }
@@ -943,31 +987,48 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) s->iop.bio = s->orig_bio;bio_get (s->iop.bio); -if ((bio_op (bio) !=REQ_OP_DISCARD ) || -blk_queue_discard (bdev_get_queue (dc->bdev))) -closure_bio_submit (s->iop.c, bio, cl); +if (bio_op (bio) ==REQ_OP_DISCARD && + !blk_queue_discard (bdev_get_queue (dc->bdev))) + goto insert_data; + +/* I/O request sent to backing device */ + bio->bi_end_io = backing_request_endio; +closure_bio_submit (s->iop.c, bio, cl); + }else if (s->iop.writeback) {bch_writeback_add (dc); s->iop.bio = bio;if (bio->bi_opf &REQ_PREFLUSH ) { -/* Also need to send a flush to the backing device */ -struct bio *flush =bio_alloc_bioset (GFP_NOIO ,0 , - dc->disk.bio_split); - +/* +* Also need to send a flush to the backing +* device, if failed on backing device. +*/ +struct bio *flush; + + flush =bio_alloc_bioset (GFP_NOIO ,0 , + dc->disk.bio_split); +if (!flush) { + s->iop.status =BLK_STS_RESOURCE ; + goto insert_data; + }bio_copy_dev (flush, bio); - flush->bi_end_io = request_endio; + flush->bi_end_io = backing_request_endio; flush->bi_private = cl; flush->bi_opf =REQ_OP_WRITE |REQ_PREFLUSH ; - +/* I/O request sent to backing device */ closure_bio_submit (s->iop.c, flush, cl); } +bch_writeback_add (dc); + }else { s->iop.bio =bio_clone_fast (bio,GFP_NOIO , dc->disk.bio_split); - +/* I/O request sent to backing device */ + bio->bi_end_io = backing_request_endio;closure_bio_submit (s->iop.c, bio, cl); } +insert_data:closure_call (&s->iop.cl, bch_data_insert,NULL , cl);continue_at (cl, cached_dev_write_complete,NULL ); }
@@ -981,6 +1042,7 @@ static void cached_dev_nodata(struct closure *cl)bch_journal_meta (s->iop.c, cl);/* If it's a flush, we send the flush to the backing device too */ + bio->bi_end_io = backing_request_endio;closure_bio_submit (s->iop.c, bio, cl);continue_at (cl, cached_dev_bio_complete,NULL );
@@ -1078,6 +1140,7 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q,cached_dev_read (dc, s); } }else +/* I/O request sent to backing device */ detached_dev_do_request (d, bio);return BLK_QC_T_NONE ;
drivers/md/bcache/super.c +1 -0
@@ -265,6 +265,7 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent) bio->bi_private = dc;closure_get (cl); +/* I/O request sent to backing device */ __write_super (&dc->sb, bio);closure_return_with_destructor (cl, bch_write_bdev_super_unlock);
drivers/md/bcache/writeback.c +1 -0
@@ -289,6 +289,7 @@ static void write_dirty(struct closure *cl)bio_set_dev (&io->bio, io->dc->bdev); io->bio.bi_end_io = dirty_endio; +/* I/O request sent to backing device */ closure_bio_submit (io->dc->disk.c, &io->bio, cl); }
Signature · 1 line
2.15.1
PATCH v4 11/13 bcache: add io_disable to struct cached_dev
In reply to Coly LiIf a bcache device is configured to writeback mode, current code does not handle write I/O errors on backing devices properly.
In writeback mode, write request is written to cache device, and latter being flushed to backing device. If I/O failed when writing from cache device to the backing device, bcache code just ignores the error and upper layer code is NOT noticed that the backing device is broken.
This patch tries to handle backing device failure like how the cache device failure is handled,
- Add a error counter 'io_errors' and error limit 'error_limit' in struct cached_dev. Add another io_disable to struct cached_dev to disable I/Os on the problematic backing device.
- When I/O error happens on backing device, increase io_errors counter. And if io_errors reaches error_limit, set cache_dev->io_disable to true, and stop the bcache device.
The result is, if backing device is broken of disconnected, and I/O errors reach its error limit, backing device will be disabled and the associated bcache device will be removed from system.
Changelog:
v2: indeed this is new added in v2 patch set.
--- drivers/md/bcache/bcache.h | 7 +++++++ drivers/md/bcache/io.c | 14 ++++++++++++++ drivers/md/bcache/request.c | 14 ++++++++++++-- drivers/md/bcache/super.c | 22 ++++++++++++++++++++++ drivers/md/bcache/sysfs.c | 15 ++++++++++++++- 5 files changed, 69 insertions(+), 3 deletions(-)
drivers/md/bcache/bcache.h +7 -0
@@ -360,6 +360,7 @@ struct cached_dev {unsigned sequential_cutoff;unsigned readahead; + unsigned io_disable:1; unsigned verify:1; unsigned bypass_torture_test:1;
@@ -379,6 +380,10 @@ struct cached_dev {unsigned writeback_rate_i_term_inverse;unsigned writeback_rate_p_term_inverse;unsigned writeback_rate_minimum; + +#define DEFAULT_CACHED_DEV_ERROR_LIMIT 64 +atomic_t io_errors; +unsigned error_limit; };enum alloc_reserve {
@@ -882,6 +887,7 @@ static inline void closure_bio_submit(struct cache_set *c,/* Forward declarations */ +void bch_count_backing_io_errors (struct cached_dev *dc,struct bio *bio);void bch_count_io_errors (struct cache *,blk_status_t ,int ,const char *);void bch_bbio_count_io_errors (struct cache_set *,struct bio *,blk_status_t ,const char *);
@@ -909,6 +915,7 @@ int bch_bucket_alloc_set(struct cache_set *, unsigned, struct bkey *, int, bool);bool bch_alloc_sectors (struct cache_set *,struct bkey *,unsigned ,unsigned ,unsigned ,bool ); +bool bch_cached_dev_error (struct cached_dev *dc); __printf(2, 3)bool bch_cache_set_error (struct cache_set *,const char *, ...);
drivers/md/bcache/io.c +14 -0
@@ -50,6 +50,20 @@ void bch_submit_bbio(struct bio *bio, struct cache_set *c, }/* IO errors */ +void bch_count_backing_io_errors (struct cached_dev *dc,struct bio *bio) +{ +char buf[BDEVNAME_SIZE ]; +unsigned errors; + +WARN_ONCE (!dc,"NULL pointer of struct cached_dev" ); + + errors =atomic_add_return (1 , &dc->io_errors); +if (errors < dc->error_limit) +pr_err ("%s: IO error on backing device, unrecoverable" , +bio_devname (bio, buf)); +else +bch_cached_dev_error (dc); +} void bch_count_io_errors(struct cache *ca, blk_status_t error,
drivers/md/bcache/request.c +12 -2
@@ -637,6 +637,8 @@ static void backing_request_endio(struct bio *bio)if (bio->bi_status) { struct search *s = container_of(cl, struct search, cl); +struct cached_dev *dc =container_of (s->d, + struct cached_dev, disk);/* * If a bio has REQ_PREFLUSH for writeback mode, it is * speically assembled in cached_dev_write() for a non-zero
@@ -657,6 +659,7 @@ static void backing_request_endio(struct bio *bio) } s->recoverable = false;/* should count I/O error for backing device here */ +bch_count_backing_io_errors (dc, bio); }bio_put (bio);
@@ -1067,8 +1070,14 @@ static void detatched_dev_end_io(struct bio *bio)bio_data_dir (bio), &ddip->d->disk->part0, ddip->start_time); -kfree (ddip); +if (bio->bi_status) { +struct cached_dev *dc =container_of (ddip->d, + struct cached_dev, disk); +/* should count I/O error for backing device here */ +bch_count_backing_io_errors (dc, bio); + } +kfree (ddip); bio->bi_end_io (bio); }
@@ -1107,7 +1116,8 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q, struct cached_dev *dc = container_of(d, struct cached_dev, disk);int rw =bio_data_dir (bio); -if (unlikely (d->c &&test_bit (CACHE_SET_IO_DISABLE , &d->c->flags))) { +if (unlikely ((d->c &&test_bit (CACHE_SET_IO_DISABLE , &d->c->flags)) || + dc->io_disable)) { bio->bi_status =BLK_STS_IOERR ;bio_endio (bio);return BLK_QC_T_NONE ;
drivers/md/bcache/super.c +22 -0
@@ -1188,6 +1188,10 @@ static int cached_dev_init(struct cached_dev *dc, unsigned block_size)max (dc->disk.disk->queue->backing_dev_info->ra_pages, q->backing_dev_info->ra_pages); +atomic_set (&dc->io_errors,0 ); + dc->io_disable = false; + dc->error_limit =DEFAULT_CACHED_DEV_ERROR_LIMIT ; +bch_cached_dev_request_init (dc);bch_cached_dev_writeback_init (dc);return 0 ;
@@ -1339,6 +1343,24 @@ int bch_flash_dev_create(struct cache_set *c, uint64_t size)return flash_dev_run (c, u); } +bool bch_cached_dev_error (struct cached_dev *dc) +{ +char name[BDEVNAME_SIZE ]; + +if (!dc ||test_bit (BCACHE_DEV_CLOSING , &dc->disk.flags)) +return false; + + dc->io_disable = true; +/* make others know io_disable is true earlier */ +smp_mb (); + +pr_err ("bcache: stop %s: too many IO errors on backing device %s\n" , + dc->disk.name,bdevname (dc->bdev, name)); + +bcache_device_stop (&dc->disk); +return true; +} +/* Cache set */ __printf (2 ,3 )
drivers/md/bcache/sysfs.c +14 -1
@@ -131,7 +131,9 @@ SHOW(__bch_cached_dev)var_print (writeback_delay);var_print (writeback_percent);sysfs_hprint (writeback_rate, dc->writeback_rate.rate <<9 ); - +sysfs_hprint (io_errors,atomic_read (&dc->io_errors)); +sysfs_printf (io_error_limit,"%i" , dc->error_limit); +sysfs_printf (io_disable,"%i" , dc->io_disable);var_print (writeback_rate_update_seconds);var_print (writeback_rate_i_term_inverse);var_print (writeback_rate_p_term_inverse);
@@ -223,6 +225,14 @@ STORE(__cached_dev)d_strtoul (writeback_rate_i_term_inverse);d_strtoul_nonzero (writeback_rate_p_term_inverse); +sysfs_strtoul_clamp (io_error_limit, dc->error_limit,0 ,INT_MAX ); + +if (attr == &sysfs_io_disable) { +int v =strtoul_or_return (buf); + + dc->io_disable = v ?1 :0 ; + } +d_strtoi_h (sequential_cutoff);d_strtoi_h (readahead);
@@ -330,6 +340,9 @@ static struct attribute *bch_cached_dev_files[] = { &sysfs_writeback_rate_i_term_inverse, &sysfs_writeback_rate_p_term_inverse, &sysfs_writeback_rate_debug, + &sysfs_errors, + &sysfs_io_error_limit, + &sysfs_io_disable, &sysfs_dirty_data, &sysfs_stripe_size, &sysfs_partial_stripes_expensive,
Signature · 1 line
2.15.1
PATCH v4 12/13 bcache: stop bcache device when backing device is offline
In reply to Coly LiCurrently bcache does not handle backing device failure, if backing device is offline and disconnected from system, its bcache device can still be accessible. If the bcache device is in writeback mode, I/O requests even can success if the requests hit on cache device. That is to say, when and how bcache handles offline backing device is undefined.
This patch tries to handle backing device offline in a rather simple way,
- Add cached_dev->status_update_thread kernel thread to update backing device status in every 1 second.
- Add cached_dev->offline_seconds to record how many seconds the backing device is observed to be offline. If the backing device is offline for BACKING_DEV_OFFLINE_TIMEOUT (30) seconds, set dc->io_disable to 1 and call bcache_device_stop() to stop the bache device which linked to the offline backing device.
Now if a backing device is offline for BACKING_DEV_OFFLINE_TIMEOUT seconds, its bcache device will be removed, then user space application writing on it will get error immediately, and handler the device failure in time.
This patch is quite simple, does not handle more complicated situations. Once the bcache device is stopped, users need to recovery the backing device, register and attach it manually.
--- drivers/md/bcache/bcache.h | 2 ++ drivers/md/bcache/super.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)
drivers/md/bcache/bcache.h +2 -0
@@ -338,6 +338,7 @@ struct cached_dev {struct keybuf writeback_keys; +struct task_struct *status_update_thread;/* * Order the write-half of writeback operations strongly in dispatch * order. (Maintain LBA order; don't allow reads completing out of
@@ -384,6 +385,7 @@ struct cached_dev {#define DEFAULT_CACHED_DEV_ERROR_LIMIT 64atomic_t io_errors;unsigned error_limit; +unsigned offline_seconds; };enum alloc_reserve {
drivers/md/bcache/super.c +55 -0
79 lines not on this page · raw message
Signature · 1 line
2.15.1
PATCH v4 13/13 bcache: add stop_when_cache_set_failed to struct cached_dev
In reply to Coly LiCurrent bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
This patch adds a per-cached_dev option stop_when_cache_set_failed, which is enabled (1) by default. Its value can be set via sysfs, when it is set to 0, the corresponding bcache device won't be stopped when a broken or disconnected cache set is retiring.
When the cached device has dirty data on retiring cache set, if bcache device is not stopped, following I/O request on the bcache device may result data corruption on backing device. This patch also prints out warn- ing information in kernel message.
--- drivers/md/bcache/bcache.h | 1 + drivers/md/bcache/super.c | 63 +++++++++++++++++++++++++++++++++------------- drivers/md/bcache/sysfs.c | 10 ++++++++ 3 files changed, 56 insertions(+), 18 deletions(-)
drivers/md/bcache/bcache.h +1 -0
@@ -362,6 +362,7 @@ struct cached_dev {unsigned readahead; unsigned io_disable:1; + unsigned stop_when_cache_set_failed:1; unsigned verify:1; unsigned bypass_torture_test:1;
drivers/md/bcache/super.c +45 -18
84 lines not on this page · raw message
drivers/md/bcache/sysfs.c +10 -0
@@ -93,6 +93,7 @@ read_attribute(partial_stripes_expensive);rw_attribute (synchronous);rw_attribute (journal_delay_ms);rw_attribute (io_disable); +rw_attribute (stop_when_cache_set_failed);rw_attribute (discard);rw_attribute (running);rw_attribute (label);
@@ -134,6 +135,8 @@ SHOW(__bch_cached_dev)sysfs_hprint (io_errors,atomic_read (&dc->io_errors));sysfs_printf (io_error_limit,"%i" , dc->error_limit);sysfs_printf (io_disable,"%i" , dc->io_disable); +sysfs_printf (stop_when_cache_set_failed,"%i" , + dc->stop_when_cache_set_failed);var_print (writeback_rate_update_seconds);var_print (writeback_rate_i_term_inverse);var_print (writeback_rate_p_term_inverse);
@@ -233,6 +236,12 @@ STORE(__cached_dev) dc->io_disable = v ?1 :0 ; } +if (attr == &sysfs_stop_when_cache_set_failed) { +int v =strtoul_or_return (buf); + + dc->stop_when_cache_set_failed = v ?1 :0 ; + } +d_strtoi_h (sequential_cutoff);d_strtoi_h (readahead);
@@ -343,6 +352,7 @@ static struct attribute *bch_cached_dev_files[] = { &sysfs_errors, &sysfs_io_error_limit, &sysfs_io_disable, + &sysfs_stop_when_cache_set_failed, &sysfs_dirty_data, &sysfs_stripe_size, &sysfs_partial_stripes_expensive,
Signature · 1 line
2.15.1
Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev
Hello Coly,
Quoted 5 lines · you
Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
In the current state, this functionality is rather user-unfriendly.
1. The new "stop_when_cache_set_failed" option doesn't seem to be persistent. (At least, I did not see any explicit mechanism of saving/restoring it.) That is, users will have to set it on each system startup.
2. If the new option is set to zero, it will (supposedly) lead to data corruption/loss when cache set of a "dirty" bcache device is detached. The option that an average home user may want to switch shouldn't be a way to shoot oneself in the foot!
As a remedy, the option could be changed to have the states "always" and "auto" instead of "1" and "0", so that "auto" would still stop the bcache device if it (or the entire cache set) is "dirty". (Alternatively, it could be renamed to "force_stop_when_cache_set_failed" or "always_stop_when_cache_set_failed" with values "1" and "0", if string values are undesirable.)
Also, the printed warning is somewhat misleading: it says "To disable this warning message, please set /sys/block/%s/bcache/stop_when_cache_set_failed to 1", whereas the suggested change would lead to behaviour change rather that to just disabling the warning.
3. If (2) is implemented, the default value for the option could be changed to "auto". (The rationale is that enterprise users who want it enabled are better prepared to tune their device settings on startup.) However, this is only important if (1) is not implemented.
Quoted 39 lines
This patch adds a per-cached_dev option
stop_when_cache_set_failed, which is enabled (1) by default. Its value can be set via sysfs, when it is set to 0, the corresponding bcache device won't be stopped when a broken or disconnected cache set is retiring.When the cached device has dirty data on retiring cache set, if bcache device is not stopped, following I/O request on the bcache device may result data corruption on backing device. This patch also prints out warn- ing information in kernel message.
Signed-off-by Coly Li <[email protected]>Cc Nix <[email protected]>Cc Michael Lyle <[email protected]>Cc Junhui Tang <[email protected]>Cc Hannes Reinecke <[email protected]>--- drivers/md/bcache/bcache.h | 1 + drivers/md/bcache/super.c | 63 +++++++++++++++++++++++++++++++++------------- drivers/md/bcache/sysfs.c | 10 ++++++++ 3 files changed, 56 insertions(+), 18 deletions(-)drivers/md/bcache/bcache.h +1
-0@@ -362,6 +362,7 @@ struct cached_dev {unsigned readahead; unsigned io_disable:1; + unsigned stop_when_cache_set_failed:1; unsigned verify:1; unsigned bypass_torture_test:1;drivers/md/bcache/super.c +0
-0
drivers/md/bcache/super.c static int cached_dev_init(struct cached_dev *dc, unsigned block_size) · line 1248
Quoted 3 lines
drivers/md/bcache/super.c +0
-0@@ -1246,6 +1246,7 @@ static int cached_dev_init(struct cached_dev *dc, unsigned block_size)atomic_set (&dc->io_errors,0 ); dc->io_disable = false; dc->error_limit =DEFAULT_CACHED_DEV_ERROR_LIMIT ;
+ dc->>stop_when_cache_set_failed = 1;
drivers/md/bcache/super.c static void cache_set_flush(struct closure *cl) · line 1545
Quoted 8 lines
drivers/md/bcache/super.c +1
-0bch_cached_dev_request_init (dc);bch_cached_dev_writeback_init (dc);@@ -1541,33 +1542,59 @@ static void cache_set_flush(struct closure *cl)closure_return (cl); } +/*
+ * dc->>stop_when_cache_set_failed is default to true. If it is explicitly
drivers/md/bcache/super.c static void cache_set_flush(struct closure *cl) · line 1555
Quoted 9 lines
drivers/md/bcache/super.c +9
-0+* set to false by user, the bcache device won't be stopped when cache set +* is broken or disconnected. If there is dirty data on failed cache set, +* not stopping bcache device may result data corruption on backing device, +* pr_warn() notices the protential risk in kernel message. +*/ +static void try_stop_bcache_device (struct cache_set *c, +struct bcache_device *d, +struct cached_dev *dc) +{
+ if (dc->>stop_when_cache_set_failed)
Quoted 84 lines
drivers/md/bcache/super.c +32
-18+bcache_device_stop (d); + else if (!dc->stop_when_cache_set_failed && + atomic_read(&dc->has_dirty)) +pr_warn ("bcache: device %s won't be stopped while unregistering" +" broken dirty cache set %pU, your data has potential " +"risk to be corrupted. To disable this warning message," +" please set /sys/block/%s/bcache/stop_when_" +"cache_set_failed to 1." , + d->name, c->sb.set_uuid, d->name); +} +static void __cache_set_unregister (struct closure *cl) { struct cache_set *c = container_of(cl, struct cache_set, caching);struct cached_dev *dc; +struct bcache_device *d;size_t i;mutex_lock (&bch_register_lock); -for (i =0 ; i < c->devices_max_used; i++) -if (c->devices[i]) { -if (!UUID_FLASH_ONLY (&c->uuids[i]) && -test_bit (CACHE_SET_UNREGISTERING , &c->flags)) { - dc =container_of (c->devices[i], - struct cached_dev, disk); -bch_cached_dev_detach (dc); -/* -* If we come here by too many I/O errors, -* bcache device should be stopped too, to -* keep data consistency on cache and -* backing devices. -*/ -if (test_bit (CACHE_SET_IO_DISABLE , &c->flags)) -bcache_device_stop (c->devices[i]); - }else { -bcache_device_stop (c->devices[i]); - } +for (i =0 ; i < c->devices_max_used; i++) { + d = c->devices[i]; +if (!d) +continue ; + +if (!UUID_FLASH_ONLY (&c->uuids[i]) && +test_bit (CACHE_SET_UNREGISTERING , &c->flags)) { + dc = container_of(d, struct cached_dev, disk); +bch_cached_dev_detach (dc); +/* +* If we come here by too many I/O errors, +* bcache device should be stopped too, to +* keep data consistency on cache and +* backing devices. +*/ +if (test_bit (CACHE_SET_IO_DISABLE , &c->flags)) +try_stop_bcache_device (c, d, dc); + }else { +bcache_device_stop (d); } + }mutex_unlock (&bch_register_lock);drivers/md/bcache/sysfs.c +3
-0@@ -93,6 +93,7 @@ read_attribute(partial_stripes_expensive);rw_attribute (synchronous);rw_attribute (journal_delay_ms);rw_attribute (io_disable); +rw_attribute (stop_when_cache_set_failed);rw_attribute (discard);rw_attribute (running);rw_attribute (label);@@ -134,6 +135,8 @@ SHOW(__bch_cached_dev)sysfs_hprint (io_errors,atomic_read (&dc->io_errors));sysfs_printf (io_error_limit,"%i" , dc->error_limit);sysfs_printf (io_disable,"%i" , dc->io_disable); +sysfs_printf (stop_when_cache_set_failed,"%i" , + dc->stop_when_cache_set_failed);var_print (writeback_rate_update_seconds);var_print (writeback_rate_i_term_inverse);var_print (writeback_rate_p_term_inverse);
drivers/md/bcache/sysfs.c STORE(__cached_dev) · line 241
Quoted 6 lines
drivers/md/bcache/sysfs.c +3
-0@@ -233,6 +236,12 @@ STORE(__cached_dev) dc->io_disable = v ?1 :0 ; } +if (attr == &sysfs_stop_when_cache_set_failed) { +int v =strtoul_or_return (buf); +
+ dc->>stop_when_cache_set_failed = v ? 1 : 0;
drivers/md/bcache/sysfs.c static struct attribute *bch_cached_dev_files[] = { · line 358
Quoted 13 lines
drivers/md/bcache/sysfs.c +3
-0+ } +d_strtoi_h (sequential_cutoff);d_strtoi_h (readahead);@@ -343,6 +352,7 @@ static struct attribute *bch_cached_dev_files[] = { &sysfs_errors, &sysfs_io_error_limit, &sysfs_io_disable, + &sysfs_stop_when_cache_set_failed, &sysfs_dirty_data, &sysfs_stripe_size, &sysfs_partial_stripes_expensive,
Pavel Goran
Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev
Quoted 32 lines · Pavel Goran
Hello Coly,
Quoted 5 lines · you
Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
In the current state, this functionality is rather user-unfriendly.
1. The new "
stop_when_cache_set_failed" option doesn't seem to be persistent. (At least, I did not see any explicit mechanism of saving/restoring it.) That is, users will have to set it on each system startup.2. If the new option is set to zero, it will (supposedly) lead to data corruption/loss when cache set of a "dirty" bcache device is detached. The option that an average home user may want to switch shouldn't be a way to shoot oneself in the foot!
Quoted 1 line
As a remedy, the option could be changed to have the states "always" and
"auto" instead of "1" and "0", so that "auto" would still stop the bcache device if it (or the entire cache set) is "dirty". (Alternatively, it could be renamed to "
force_stop_when_cache_set_failed" or "always_stop_when_cache_set_failed" with values "1" and "0", if string values are undesirable.)Also, the printed warning is somewhat misleading: it says "To disable this warning message, please set /sys/block/%s/bcache/stop_when_cache_set_failed to 1", whereas the suggested change would lead to behaviour change rather that to just disabling the warning.
Quoted 5 lines
3. If (2) is implemented, the default value for the option could be changed to "auto". (The rationale is that enterprise users who want it enabled are better prepared to tune their device settings on startup.) However, this is only important if (1) is not implemented.
Hi Pavel,
I don't like "auto" since its behavior is unpredictable, sometimes whole things are gone, sometimes bcache device exists but slower, for large scale deployed environment, it is confused and misleading.
After thinking for a while, I feel the "auto"/"always" options can make both enterprise and home users get an agreement. I am not able to find other better solution so far, thank you for the hint, brilliant!
Personally I intend to set "always" as the default option, because I maintain bcache for enterprise workloads. The persistent option problem pointed by you does make sense, I will think how to solve it later (probably from user space tools).
How do you think of this ?
Coly Li
[code snipped]
Re[2]: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev
Hello Coly,
Quoted 34 lines · you
Quoted 28 lines
Quoted 5 lines
Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
In the current state, this functionality is rather user-unfriendly.
1. The new "
stop_when_cache_set_failed" option doesn't seem to be persistent. (At least, I did not see any explicit mechanism of saving/restoring it.) That is, users will have to set it on each system startup.2. If the new option is set to zero, it will (supposedly) lead to data corruption/loss when cache set of a "dirty" bcache device is detached. The option that an average home user may want to switch shouldn't be a way to shoot oneself in the foot!
Quoted 1 line
As a remedy, the option could be changed to have the states "always" and
"auto" instead of "1" and "0", so that "auto" would still stop the bcache device if it (or the entire cache set) is "dirty". (Alternatively, it could be renamed to "
force_stop_when_cache_set_failed" or "always_stop_when_cache_set_failed" with values "1" and "0", if string values are undesirable.)Also, the printed warning is somewhat misleading: it says "To disable this warning message, please set /sys/block/%s/bcache/stop_when_cache_set_failed to 1", whereas the suggested change would lead to behaviour change rather that to just disabling the warning.
Quoted 5 lines
3. If (2) is implemented, the default value for the option could be changed to "auto". (The rationale is that enterprise users who want it enabled are better prepared to tune their device settings on startup.) However, this is only important if (1) is not implemented.
Quoted 5 lines
Hi Pavel,
I don't like "auto" since its behavior is unpredictable, sometimes whole things are gone, sometimes bcache device exists but slower, for large scale deployed environment, it is confused and misleading.
In fact, during normal operations, this mode can be made rather predictable:
the device is stopped if it's "writeback", the device isn't stopped if it's "writearound" or "writethrough". The only time it's unpredictable is when the device is being transitioned from "writeback" to other mode.
However, I can see how this adds a certain informational inconsistency to the
picture: an administrator won't be able to say "everything is good" just because all bcache devices work; he'll have to be aware that non-writeback devices get special handling.
Quoted 8 lines
After thinking for a while, I feel the "auto"/"always" options can make both enterprise and home users get an agreement. I am not able to find other better solution so far, thank you for the hint, brilliant!
Personally I intend to set "always" as the default option, because I maintain bcache for enterprise workloads. The persistent option problem pointed by you does make sense, I will think how to solve it later (probably from user space tools).
Providing persistency via user-space tools looks like a bad idea. User-space tools tend to lag behind kernel, so users will not see the new functionality in their distributions for several months or several years. Also, if you're talking about setting the option from user space on each startup, then additionally there is a problem of different incompatible init systems, which will further make things more complicated.
Is there a way to save the option somewhere in bcache superblock, like the cache mode is saved?
Quoted 1 line
How do you think of this ?
Actually, I'd love to hear thoughts from other users and developers. Three opinions is too few for this kind of discussion.
Quoted 1 line
Coly Li
Pavel Goran
Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev
Quoted 44 lines · Pavel Goran
Hello Coly,
Quoted 39 lines · you
Quoted 28 lines
Quoted 5 lines
Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
In the current state, this functionality is rather user-unfriendly.
1. The new "
stop_when_cache_set_failed" option doesn't seem to be persistent. (At least, I did not see any explicit mechanism of saving/restoring it.) That is, users will have to set it on each system startup.2. If the new option is set to zero, it will (supposedly) lead to data corruption/loss when cache set of a "dirty" bcache device is detached. The option that an average home user may want to switch shouldn't be a way to shoot oneself in the foot!
Quoted 1 line
As a remedy, the option could be changed to have the states "always" and
"auto" instead of "1" and "0", so that "auto" would still stop the bcache device if it (or the entire cache set) is "dirty". (Alternatively, it could be renamed to "
force_stop_when_cache_set_failed" or "always_stop_when_cache_set_failed" with values "1" and "0", if string values are undesirable.)Also, the printed warning is somewhat misleading: it says "To disable this warning message, please set /sys/block/%s/bcache/stop_when_cache_set_failed to 1", whereas the suggested change would lead to behaviour change rather that to just disabling the warning.
Quoted 5 lines
3. If (2) is implemented, the default value for the option could be changed to "auto". (The rationale is that enterprise users who want it enabled are better prepared to tune their device settings on startup.) However, this is only important if (1) is not implemented.
Hi Pavel,
I don't like "auto" since its behavior is unpredictable, sometimes whole things are gone, sometimes bcache device exists but slower, for large scale deployed environment, it is confused and misleading.
Hi Pavel,
Quoted 10 lines
In fact, during normal operations, this mode can be made rather predictable:
the device is stopped if it's "writeback", the device isn't stopped if it's "writearound" or "writethrough". The only time it's unpredictable is when the device is being transitioned from "writeback" to other mode.However, I can see how this adds a certain informational inconsistency to the
picture: an administrator won't be able to say "everything is good" just because all bcache devices work; he'll have to be aware that non-writeback devices get special handling.
You got what I meant. I just hope people can know how errors are handled without extra document. In auto mode, print out kernel message to indicate why bcache device is stopped or not might be useful to users to understand what's going on.
Quoted 9 lines
Quoted 8 lines
After thinking for a while, I feel the "auto"/"always" options can make both enterprise and home users get an agreement. I am not able to find other better solution so far, thank you for the hint, brilliant!
Personally I intend to set "always" as the default option, because I maintain bcache for enterprise workloads. The persistent option problem pointed by you does make sense, I will think how to solve it later (probably from user space tools).
Quoted 6 lines
Providing persistency via user-space tools looks like a bad idea. User-space tools tend to lag behind kernel, so users will not see the new functionality in their distributions for several months or several years. Also, if you're talking about setting the option from user space on each startup, then additionally there is a problem of different incompatible init systems, which will further make things more complicated.
OK, I get your point, it makes sense. This is just a rough thought from me, the motivation was to avoid on-disk format change.
Quoted 4 lines
Is there a way to save the option somewhere in bcache superblock, like the cache mode is saved?
It is possible to add flag in bcache superblock, but it means on disk format modification, this is something I try best to avoid. I need to check the code before I have the answer of yes or no.
Quoted 4 lines
Quoted 1 line
How do you think of this ?
Actually, I'd love to hear thoughts from other users and developers. Three opinions is too few for this kind of discussion.
Sure I agree. Indeed the suggestion and feedback for you and Nix helps me quite a lot :-) Firstly I try to avoid data corruption and provide flexibility to enterprise and personal users, then we can improve it in future if someone may provide better suggestion than yours :-)
Thanks.
Coly Li
Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev
In reply to Coly LiQuoted 5 lines · Coly Li
Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
Actually I'm much interested in the server use case. On laptops, it's relatively easy to recover if you know what you're doing, because they usually have a user in front of them with console access -- but if a remote headless server with a hundred users a thousand miles away has its cache device wear out I would really rather the hundred users get served, if more slowly, rather than the whole machine going down for hours or days until I can get someone there to bash on the hardware!
(Sure, ideally you'd detect the wearing out in advance, but SSDs are not always nice and helpful like that and sometimes just go instantly readonly or simply vanish off the bus entirely without warning.)
Signature · 1 line
NULL && (void)
Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev
Quoted 8 lines · Nix
Quoted 5 lines · Coly Li
Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix <[email protected]> points out, users may still want to access the bcache device after cache device failed, for example on laptops.
Quoted 8 lines
Actually I'm much interested in the server use case. On laptops, it's relatively easy to recover if you know what you're doing, because they usually have a user in front of them with console access -- but if a remote headless server with a hundred users a thousand miles away has its cache device wear out I would really rather the hundred users get served, if more slowly, rather than the whole machine going down for hours or days until I can get someone there to bash on the hardware!
Hi Nix,
Thanks for the input, I didn't think of such use case before. It makes a lot sense !
Quoted 4 lines
(Sure, ideally you'd detect the wearing out in advance, but SSDs are not always nice and helpful like that and sometimes just go instantly readonly or simply vanish off the bus entirely without warning.)
Yes. Then in the v5 patch set, I will add an option for "always"/"auto", which will leave bcache device alive if the broken cache set is clean.
Thank you all again, for the insight and brilliant suggestion !
Coly Li