Skip to contentSkip to conversation

bcache detach lead to xfs force shutdown

PATCH New Zhang Zhen 3 participants 13 messages Feb 21, 2022 – Mar 10, 2022 bcache (block layer cache)

lore
Share & tools
lore

Zhang Zhen

author
#0 lore raw

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:

Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive.
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f
Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q,
1181 struct bio *bio)
1182 {

1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data;
1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk);
1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) ||
1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;

1193 }

901 static inline void closure_bio_submit(struct cache_set *c,
902 struct bio *bio,
903 struct closure *cl)
904 {
905 closure_get(cl);
906 if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) {

 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;

910 }
911 generic_make_request(bio);
912 }

Can the cache set detached and don't return error status to fs?

#1 lore raw
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel?

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Thanks.

Coly Li

author
#2 lore raw
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
echo 2 > /proc/sys/vm/drop_caches
ls -R /media/disk1 > /dev/null
done

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Quoted 6 lines

Thanks.

Coly Li

From cb4dff3092707a31017cb3736be39039ece0e646 Mon Sep 17 00:00:00 2001
From: Zhen Zhang <[email protected]>
Date: Wed, 23 Feb 2022 03:40:29 -0800
Subject: [PATCH] Bcache: don't return BLK_STS_IOERR during cache detach

Before this patch, if cache device missing, cached_dev_submit_bio return io err to fs during cache detach, randomly lead to xfs do force shutdown.

This patch delay the cache io submit in cached_dev_submit_bio
and wait for cache set detach finish.
So if the cache device become missing, bcache detach cache set automatically, and the io will sumbit normally.

Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive.
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f
Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)
---
 drivers/md/bcache/bcache.h  | 5 -----
 drivers/md/bcache/request.c | 8 ++++----
 drivers/md/bcache/super.c   | 3 ++-
 3 files changed, 6 insertions(+), 10 deletions(-)
drivers/md/bcache/bcache.h +0 -5
index 9ed9c955add7..e5227dd08e3a 100644
@@ -928,11 +928,6 @@ static inline void closure_bio_submit(struct cache_set *c,
 				      struct closure *cl)
 {
 	closure_get(cl);
-	if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) {
-		bio->bi_status = BLK_STS_IOERR;
-		bio_endio(bio);
-		return;
-	}
 	submit_bio_noacct(bio);
 }
 
drivers/md/bcache/request.c +4 -4
index d15aae6c51c1..36f0ee95b51f 100644
@@ -13,6 +13,7 @@
 #include "request.h"
 #include "writeback.h"
 
+#include <linux/delay.h>
 #include <linux/module.h>
 #include <linux/hash.h>
 #include <linux/random.h>
@@ -1172,11 +1173,10 @@ void cached_dev_submit_bio(struct bio *bio)
 	unsigned long start_time;
 	int rw = bio_data_dir(bio);
 
-	if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) ||
+	while (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) ||
 		     dc->io_disable)) {
-		bio->bi_status = BLK_STS_IOERR;
-		bio_endio(bio);
-		return;
+		/* wait for detach finish and d->c == NULL. */
+		msleep(2);
 	}
 
 	if (likely(d->c)) {
drivers/md/bcache/super.c +2 -1
index 140f35dc0c45..8d9a5e937bc8 100644
@@ -661,7 +661,8 @@ int bch_prio_write(struct cache *ca, bool wait)
 		p->csum		= bch_crc64(&p->magic, meta_bucket_bytes(&ca->sb) - 8);
 
 		bucket = bch_bucket_alloc(ca, RESERVE_PRIO, wait);
-		BUG_ON(bucket == -1);
+		if (bucket == -1)
+			return -1;
 
 		mutex_unlock(&ca->set->bucket_lock);
 		prio_io(ca, bucket, REQ_OP_WRITE, 0);
Signature · 1 line
2.25.1
#3 lore raw
Quoted 91 lines · Zhang Zhen
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

Quoted 6 lines

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Hold on. Why do you think it should be fixed? As I said, it is as-designed behavior.

Coly Li

#4 lore raw

On 2 Mar 2022, Coly Li verbalised:

Quoted 16 lines · Zhang Zhen
Quoted 15 lines

The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

Quoted 8 lines
Quoted 6 lines

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Hold on. Why do you think it should be fixed? As I said, it is as-designed behavior.

I was thinking exactly as you were... but, y'know, this might be a genuine resiliency improvement, *if* done only when the cache is writearound (or none, I suppose!). Obviously if writes are being cached a cache device failure must trigger I/O failures visible to userspace... but if it's just speeding up the underlying device, having it fail would ideally just fail it out and cause all I/Os to proceed uncached, including the one that was underway when the cache I/O error was encountered.

Signature · 1 line
NULL && (void)
author
#5 lore raw
In reply to Coly Li
Quoted 95 lines · Coly Li
Quoted 94 lines · Zhang Zhen
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

Quoted 7 lines
Quoted 3 lines

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Hold on. Why do you think it should be fixed? As I said, it is as-designed behavior.

We use bcache in writearound mode, just cache read io. Currently, bcache return io error during detach, randomly lead to xfs force shutdown.

After bcache auto detach finished, some dir read write normaly, but the others can't read write because of xfs force shutdown. This inconsistency confuses filesystem users.

Thanks.

Quoted 3 lines

Coly Li

#6 lore raw
Quoted 104 lines · Zhang Zhen
Quoted 95 lines · Coly Li
Quoted 94 lines · Zhang Zhen
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

Quoted 7 lines
Quoted 3 lines

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Hold on. Why do you think it should be fixed? As I said, it is as-designed behavior.

Quoted 7 lines

We use bcache in writearound mode, just cache read io. Currently, bcache return io error during detach, randomly lead to xfs force shutdown.

After bcache auto detach finished, some dir read write normaly, but the others can't read write because of xfs force shutdown. This inconsistency confuses filesystem users.

Hi Zhen and Nix,

OK, I come to realize the motivation. Yes you are right, this is an awkward issue and good to be fixed.

Coly Li

#7 lore raw
Quoted 109 lines · Coly Li
Quoted 108 lines · Zhang Zhen
Quoted 95 lines · Coly Li
Quoted 94 lines · Zhang Zhen
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

Quoted 7 lines
Quoted 3 lines

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Hold on. Why do you think it should be fixed? As I said, it is as-designed behavior.

We use bcache in writearound mode, just cache read io. Currently, bcache return io error during detach, randomly lead to xfs force shutdown.

Quoted 9 lines
Quoted 3 lines

After bcache auto detach finished, some dir read write normaly, but the others can't read write because of xfs force shutdown. This inconsistency confuses filesystem users.

Hi Zhen and Nix,

OK, I come to realize the motivation. Yes you are right, this is an awkward issue and good to be fixed.

Hi Coly,

So you will pick this patch into your tree ?

Quoted 4 lines

Coly Li

#8 lore raw
Quoted 13 lines · On 3/7/22 3:56 PM, [email protected] wrote:
Quoted 13 lines
Quoted 2 lines
Quoted 2 lines

Hold on. Why do you think it should be fixed? As I said, it is >>

as-designed behavior.

Quoted 8 lines
Quoted 1 line

We use bcache in writearound mode, just cache read io. Currently, bcache return io error during detach, randomly lead to xfs force shutdown.

After bcache auto detach finished, some dir read write normaly, but the others can't read write because of xfs force shutdown. This inconsistency confuses filesystem users.

Quoted 7 lines
Quoted 4 lines

Hi Zhen and Nix,

OK, I come to realize the motivation. Yes you are right, this is an awkward issue and good to be fixed.

Hi Coly,

So you will pick this patch into your tree ?

I need to look the patch firstly, could you please post a proposal patch for review ?

Coly Li

#9 lore raw
Quoted 17 lines · Coly Li
Quoted 15 lines · On 3/7/22 3:56 PM, [email protected] wrote:
Quoted 15 lines
Quoted 2 lines
Quoted 2 lines

Hold on. Why do you think it should be fixed? As I said, it is >>

as-designed behavior.

Quoted 8 lines
Quoted 1 line

We use bcache in writearound mode, just cache read io. Currently, bcache return io error during detach, randomly lead to xfs force shutdown.

After bcache auto detach finished, some dir read write normaly, but the others can't read write because of xfs force shutdown. This inconsistency confuses filesystem users.

Hi Zhen and Nix,

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

OK, I come to realize the motivation. Yes you are right, this is an awkward issue and good to be fixed.

Hi Coly,

So you will pick this patch into your tree ?

I need to look the patch firstly, could you please post a proposal patch for review ?

Ok.

Quoted 5 lines

Coly Li

author
#10 lore raw

PATCH 1/1 Bcache: don't return BLK_STS_IOERR during cache detach

In reply to Coly Li

Before this patch, if cache device missing, cached_dev_submit_bio return io err to fs during cache detach, randomly lead to xfs do force shutdown.

This patch delay the cache io submit in cached_dev_submit_bio
and wait for cache set detach finish.
So if the cache device become missing, bcache detach cache set automatically, and the io will sumbit normally.

Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive.
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f
Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)
---
 drivers/md/bcache/bcache.h  | 5 -----
 drivers/md/bcache/request.c | 8 ++++----
 drivers/md/bcache/super.c   | 3 ++-
 3 files changed, 6 insertions(+), 10 deletions(-)
drivers/md/bcache/bcache.h +0 -5
index 9ed9c955add7..e5227dd08e3a 100644
@@ -928,11 +928,6 @@ static inline void closure_bio_submit(struct cache_set *c,
 				      struct closure *cl)
 {
 	closure_get(cl);
-	if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) {
-		bio->bi_status = BLK_STS_IOERR;
-		bio_endio(bio);
-		return;
-	}
 	submit_bio_noacct(bio);
 }
 
drivers/md/bcache/request.c +4 -4
index d15aae6c51c1..36f0ee95b51f 100644
@@ -13,6 +13,7 @@
 #include "request.h"
 #include "writeback.h"
 
+#include <linux/delay.h>
 #include <linux/module.h>
 #include <linux/hash.h>
 #include <linux/random.h>
@@ -1172,11 +1173,10 @@ void cached_dev_submit_bio(struct bio *bio)
 	unsigned long start_time;
 	int rw = bio_data_dir(bio);
 
-	if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) ||
+	while (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) ||
 		     dc->io_disable)) {
-		bio->bi_status = BLK_STS_IOERR;
-		bio_endio(bio);
-		return;
+		/* wait for detach finish and d->c == NULL. */
+		msleep(2);
 	}
 
 	if (likely(d->c)) {
drivers/md/bcache/super.c +2 -1
index 140f35dc0c45..8d9a5e937bc8 100644
@@ -661,7 +661,8 @@ int bch_prio_write(struct cache *ca, bool wait)
 		p->csum		= bch_crc64(&p->magic, meta_bucket_bytes(&ca->sb) - 8);
 
 		bucket = bch_bucket_alloc(ca, RESERVE_PRIO, wait);
-		BUG_ON(bucket == -1);
+		if (bucket == -1)
+			return -1;
 
 		mutex_unlock(&ca->set->bucket_lock);
 		prio_io(ca, bucket, REQ_OP_WRITE, 0);
Signature · 1 line
2.25.1
#11 lore raw
In reply to Zhang Zhen
Quoted 91 lines · Zhang Zhen
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

Quoted 6 lines

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Could you please post the patch in email body? Otherwise I am not able to reply you in line. I read your patch, it is unacceptable in general, but I'd like to provide my point for your reference.

Coly Li

author
#12 lore raw
Quoted 95 lines · Coly Li
Quoted 94 lines · Zhang Zhen
Quoted 62 lines · Coly Li
Quoted 61 lines · Zhang Zhen

Hi coly,

We encounted a bcache detach problem, during the io process,the cache device become missing.

The io error status returned to xfs, and in some case, the xfs do force shutdown.

The dmesg as follows:
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p44: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p57: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_count_io_errors() nvme0n1p56: IO error on writing btree.
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x80034658 len 32 error 12 Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() bcache: error on 004f8aa7-561a-4ba7-bf7b-292e461d3f18:
Feb  2 20:59:23  kernel: journal io error
Feb  2 20:59:23  kernel: bcache: bch_cache_set_error() , disabling caching
Feb  2 20:59:23  kernel: bcache: bch_btree_insert() error -5
Feb  2 20:59:23  kernel: bcache: conditional_stop_bcache_device() stop_when_cache_set_failed of bcache43 is "auto" and cache is clean, keep it alive. Feb  2 20:59:23  kernel: XFS (bcache43): metadata I/O error in "xlog_iodone" at daddr 0x400123e60 len 64 error 12
Feb  2 20:59:23  kernel: XFS (bcache43): xfs_do_force_shutdown(0x2) called from line 1298 of file fs/xfs/xfs_log.c. Return address = 00000000c1c8077f Feb  2 20:59:23  kernel: XFS (bcache43): Log I/O Error Detected. Shutting down filesystem
Feb  2 20:59:23  kernel: XFS (bcache43): Please unmount the filesystem and rectify the problem(s)

We checked the code, the error status is returned in cached_dev_make_request and closure_bio_submit function.

1180 static blk_qc_t cached_dev_make_request(struct request_queue *q, 1181                     struct bio *bio)
1182 {
1183     struct search *s;
1184     struct bcache_device *d = bio->bi_disk->private_data; 1185     struct cached_dev *dc = container_of(d, struct cached_dev, disk); 1186     int rw = bio_data_dir(bio);
1187
1188     if (unlikely((d->c && test_bit(CACHE_SET_IO_DISABLE, &d->c->flags)) || 1189              dc->io_disable)) {
1190         bio->bi_status = BLK_STS_IOERR;
1191         bio_endio(bio);
1192         return BLK_QC_T_NONE;
1193     }

 901 static inline void closure_bio_submit(struct cache_set *c, 902                       struct bio *bio,
 903                       struct closure *cl)
 904 {
 905     closure_get(cl);
 906     if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags))) { 907         bio->bi_status = BLK_STS_IOERR;
 908         bio_endio(bio);
 909         return;
 910     }
 911     generic_make_request(bio);
 912 }

Quoted 8 lines
Quoted 1 line

Can the cache set detached and don't return error status to fs?

Hi Zhang,

What is your kernel version and where do you get the kernel? My kernel version is 4.18 of Centos.

The code of this part is same with upstream kernel.

Quoted 2 lines

It seems like an as designed behavior, could you please describe more detail about the operation sequence?

Yes, i think so too.
The reproduce opreation as follows:
1. mount a bcache disk with xfs

/dev/bcache1 on /media/disk1 type xfs

2. run ls in background
#!/bin/bash

while true
do
  echo 2 > /proc/sys/vm/drop_caches
  ls -R /media/disk1 > /dev/null
done

3. remove cache disk sdc
echo 1 >/sys/block/sdc/device/delete

Quoted 7 lines
Quoted 3 lines

4. dmesg should get xfs error

I write a patch to improve,please help to review it, thanks.

Could you please post the patch in email body? Otherwise I am not able to reply you in line. I read your patch, it is unacceptable in general, but I'd like to provide my point for your reference.

Ok,i will send the patch in another email.

Thanks.

Quoted 6 lines

Coly Li