Split brain detection
cover Split brain detection
Two patch series for split brain detection: that is, detect when different members of a filesystem have been used, independently, in degraded mode and diverged - when that happens we can no longer use those members in combination.
First patch adds superblock facilities: we add a superblock sequence number field to bch_member, and a write time field to the superblock to detect cases when sb->seq is equal but they actually have diverged.
Second patch does the actual split brain detection and kicking out of devices when necessary.
Kent Overstreet (2):
bcachefs: bch_member->seq
bcachefs: Split brain detection
fs/bcachefs/bcachefs_format.h | 4 ++- fs/bcachefs/errcode.h | 1 + fs/bcachefs/sb-members.c | 5 ++++ fs/bcachefs/super-io.c | 10 +++++++ fs/bcachefs/super.c | 56 ++++++++++++++++++++++++++++------- 5 files changed, 64 insertions(+), 12 deletions(-)
Signature · 1 line
2.42.0
PATCH 1/2 bcachefs: bch_member->seq
Add new fields for split brain detection:
- bch_member->seq, which tracks the sequence number of the last superblock write that happened to each member device
- bch_sb->write_time, which tracks the time of the last superblock write, to allow detection of when two members have diverged but had the same number of superblock writes.
--- fs/bcachefs/bcachefs_format.h | 4 +++- fs/bcachefs/sb-members.c | 5 +++++ fs/bcachefs/super-io.c | 10 ++++++++++ 3 files changed, 18 insertions(+), 1 deletion(-)
fs/bcachefs/bcachefs_format.h +3 -1
@@ -1294,6 +1294,7 @@ struct bch_member {__le64 errors[BCH_MEMBER_ERROR_NR ];__le64 errors_at_reset[BCH_MEMBER_ERROR_NR ];__le64 errors_reset_time; +__le64 seq; };#define BCH_MEMBER_V1_BYTES 56
@@ -1761,7 +1762,8 @@ struct bch_sb {__le32 time_base_hi;__le32 time_precision; -__le64 flags[8 ]; +__le64 flags[7 ]; +__le64 write_time;__le64 features[2 ];__le64 compat[2 ];
fs/bcachefs/sb-members.c +5 -0
@@ -235,6 +235,11 @@ static void member_to_text(struct printbuf *out,prt_printf (out,"(never)" );prt_newline (out); +prt_printf (out,"Last superblock write:" ); +prt_tab (out); +prt_u64 (out,le64_to_cpu (m.seq)); +prt_newline (out); +prt_printf (out,"State:" );prt_tab (out); prt_printf(out, "%s",
fs/bcachefs/super-io.c +10 -0
@@ -890,6 +890,11 @@ int bch2_write_super(struct bch_fs *c)le64_add_cpu (&c->disk_sb.sb->seq,1 ); +struct bch_sb_field_members_v2 *mi =bch2_sb_field_get (c->disk_sb.sb, members_v2); +for_each_online_member (ca, c, i) +__bch2_members_v2_get_mut (mi, i)->seq = c->disk_sb.sb->seq; + c->disk_sb.sb->write_time =cpu_to_le64 (ktime_get_real_seconds ()); +if (test_bit (BCH_FS_ERROR , &c->flags))SET_BCH_SB_HAS_ERRORS (c->disk_sb.sb,1 );if (test_bit (BCH_FS_TOPOLOGY_ERROR , &c->flags))
@@ -1193,6 +1198,11 @@ void bch2_sb_to_text(struct printbuf *out, struct bch_sb *sb,prt_printf (out,"%llu" ,le64_to_cpu (sb->seq));prt_newline (out); +prt_printf (out,"Time of last write:" ); +prt_tab (out); +bch2_prt_date_seconds (out,le64_to_cpu (sb->write_time)); +prt_newline (out); +prt_printf (out,"Superblock size:" );prt_tab (out);prt_printf (out,"%zu" ,vstruct_bytes (sb));
Signature · 1 line
2.42.0
PATCH 2/2 bcachefs: Split brain detection
In reply to Kent OverstreetUse the new bch_member->seq, sb->write_time fields to detect split brain and kick out devices when necessary.
--- fs/bcachefs/errcode.h | 1 + fs/bcachefs/super.c | 56 ++++++++++++++++++++++++++++++++++--------- 2 files changed, 46 insertions(+), 11 deletions(-)
fs/bcachefs/errcode.h +1 -0
@@ -169,6 +169,7 @@ x(EINVAL, device_size_too_small) \ x(EINVAL, device_not_a_member_of_filesystem) \ x(EINVAL, device_has_been_removed) \ + x(EINVAL, device_splitbrain) \ x(EINVAL, device_already_online) \ x(EINVAL, insufficient_devices_to_start) \x (EINVAL , invalid) \
fs/bcachefs/super.c +45 -11
97 lines not on this page · raw message
Signature · 1 line
2.42.0