Skip to contentSkip to conversation

Split brain detection

PATCH New Kent Overstreet 1 participant 3 messages Nov 2, 2023

lore
Share & tools
lore
#0 lore raw

cover Split brain detection

Two patch series for split brain detection: that is, detect when different members of a filesystem have been used, independently, in degraded mode and diverged - when that happens we can no longer use those members in combination.

First patch adds superblock facilities: we add a superblock sequence number field to bch_member, and a write time field to the superblock to detect cases when sb->seq is equal but they actually have diverged.

Second patch does the actual split brain detection and kicking out of devices when necessary.

Kent Overstreet (2):
bcachefs: bch_member->seq
bcachefs: Split brain detection

 fs/bcachefs/bcachefs_format.h |  4 ++-
 fs/bcachefs/errcode.h         |  1 +
 fs/bcachefs/sb-members.c      |  5 ++++
 fs/bcachefs/super-io.c        | 10 +++++++
 fs/bcachefs/super.c           | 56 ++++++++++++++++++++++++++++-------
 5 files changed, 64 insertions(+), 12 deletions(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/2 bcachefs: bch_member->seq

Add new fields for split brain detection:

- bch_member->seq, which tracks the sequence number of the last superblock write that happened to each member device

- bch_sb->write_time, which tracks the time of the last superblock write, to allow detection of when two members have diverged but had the same number of superblock writes.

---
 fs/bcachefs/bcachefs_format.h |  4 +++-
 fs/bcachefs/sb-members.c      |  5 +++++
 fs/bcachefs/super-io.c        | 10 ++++++++++
 3 files changed, 18 insertions(+), 1 deletion(-)
fs/bcachefs/bcachefs_format.h +3 -1
index 5b44598b9df9..b22868ca9770 100644
@@ -1294,6 +1294,7 @@ struct bch_member {
 	__le64			errors[BCH_MEMBER_ERROR_NR];
 	__le64			errors_at_reset[BCH_MEMBER_ERROR_NR];
 	__le64			errors_reset_time;
+	__le64			seq;
 };
 
 #define BCH_MEMBER_V1_BYTES	56
@@ -1761,7 +1762,8 @@ struct bch_sb {
 	__le32			time_base_hi;
 	__le32			time_precision;
 
-	__le64			flags[8];
+	__le64			flags[7];
+	__le64			write_time;
 	__le64			features[2];
 	__le64			compat[2];
 
fs/bcachefs/sb-members.c +5 -0
index 6a7e20de971c..4c6908cfca05 100644
@@ -235,6 +235,11 @@ static void member_to_text(struct printbuf *out,
 		prt_printf(out, "(never)");
 	prt_newline(out);
 
+	prt_printf(out, "Last superblock write:");
+	prt_tab(out);
+	prt_u64(out, le64_to_cpu(m.seq));
+	prt_newline(out);
+
 	prt_printf(out, "State:");
 	prt_tab(out);
 	prt_printf(out, "%s",
fs/bcachefs/super-io.c +10 -0
index a93e53d0b37e..3ebe14c806ac 100644
@@ -890,6 +890,11 @@ int bch2_write_super(struct bch_fs *c)
 
 	le64_add_cpu(&c->disk_sb.sb->seq, 1);
 
+	struct bch_sb_field_members_v2 *mi = bch2_sb_field_get(c->disk_sb.sb, members_v2);
+	for_each_online_member(ca, c, i)
+		__bch2_members_v2_get_mut(mi, i)->seq = c->disk_sb.sb->seq;
+	c->disk_sb.sb->write_time = cpu_to_le64(ktime_get_real_seconds());
+
 	if (test_bit(BCH_FS_ERROR, &c->flags))
 		SET_BCH_SB_HAS_ERRORS(c->disk_sb.sb, 1);
 	if (test_bit(BCH_FS_TOPOLOGY_ERROR, &c->flags))
@@ -1193,6 +1198,11 @@ void bch2_sb_to_text(struct printbuf *out, struct bch_sb *sb,
 	prt_printf(out, "%llu", le64_to_cpu(sb->seq));
 	prt_newline(out);
 
+	prt_printf(out, "Time of last write:");
+	prt_tab(out);
+	bch2_prt_date_seconds(out, le64_to_cpu(sb->write_time));
+	prt_newline(out);
+
 	prt_printf(out, "Superblock size:");
 	prt_tab(out);
 	prt_printf(out, "%zu", vstruct_bytes(sb));
Signature · 1 line
2.42.0
#2 lore raw

PATCH 2/2 bcachefs: Split brain detection

In reply to Kent Overstreet

Use the new bch_member->seq, sb->write_time fields to detect split brain and kick out devices when necessary.

---
 fs/bcachefs/errcode.h |  1 +
 fs/bcachefs/super.c   | 56 ++++++++++++++++++++++++++++++++++---------
 2 files changed, 46 insertions(+), 11 deletions(-)
fs/bcachefs/errcode.h +1 -0
index 2a11f32cf30a..e1f733eeb7f0 100644
@@ -169,6 +169,7 @@
 	x(EINVAL,			device_size_too_small)			\
 	x(EINVAL,			device_not_a_member_of_filesystem)	\
 	x(EINVAL,			device_has_been_removed)		\
+	x(EINVAL,			device_splitbrain)			\
 	x(EINVAL,			device_already_online)			\
 	x(EINVAL,			insufficient_devices_to_start)		\
 	x(EINVAL,			invalid)				\
fs/bcachefs/super.c +45 -11

97 lines not on this page · raw message

Signature · 1 line
2.42.0