bcachefs - semvar, forward compatibility
cover bcachefs - semvar, forward compatibility
So - in the upstreaming discussion, Brian mentioned code review, so now seems like a good time to start making sure bcachefs patches hit the list.
In the last cabal meeting, we started talking about on disk compatibility issues for mainlining. Since (IIRC) the first release, we've maintained backwards compatibility (support for very old versions has been dropped, but there's always been an upgrade path) - but we generally haven't been addressing forwards compatibility yet - we've been doing a lot of forced incompatible version upgrades, where the old version is no longer able to mount the filesystem after it's been mounted by the new version.
Obviously, we can't do that anymore after we're in mainline and out of EXPERIMENTAL. There were two main issues to address:
Major/minor version numbers
---------------------------
bcachefs started out with the traditional compatible/incompatible feature bits in the superblock, but they are no longer my preferred approach.
The problem with feature bits is that there was an ordering in which new on disk format features were released, and feature bits lose that
ordering: they make it possible for users to create filesystems where x, y, and z modern feature bits are enabled, but not feature bit a from 5 years ago - and the code was never written to expect that and you certainly never tested that configuration, so things break in incredibly fun ways.
Assigning every new on disk feature a distinct version number instead of a feature bit preserves this ordering and makes it impossible for users to create or use filesystems with features selected that historically should not have existed. This has been the practice in bcachefs for awhile now, and I've been quite happy with it.
The missing bit that this patch series adds is to split the version number field into major and minor versions. Incrementing the minor version number corresponds to adding a new compat feature flag, if we were using feature bits: incrementing the major version number corresponds to adding a new incompatible feature bit.
IOW, we'll allow mounting a filesystem with a version number greater than the currently supported version as long as the major version number is the same. As with compat feature bits, if you do so the filesystem will be downgraded to the currently supported version, indicating those new on disk structures may now be inconsistent.
Forwards compatibility of on disk structures:
---------------------------------------------
We need to be able to roll out new on disk data structures without causing problems for old versions - old versions should just ignore metadata they don't understand. This had been planned for in the past and most of the work was done, so there wasn't much left.
Specifically, we need to be able to roll out new
- Superblock sections: already handled, audited and cleaned up a bit
- Journal entry types: already handled, audited
- Btrees: addressed by this patchset
- Bkey types: addressed by this patchset
- New fields for existing bkeys: addressed by the patch series that introduced "bch2_bkey_get_val_typed()", but this is the trickiest to handle and likely more work will be required
With all this in place, we'll be able to roll out most of the new features we want that require new on disk data structures as forwards compatible changes, including everything currently in the pipeline. That includes
- Snapshot nodes are gaining skiplist entries soon: this will fix O(n) issues with bch2_snapshot_is_ancestor()
- rebalance_work btree: Rebalance is the last operation that happens during normal operation that requires metadata scanning - soon I'll be adding a rebalance_work btree that references extents that rebalance will have work to do on in the future (e.g. for the background_compression or background_target io options).
- inodes_deleted btree: After unclean shutdown we still have to scan the entire inodes btree for deleted inodes, I'll be adding another bitset btree to address this - and also adding a tmpdir feature as well.
Things that will require incompatible changes:
- New key types that replace existing key types, or in general new data structures that replace existing data structures
Where we can maintain both the old and new data structures this isn't a problem - e.g. we can roll out a new bch_sb_members_v2 superblock section and just also keep writing out bch_sb_members for old versions to use; but we won't be able to roll out e.g. a new extent key type without an incompatible change.
- New btree node header/journal entry headers - we'd like bigger nonces, so this will need to happen eventually
- New extent_entry types: this one is a bit unfortunate, because extents contain a list of variable size fields (e.g. ptrs, different sized crc entries) and the entries themselves don't specify their size - the code that's reading it has to know how big every extent entry type is.
This just came up with rebalance_work - rebalance_work needs a new extent entry type, so I rolled that out ahead of time so we can roll out the rest of the functionality as a compatible change.
Forced version upgrades:
------------------------
Going forward, we will still be doing forced version upgrades for awhile
- but only to forwards-compatible versions. After the next incompatible (version 2.0) release, we likely won't be doing forced version upgrades at all anymore.
Currently, version upgrades generally require a fsck. Another thing this patchset addresses is enumerating all our recovery (including version upgrade and fsck passes); this will let us specify "upgrading to this version only requires this pass to run".
Kent Overstreet (10):
bcachefs: Allow for unknown btree IDs
bcachefs: Allow for unknown key types
bcachefs: Refactor bch_sb_field_ops handling
bcachefs: Change check for invalid key types
bcachefs: BCH_SB_VERSION_UPGRADE_COMPLETE()
bcachefs: version_upgrade is now an enum
bcachefs: Kill bch2_bucket_gens_read()
bcachefs: Stash journal replay params in bch_fs
bcachefs: Enumerate recovery passes
bcachefs: bcachefs_metadata_version_major_minor
fs/bcachefs/alloc_background.c | 129 +++++----- fs/bcachefs/alloc_background.h | 18 +- fs/bcachefs/alloc_foreground.c | 9 +- fs/bcachefs/backpointers.c | 23 +- fs/bcachefs/backpointers.h | 2 +- fs/bcachefs/bcachefs.h | 62 ++++- fs/bcachefs/bcachefs_format.h | 63 +++-- fs/bcachefs/bkey_methods.c | 81 ++++--- fs/bcachefs/bkey_methods.h | 20 +- fs/bcachefs/btree_cache.c | 23 +- fs/bcachefs/btree_cache.h | 22 +- fs/bcachefs/btree_gc.c | 26 +- fs/bcachefs/btree_io.c | 9 +- fs/bcachefs/btree_iter.c | 4 +- fs/bcachefs/btree_update_interior.c | 18 +- fs/bcachefs/btree_update_leaf.c | 17 +- fs/bcachefs/dirent.c | 3 +- fs/bcachefs/dirent.h | 4 +- fs/bcachefs/ec.c | 3 +- fs/bcachefs/ec.h | 4 +- fs/bcachefs/extents.c | 12 +- fs/bcachefs/extents.h | 9 +- fs/bcachefs/fsck.c | 77 +----- fs/bcachefs/fsck.h | 10 +- fs/bcachefs/inode.c | 12 +- fs/bcachefs/inode.h | 12 +- fs/bcachefs/journal_io.c | 15 +- fs/bcachefs/lru.c | 3 +- fs/bcachefs/lru.h | 3 +- fs/bcachefs/move.c | 10 +- fs/bcachefs/opts.c | 5 + fs/bcachefs/opts.h | 5 +- fs/bcachefs/quota.c | 3 +- fs/bcachefs/quota.h | 4 +- fs/bcachefs/recovery.c | 353 ++++++++++++++-------------- fs/bcachefs/reflink.c | 9 +- fs/bcachefs/reflink.h | 8 +- fs/bcachefs/subvolume.c | 16 +- fs/bcachefs/subvolume.h | 14 +- fs/bcachefs/super-io.c | 91 +++++-- fs/bcachefs/super-io.h | 3 +- fs/bcachefs/super.c | 1 + fs/bcachefs/xattr.c | 3 +- fs/bcachefs/xattr.h | 3 +- 44 files changed, 700 insertions(+), 521 deletions(-)
Signature · 1 line
2.40.1
PATCH 1/10 bcachefs: Allow for unknown btree IDs
We need to allow filesystems with metadata from newer versions to be mountable and usable by older versions.
This patch enables us to roll out new btrees without a new major version number; we can now handle btree roots for unknown btree types.
The unknown btree roots will be retained, and fsck (including backpointers) will check them, the same as other btree types.
We add a dynamic array for the extra, unknown btree roots, in addition to the fixed size btree root array, and add new helpers for looking up btree roots.
--- fs/bcachefs/backpointers.c | 14 ++++++++------ fs/bcachefs/bcachefs.h | 3 ++- fs/bcachefs/btree_cache.c | 21 +++++++++++++-------- fs/bcachefs/btree_cache.h | 22 +++++++++++++++++++++- fs/bcachefs/btree_gc.c | 22 ++++++++++++++++++---- fs/bcachefs/btree_io.c | 2 +- fs/bcachefs/btree_iter.c | 4 ++-- fs/bcachefs/btree_update_interior.c | 18 +++++++++--------- fs/bcachefs/move.c | 10 ++++++++-- fs/bcachefs/recovery.c | 16 ++++++++-------- fs/bcachefs/super.c | 1 + 11 files changed, 91 insertions(+), 42 deletions(-)
fs/bcachefs/backpointers.c +8 -6
@@ -272,6 +272,7 @@ struct bkey_s_c bch2_backpointer_get_key(struct btree_trans *trans, unsigned iter_flags) {struct bch_fs *c = trans->c; +struct btree_root *r =bch2_btree_id_root (c, bp.btree_id);struct bpos bucket =bp_pos_to_bucket (c, bp_pos);struct bkey_s_c k;
@@ -279,7 +280,7 @@ struct bkey_s_c bch2_backpointer_get_key(struct btree_trans *trans, bp.btree_id, bp.pos,0 , -min (bp.level, c->btree_roots[bp.btree_id].level), +min (bp.level, r->level), iter_flags); k =bch2_btree_iter_peek_slot (iter);if (bkey_err (k)) {
@@ -287,8 +288,8 @@ struct bkey_s_c bch2_backpointer_get_key(struct btree_trans *trans,return k; } -if (bp.level == c->btree_roots[bp.btree_id].level +1 ) - k =bkey_i_to_s_c (&c->btree_roots[bp.btree_id].key); +if (bp.level == r->level +1 ) + k =bkey_i_to_s_c (&r->key);if (k.k &&extent_matches_bp (c, bp.btree_id, bp.level, k, bucket, bp))return k;
@@ -531,6 +532,7 @@ static int check_btree_root_to_backpointers(struct btree_trans *trans, struct bpos_level *last_flushed) {struct bch_fs *c = trans->c; +struct btree_root *r =bch2_btree_id_root (c, btree_id);struct btree_iter iter;struct btree *b;struct bkey_s_c k;
@@ -539,8 +541,7 @@ static int check_btree_root_to_backpointers(struct btree_trans *trans,const union bch_extent_entry *entry;int ret; -bch2_trans_node_iter_init (trans, &iter, btree_id,POS_MIN ,0 , - c->btree_roots[btree_id].level,0 ); +bch2_trans_node_iter_init (trans, &iter, btree_id,POS_MIN ,0 , r->level,0 ); b =bch2_btree_iter_peek_node (&iter); ret =PTR_ERR_OR_ZERO (b);if (ret)
@@ -640,12 +641,13 @@ static int bch2_check_extents_to_backpointers_pass(struct btree_trans *trans,struct bpos bucket_start, struct bpos bucket_end) { + struct bch_fs *c = trans->c;struct btree_iter iter;enum btree_id btree_id;struct bpos_level last_flushed = {UINT_MAX };int ret =0 ; -for (btree_id =0 ; btree_id <BTREE_ID_NR ; btree_id++) { +for (btree_id =0 ; btree_id <btree_id_nr_alive (c); btree_id++) {unsigned depth =btree_type_has_ptrs (btree_id) ?0 :1 ; bch2_trans_node_iter_init(trans, &iter, btree_id, POS_MIN, 0,
fs/bcachefs/bcachefs.h +2 -1
@@ -749,7 +749,8 @@ struct bch_fs {struct bio_set btree_bio;struct workqueue_struct *io_complete_wq; -struct btree_root btree_roots[BTREE_ID_NR ]; +struct btree_root btree_roots_known[BTREE_ID_NR ]; +DARRAY (struct btree_root ) btree_roots_extra;struct mutex btree_root_lock;struct btree_cache btree_cache;
fs/bcachefs/btree_cache.c +13 -8
@@ -32,13 +32,15 @@ void bch2_recalc_btree_reserve(struct bch_fs *c) {unsigned i, reserve =16 ; -if (!c->btree_roots[0 ].b) +if (!c->btree_roots_known[0 ].b) reserve +=8 ; -for (i =0 ; i <BTREE_ID_NR ; i++) -if (c->btree_roots[i].b) - reserve +=min_t (unsigned,1 , - c->btree_roots[i].b->c.level) *8 ; +for (i =0 ; i <btree_id_nr_alive (c); i++) { +struct btree_root *r =bch2_btree_id_root (c, i); + +if (r->b) + reserve +=min_t (unsigned,1 , r->b->c.level) *8 ; + } c->btree_cache.reserve = reserve; }
@@ -457,9 +459,12 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c)kvpfree (c->verify_ondisk,btree_bytes (c)); -for (i =0 ; i <BTREE_ID_NR ; i++) -if (c->btree_roots[i].b) -list_add (&c->btree_roots[i].b->list, &bc->live); +for (i =0 ; i <btree_id_nr_alive (c); i++) { +struct btree_root *r =bch2_btree_id_root (c, i); + +if (r->b) +list_add (&r->b->list, &bc->live); + }list_splice (&bc->freeable, &bc->live);
fs/bcachefs/btree_cache.h +21 -1
@@ -101,7 +101,27 @@ static inline unsigned btree_blocks(struct bch_fs *c) (BTREE_FOREGROUND_MERGE_THRESHOLD (c) + \ (BTREE_FOREGROUND_MERGE_THRESHOLD(c) >> 2)) -#define btree_node_root (_c, _b) ((_c)->btree_roots[(_b)->c.btree_id].b) +static inline unsigned btree_id_nr_alive (struct bch_fs *c) +{ +return BTREE_ID_NR + c->btree_roots_extra.nr; +} + +static inline struct btree_root *bch2_btree_id_root (struct bch_fs *c,unsigned id) +{ +if (likely (id <BTREE_ID_NR )) { +return &c->btree_roots_known[id]; + }else { +unsigned idx = id -BTREE_ID_NR ; + +EBUG_ON (idx >= c->btree_roots_extra.nr); +return &c->btree_roots_extra.data[idx]; + } +} + +static inline struct btree *btree_node_root (struct bch_fs *c,struct btree *b) +{ +return bch2_btree_id_root (c, b->c.btree_id)->b; +}void bch2_btree_node_to_text (struct printbuf *,struct bch_fs *,const struct btree *);
fs/bcachefs/btree_gc.c +18 -4
@@ -529,8 +529,13 @@ static int bch2_repair_topology(struct bch_fs *c)bch2_trans_init (&trans, c,0 ,0 ); -for (i =0 ; i <BTREE_ID_NR && !ret; i++) { - b = c->btree_roots[i].b; +for (i =0 ; i <btree_id_nr_alive (c)&& !ret; i++) { +struct btree_root *r =bch2_btree_id_root (c, i); + +if (!r->alive) +continue ; + + b = r->b;if (btree_node_fake (b))continue ;
@@ -883,7 +888,7 @@ static int bch2_gc_btree(struct btree_trans *trans, enum btree_id btree_id,return ret;mutex_lock (&c->btree_root_lock); - b = c->btree_roots[btree_id].b; + b =bch2_btree_id_root (c, btree_id)->b;if (!btree_node_fake (b)) {struct bkey_s_c k =bkey_i_to_s_c (&b->key);
@@ -1006,7 +1011,7 @@ static int bch2_gc_btree_init(struct btree_trans *trans,struct printbuf buf =PRINTBUF ;int ret =0 ; - b = c->btree_roots[btree_id].b; + b =bch2_btree_id_root (c, btree_id)->b;if (btree_node_fake (b))return 0 ;
@@ -1075,6 +1080,15 @@ static int bch2_gc_btrees(struct bch_fs *c, bool initial, bool metadata_only) ?bch2_gc_btree_init (&trans, ids[i], metadata_only) :bch2_gc_btree (&trans, ids[i], initial, metadata_only); +for (i =BTREE_ID_NR ; i <btree_id_nr_alive (c) && !ret; i++) { +if (!bch2_btree_id_root (c, i)->alive) +continue ; + + ret = initial + ?bch2_gc_btree_init (&trans, i, metadata_only) + :bch2_gc_btree (&trans, i, initial, metadata_only); + } +if (ret <0 )bch_err_fn (c, ret);
fs/bcachefs/btree_io.c +1 -1
@@ -517,7 +517,7 @@ static void btree_pos_to_text(struct printbuf *out, struct bch_fs *c,prt_printf (out,"%s level %u/%u\n " , bch2_btree_ids[b->c.btree_id], b->c.level, - c->btree_roots[b->c.btree_id].level); +bch2_btree_id_root (c, b->c.btree_id)->level);bch2_bkey_val_to_text (out, c,bkey_i_to_s_c (&b->key)); }
fs/bcachefs/btree_iter.c +2 -2
@@ -238,7 +238,7 @@ static void bch2_btree_path_verify(struct btree_trans *trans,for (i =0 ; i < (!path->cached ?BTREE_MAX_DEPTH :1 ); i++) {if (!path->l[i].b) {BUG_ON (!path->cached && - c->btree_roots[path->btree_id].b->c.level > i); +bch2_btree_id_root (c, path->btree_id)->b->c.level > i);break ; }
@@ -732,7 +732,7 @@ static inline int btree_path_lock_root(struct btree_trans *trans, unsigned long trace_ip) {struct bch_fs *c = trans->c; -struct btree *b, **rootp = &c->btree_roots[path->btree_id].b; +struct btree *b, **rootp = &bch2_btree_id_root (c, path->btree_id)->b;enum six_lock_type lock_type;unsigned i;int ret;
fs/bcachefs/btree_update_interior.c +9 -9
@@ -1199,7 +1199,7 @@ static void bch2_btree_set_root_inmem(struct bch_fs *c, struct btree *b) (b->c.level <btree_node_root (c, b)->c.level || !btree_node_dying(btree_node_root(c, b)))); -btree_node_root (c, b) = b; +bch2_btree_id_root (c, b->c.btree_id)->b = b;mutex_unlock (&c->btree_root_lock);bch2_recalc_btree_reserve (c);
@@ -2402,7 +2402,7 @@ bool bch2_btree_interior_updates_flush(struct bch_fs *c)void bch2_journal_entry_to_btree_root (struct bch_fs *c,struct jset_entry *entry) { -struct btree_root *r = &c->btree_roots[entry->btree_id]; +struct btree_root *r =bch2_btree_id_root (c, entry->btree_id);mutex_lock (&c->btree_root_lock);
@@ -2428,15 +2428,15 @@ bch2_btree_roots_to_journal_entries(struct bch_fs *c,mutex_lock (&c->btree_root_lock); -for (i =0 ; i <BTREE_ID_NR ; i++) -if (c->btree_roots[i].alive && !test_bit (i, &have)) { -journal_entry_set (end, - BCH_JSET_ENTRY_btree_root, - i, c->btree_roots[i].level, - &c->btree_roots[i].key, - c->btree_roots[i].key.k.u64s); +for (i =0 ; i <btree_id_nr_alive (c); i++) { +struct btree_root *r =bch2_btree_id_root (c, i); + +if (r->alive && !test_bit (i, &have)) { +journal_entry_set (end, BCH_JSET_ENTRY_btree_root, + i, r->level, &r->key, r->key.k.u64s); end =vstruct_next (end); } + }mutex_unlock (&c->btree_root_lock);
fs/bcachefs/move.c +8 -2
@@ -632,7 +632,7 @@ int bch2_move_data(struct bch_fs *c,bch2_moving_ctxt_init (&ctxt, c, rate, stats, wp, wait_on_copygc);for (id = start_btree_id; - id <=min_t (unsigned, end_btree_id,BTREE_ID_NR -1 ); + id <=min_t (unsigned, end_btree_id,btree_id_nr_alive (c) -1 ); id++) { stats->btree_id = id;
@@ -640,6 +640,9 @@ int bch2_move_data(struct bch_fs *c, id != BTREE_ID_reflink)continue ; +if (!bch2_btree_id_root (c, id)->b) +continue ; + ret = __bch2_move_data(&ctxt, id == start_btree_id ? start_pos : POS_MIN, id == end_btree_id ? end_pos : POS_MAX,
@@ -861,10 +864,13 @@ static int bch2_move_btree(struct bch_fs *c, stats->data_type = BCH_DATA_btree;for (id = start_btree_id; - id <=min_t (unsigned, end_btree_id,BTREE_ID_NR -1 ); + id <=min_t (unsigned, end_btree_id,btree_id_nr_alive (c) -1 ); id++) { stats->btree_id = id; +if (!bch2_btree_id_root (c, id)->b) +continue ; +bch2_trans_node_iter_init (&trans, &iter, id,POS_MIN ,0 ,0 ,BTREE_ITER_PREFETCH ); retry:
fs/bcachefs/recovery.c +8 -8
@@ -702,13 +702,13 @@ static int journal_replay_entry_early(struct bch_fs *c,case BCH_JSET_ENTRY_btree_root: {struct btree_root *r; -if (entry->btree_id >=BTREE_ID_NR ) { -bch_err (c,"filesystem has unknown btree type %u" , - entry->btree_id); -return -EINVAL ; +while (entry->btree_id >= c->btree_roots_extra.nr +BTREE_ID_NR ) { + ret =darray_push (&c->btree_roots_extra, (struct btree_root ) {NULL }); +if (ret) +return ret; } - r = &c->btree_roots[entry->btree_id]; + r =bch2_btree_id_root (c, entry->btree_id);if (entry->u64s) { r->level = entry->level;
@@ -980,8 +980,8 @@ static int read_btree_roots(struct bch_fs *c)unsigned i;int ret =0 ; -for (i =0 ; i <BTREE_ID_NR ; i++) { -struct btree_root *r = &c->btree_roots[i]; +for (i =0 ; i <btree_id_nr_alive (c); i++) { +struct btree_root *r =bch2_btree_id_root (c, i);if (!r->alive)continue ;
@@ -1014,7 +1014,7 @@ static int read_btree_roots(struct bch_fs *c) }for (i =0 ; i <BTREE_ID_NR ; i++) { -struct btree_root *r = &c->btree_roots[i]; +struct btree_root *r =bch2_btree_id_root (c, i);if (!r->b) { r->alive = false;
fs/bcachefs/super.c +1 -0
@@ -486,6 +486,7 @@ static void __bch2_fs_free(struct bch_fs *c)for_each_possible_cpu (cpu)kfree (per_cpu_ptr (c->btree_paths_bufs, cpu)->path); +darray_exit (&c->btree_roots_extra);free_percpu (c->btree_paths_bufs);free_percpu (c->pcpu);mempool_exit (&c->large_bkey_pool);
Signature · 1 line
2.40.1
PATCH 2/10 bcachefs: Allow for unknown key types
In reply to Kent OverstreetThis adds a new helper for lookups bkey_ops for a given key type, which returns a null bkey_ops for unknown key types; various bkey_ops users are tweaked as well to handle unknown key types.
--- fs/bcachefs/bkey_methods.c | 38 +++++++++++++++------------------ fs/bcachefs/bkey_methods.h | 12 +++++++++-- fs/bcachefs/btree_update_leaf.c | 10 +++++---- 3 files changed, 33 insertions(+), 27 deletions(-)
fs/bcachefs/bkey_methods.c +17 -21
@@ -118,17 +118,14 @@ const struct bkey_ops bch2_bkey_ops[] = {#undef x }; +const struct bkey_ops bch2_bkey_null_ops = { + .min_val_size =U8_MAX , +}; +int bch2_bkey_val_invalid (struct bch_fs *c,struct bkey_s_c k,unsigned flags,struct printbuf *err) { -const struct bkey_ops *ops; - -if (k.k->type >=KEY_TYPE_MAX ) { -prt_printf (err,"invalid type (%u >= %u)" , k.k->type,KEY_TYPE_MAX ); -return -BCH_ERR_invalid_bkey; - } - - ops = &bch2_bkey_ops[k.k->type]; +const struct bkey_ops *ops =bch2_bkey_type_ops (k.k->type);if (bkey_val_bytes (k.k) < ops->min_val_size) { prt_printf(err, "bad val size (%zu < %u)",
@@ -136,6 +133,9 @@ int bch2_bkey_val_invalid(struct bch_fs *c, struct bkey_s_c k,return -BCH_ERR_invalid_bkey; } +if (!ops->key_invalid) +return 0 ; +return ops->key_invalid (c, k, flags, err); }
@@ -340,14 +340,10 @@ void bch2_bkey_to_text(struct printbuf *out, const struct bkey *k)void bch2_val_to_text (struct printbuf *out,struct bch_fs *c,struct bkey_s_c k) { -if (k.k->type <KEY_TYPE_MAX ) { -const struct bkey_ops *ops = &bch2_bkey_ops[k.k->type]; +const struct bkey_ops *ops =bch2_bkey_type_ops (k.k->type); -if (likely (ops->val_to_text)) - ops->val_to_text (out, c, k); - }else { -prt_printf (out,"(invalid type %u)" , k.k->type); - } +if (likely (ops->val_to_text)) + ops->val_to_text (out, c, k); } void bch2_bkey_val_to_text(struct printbuf *out, struct bch_fs *c,
@@ -363,7 +359,7 @@ void bch2_bkey_val_to_text(struct printbuf *out, struct bch_fs *c,void bch2_bkey_swab_val (struct bkey_s k) { -const struct bkey_ops *ops = &bch2_bkey_ops[k.k->type]; +const struct bkey_ops *ops =bch2_bkey_type_ops (k.k->type);if (ops->swab) ops->swab (k);
@@ -371,7 +367,7 @@ void bch2_bkey_swab_val(struct bkey_s k)bool bch2_bkey_normalize (struct bch_fs *c,struct bkey_s k) { -const struct bkey_ops *ops = &bch2_bkey_ops[k.k->type]; +const struct bkey_ops *ops =bch2_bkey_type_ops (k.k->type);return ops->key_normalize ? ops->key_normalize(c, k)
@@ -380,11 +376,11 @@ bool bch2_bkey_normalize(struct bch_fs *c, struct bkey_s k)bool bch2_bkey_merge (struct bch_fs *c,struct bkey_s l,struct bkey_s_c r) { -const struct bkey_ops *ops = &bch2_bkey_ops[l.k->type]; +const struct bkey_ops *ops =bch2_bkey_type_ops (l.k->type); -return bch2_bkey_maybe_mergable (l.k, r.k) && +return ops->key_merge && +bch2_bkey_maybe_mergable (l.k, r.k) && (u64 ) l.k->size + r.k->size <=KEY_SIZE_MAX && - bch2_bkey_ops[l.k->type].key_merge && !bch2_key_merging_disabled && ops->key_merge (c, l, r); }
@@ -509,7 +505,7 @@ void __bch2_bkey_compat(unsigned level, enum btree_id btree_id,if (big_endian !=CPU_BIG_ENDIAN )bch2_bkey_swab_val (u); - ops = &bch2_bkey_ops[k->type]; + ops =bch2_bkey_type_ops (k->type);if (ops->compat) ops->compat (btree_id, version, big_endian, write, u);
fs/bcachefs/bkey_methods.h +10 -2
@@ -11,6 +11,7 @@ struct bkey;enum btree_node_type ;extern const char *const bch2_bkey_types[]; +extern const struct bkey_ops bch2_bkey_null_ops;/* * key_invalid: checks validity of @k, returns 0 if good or -EINVAL if bad. If
@@ -41,6 +42,13 @@ struct bkey_ops {extern const struct bkey_ops bch2_bkey_ops[]; +static inline const struct bkey_ops *bch2_bkey_type_ops (enum bch_bkey_type type) +{ +return likely (type <KEY_TYPE_MAX ) + ? &bch2_bkey_ops[type] + : &bch2_bkey_null_ops; +} +#define BKEY_INVALID_FROM_JOURNAL (1 << 1)int bch2_bkey_val_invalid (struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *);
@@ -75,7 +83,7 @@ static inline int bch2_mark_key(struct btree_trans *trans, struct bkey_s_c old, struct bkey_s_c new, unsigned flags) { - const struct bkey_ops *ops = &bch2_bkey_ops[old.k->type ?: new.k->type]; + const struct bkey_ops *ops = bch2_bkey_type_ops(old.k->type ?: new.k->type); return ops->atomic_trigger ? ops->atomic_trigger(trans, btree, level, old, new, flags)
@@ -125,7 +133,7 @@ static inline int bch2_trans_mark_key(struct btree_trans *trans, struct bkey_s_c old, struct bkey_i *new, unsigned flags) { - const struct bkey_ops *ops = &bch2_bkey_ops[old.k->type ?: new->k.type]; + const struct bkey_ops *ops = bch2_bkey_type_ops(old.k->type ?: new->k.type); return ops->trans_trigger ? ops->trans_trigger(trans, btree_id, level, old, new, flags)
fs/bcachefs/btree_update_leaf.c +6 -4
@@ -407,6 +407,8 @@ static int run_one_mem_trigger(struct btree_trans *trans, {struct bkey_s_c old = { &i->old_k, i->old_v };struct bkey_i *new = i->k; +const struct bkey_ops *old_ops =bch2_bkey_type_ops (old.k->type); +const struct bkey_ops *new_ops =bch2_bkey_type_ops (i->k->k.type);int ret;verify_update_old_key (trans, i);
@@ -417,8 +419,7 @@ static int run_one_mem_trigger(struct btree_trans *trans,if (!btree_node_type_needs_gc (i->btree_id))return 0 ; -if (bch2_bkey_ops[old.k->type].atomic_trigger == - bch2_bkey_ops[i->k->k.type].atomic_trigger && +if (old_ops->atomic_trigger == new_ops->atomic_trigger && ((1U << old.k->type) &BTREE_TRIGGER_WANTS_OLD_AND_NEW )) { ret = bch2_mark_key(trans, i->btree_id, i->level, old, bkey_i_to_s_c(new),
@@ -450,6 +451,8 @@ static int run_one_trans_trigger(struct btree_trans *trans, struct btree_insert_*/ struct bkey old_k = i->old_k;struct bkey_s_c old = { &old_k, i->old_v }; +const struct bkey_ops *old_ops =bch2_bkey_type_ops (old.k->type); +const struct bkey_ops *new_ops =bch2_bkey_type_ops (i->k->k.type);verify_update_old_key (trans, i);
@@ -459,8 +462,7 @@ static int run_one_trans_trigger(struct btree_trans *trans, struct btree_insert_if (!i->insert_trigger_run && !i->overwrite_trigger_run && - bch2_bkey_ops[old.k->type].trans_trigger == - bch2_bkey_ops[i->k->k.type].trans_trigger && + old_ops->trans_trigger == new_ops->trans_trigger && ((1U << old.k->type) &BTREE_TRIGGER_WANTS_OLD_AND_NEW )) { i->overwrite_trigger_run = true; i->insert_trigger_run = true;
Signature · 1 line
2.40.1
PATCH 3/10 bcachefs: Refactor bch_sb_field_ops handling
In reply to Kent OverstreetThis changes bch_sb_field_ops lookup to match how bkey_ops now works; for an unknown field type we return an empty ops struct.
--- fs/bcachefs/super-io.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
fs/bcachefs/super-io.c +18 -11
@@ -1405,21 +1405,29 @@ static const struct bch_sb_field_ops *bch2_sb_field_ops[] = {#undef x }; +static const struct bch_sb_field_ops bch2_sb_field_null_ops = { +NULL +}; + +static const struct bch_sb_field_ops *bch2_sb_field_type_ops (unsigned type) +{ +return likely (type <ARRAY_SIZE (bch2_sb_field_ops)) + ? bch2_sb_field_ops[type] + : &bch2_sb_field_null_ops; +} +static int bch2_sb_field_validate (struct bch_sb *sb,struct bch_sb_field *f,struct printbuf *err) {unsigned type =le32_to_cpu (f->type);struct printbuf field_err =PRINTBUF ; +const struct bch_sb_field_ops *ops =bch2_sb_field_type_ops (type);int ret; -if (type >=BCH_SB_FIELD_NR ) -return 0 ; - - ret = bch2_sb_field_ops[type]->validate (sb, f, &field_err); + ret = ops->validate ? ops->validate (sb, f, &field_err) :0 ;if (ret) {prt_printf (err,"Invalid superblock section %s: %s" , - bch2_sb_fields[type], - field_err.buf); + bch2_sb_fields[type], field_err.buf);prt_newline (err);bch2_sb_field_to_text (err, sb, f); }
@@ -1432,13 +1440,12 @@ void bch2_sb_field_to_text(struct printbuf *out, struct bch_sb *sb, struct bch_sb_field *f) {unsigned type =le32_to_cpu (f->type); -const struct bch_sb_field_ops *ops = type <BCH_SB_FIELD_NR - ? bch2_sb_field_ops[type] :NULL ; +const struct bch_sb_field_ops *ops =bch2_sb_field_type_ops (type);if (!out->nr_tabstops)printbuf_tabstop_push (out,32 ); -if (ops) +if (type <BCH_SB_FIELD_NR )prt_printf (out,"%s" , bch2_sb_fields[type]);else prt_printf (out,"(unknown field %u)" , type);
@@ -1446,9 +1453,9 @@ void bch2_sb_field_to_text(struct printbuf *out, struct bch_sb *sb,prt_printf (out," (size %zu):" ,vstruct_bytes (f));prt_newline (out); -if (ops && ops->to_text) { +if (ops->to_text) {printbuf_indent_add (out,2 ); - bch2_sb_field_ops[type]->to_text (out, sb, f); + ops->to_text (out, sb, f);printbuf_indent_sub (out,2 ); } }
Signature · 1 line
2.40.1
PATCH 4/10 bcachefs: Change check for invalid key types
In reply to Kent OverstreetAs part of the forward compatibility patch series, we need to allow for new key types without complaining loudly when running an old version.
This patch changes the flags parameter of bkey_invalid to an enum, and adds a new flag to indicate we're being called from the transaction commit path.
--- fs/bcachefs/alloc_background.c | 14 +++++++---- fs/bcachefs/alloc_background.h | 17 +++++++++---- fs/bcachefs/backpointers.c | 3 ++- fs/bcachefs/backpointers.h | 2 +- fs/bcachefs/bkey_methods.c | 43 ++++++++++++++++++--------------- fs/bcachefs/bkey_methods.h | 10 +++++--- fs/bcachefs/btree_update_leaf.c | 7 ++++-- fs/bcachefs/dirent.c | 3 ++- fs/bcachefs/dirent.h | 4 ++- fs/bcachefs/ec.c | 3 ++- fs/bcachefs/ec.h | 4 ++- fs/bcachefs/extents.c | 12 ++++++--- fs/bcachefs/extents.h | 9 ++++--- fs/bcachefs/inode.c | 12 ++++++--- fs/bcachefs/inode.h | 12 ++++++--- fs/bcachefs/journal_io.c | 3 ++- fs/bcachefs/lru.c | 3 ++- fs/bcachefs/lru.h | 3 ++- fs/bcachefs/quota.c | 3 ++- fs/bcachefs/quota.h | 4 ++- fs/bcachefs/reflink.c | 9 ++++--- fs/bcachefs/reflink.h | 8 +++--- fs/bcachefs/subvolume.c | 6 +++-- fs/bcachefs/subvolume.h | 6 +++-- fs/bcachefs/xattr.c | 3 ++- fs/bcachefs/xattr.h | 3 ++- 26 files changed, 133 insertions(+), 73 deletions(-)
fs/bcachefs/alloc_background.c +9 -5
@@ -223,7 +223,8 @@ static unsigned bch_alloc_v1_val_u64s(const struct bch_alloc *a) }int bch2_alloc_v1_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_alloc a =bkey_s_c_to_alloc (k);
@@ -238,7 +239,8 @@ int bch2_alloc_v1_invalid(const struct bch_fs *c, struct bkey_s_c k, }int bch2_alloc_v2_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_alloc_unpacked u;
@@ -251,7 +253,8 @@ int bch2_alloc_v2_invalid(const struct bch_fs *c, struct bkey_s_c k, }int bch2_alloc_v3_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_alloc_unpacked u;
@@ -282,7 +285,7 @@ int bch2_alloc_v4_invalid(const struct bch_fs *c, struct bkey_s_c k, }if (rw ==WRITE && - !(flags &BKEY_INVALID_FROM_JOURNAL ) && + !(flags &BKEY_INVALID_JOURNAL ) &&test_bit (BCH_FS_CHECK_BACKPOINTERS_DONE , &c->flags)) {unsigned i, bp_len =0 ;
@@ -605,7 +608,8 @@ static unsigned alloc_gen(struct bkey_s_c k, unsigned offset) }int bch2_bucket_gens_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (bkey_val_bytes (k.k) !=sizeof (struct bch_bucket_gens )) { prt_printf(err, "bad val size (%lu != %zu)",
fs/bcachefs/alloc_background.h +12 -5
@@ -8,6 +8,8 @@#include "debug.h" #include "super.h" +enum bkey_invalid_flags ; +/* How out of date a pointer gen is allowed to be: */ #define BUCKET_GC_GEN_MAX 96U
@@ -147,10 +149,14 @@ struct bkey_i_alloc_v4 *bch2_alloc_to_v4_mut(struct btree_trans *, struct bkey_sint bch2_bucket_io_time_reset (struct btree_trans *,unsigned ,size_t ,int ); -int bch2_alloc_v1_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); -int bch2_alloc_v2_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); -int bch2_alloc_v3_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); -int bch2_alloc_v4_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_alloc_v1_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *); +int bch2_alloc_v2_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *); +int bch2_alloc_v3_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *); +int bch2_alloc_v4_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_alloc_v4_swab (struct bkey_s );void bch2_alloc_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );
@@ -187,7 +193,8 @@ void bch2_alloc_to_text(struct printbuf *, struct bch_fs *, struct bkey_s_c); .min_val_size = 48, \ }) -int bch2_bucket_gens_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_bucket_gens_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_bucket_gens_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );#define bch2_bkey_ops_bucket_gens ((struct bkey_ops) { \
fs/bcachefs/backpointers.c +2 -1
@@ -38,7 +38,8 @@ static bool extent_matches_bp(struct bch_fs *c, }int bch2_backpointer_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_backpointer bp =bkey_s_c_to_backpointer (k);struct bpos bucket =bp_pos_to_bucket (c, bp.k->p);
fs/bcachefs/backpointers.h +1 -1
@@ -8,7 +8,7 @@#include "super.h" int bch2_backpointer_invalid (const struct bch_fs *,struct bkey_s_c k, -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_backpointer_to_text (struct printbuf *,const struct bch_backpointer *);void bch2_backpointer_k_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );void bch2_backpointer_swab (struct bkey_s );
fs/bcachefs/bkey_methods.c +23 -20
75 lines not on this page · raw message
fs/bcachefs/bkey_methods.h +7 -3
@@ -13,6 +13,12 @@ enum btree_node_type;extern const char *const bch2_bkey_types[];extern const struct bkey_ops bch2_bkey_null_ops; +enum bkey_invalid_flags { +BKEY_INVALID_WRITE = (1U <<0 ), +BKEY_INVALID_COMMIT = (1U <<1 ), +BKEY_INVALID_JOURNAL = (1U <<2 ), +}; +/* * key_invalid: checks validity of @k, returns 0 if good or -EINVAL if bad. If * invalid, entire key will be deleted.
@@ -22,7 +28,7 @@ extern const struct bkey_ops bch2_bkey_null_ops;*/ struct bkey_ops {int (*key_invalid)(const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err); +enum bkey_invalid_flags flags,struct printbuf *err);void (*val_to_text)(struct printbuf *,struct bch_fs *,struct bkey_s_c );void (*swab)(struct bkey_s );
@@ -49,8 +55,6 @@ static inline const struct bkey_ops *bch2_bkey_type_ops(enum bch_bkey_type type) : &bch2_bkey_null_ops; } -#define BKEY_INVALID_FROM_JOURNAL (1 << 1) -int bch2_bkey_val_invalid (struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *);int __bch2_bkey_invalid (struct bch_fs *,struct bkey_s_c ,enum btree_node_type ,unsigned ,struct printbuf *);
fs/bcachefs/btree_update_leaf.c +5 -2
@@ -856,10 +856,13 @@ static inline int do_bch2_trans_commit(struct btree_trans *trans, unsigned flagsstruct printbuf buf =PRINTBUF ;trans_for_each_update (trans, i) { -int rw = (flags &BTREE_INSERT_JOURNAL_REPLAY ) ?READ :WRITE ; +enum bkey_invalid_flags invalid_flags =0 ; + +if (!(flags &BTREE_INSERT_JOURNAL_REPLAY )) + invalid_flags |=BKEY_INVALID_WRITE |BKEY_INVALID_COMMIT ;if (unlikely (bch2_bkey_invalid (c,bkey_i_to_s_c (i->k), - i->bkey_type, rw, &buf))) + i->bkey_type, invalid_flags, &buf)))return bch2_trans_commit_bkey_invalid (trans, flags, i, &buf);btree_insert_entry_checks (trans, i); }
fs/bcachefs/dirent.c +2 -1
@@ -85,7 +85,8 @@ const struct bch_hash_desc bch2_dirent_hash_desc = { };int bch2_dirent_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_dirent d =bkey_s_c_to_dirent (k);unsigned len;
fs/bcachefs/dirent.h +3 -1
@@ -4,9 +4,11 @@#include "str_hash.h" +enum bkey_invalid_flags ;extern const struct bch_hash_desc bch2_dirent_hash_desc; -int bch2_dirent_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_dirent_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_dirent_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );#define bch2_bkey_ops_dirent ((struct bkey_ops) { \
fs/bcachefs/ec.c +2 -1
@@ -105,7 +105,8 @@ struct ec_bio {/* Stripes btree keys: */ int bch2_stripe_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {const struct bch_stripe *s =bkey_s_c_to_stripe (k).v;
fs/bcachefs/ec.h +3 -1
@@ -6,8 +6,10 @@#include "buckets_types.h" #include "extents_types.h" +enum bkey_invalid_flags ; +int bch2_stripe_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_stripe_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );
fs/bcachefs/extents.c +8 -4
@@ -163,7 +163,8 @@ int bch2_bkey_pick_read_device(struct bch_fs *c, struct bkey_s_c k,/* KEY_TYPE_btree_ptr: */ int bch2_btree_ptr_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (bkey_val_u64s (k.k) >BCH_REPLICAS_MAX ) { prt_printf(err, "value too big (%zu > %u)",
@@ -181,7 +182,8 @@ void bch2_btree_ptr_to_text(struct printbuf *out, struct bch_fs *c, }int bch2_btree_ptr_v2_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (bkey_val_u64s (k.k) > BKEY_BTREE_PTR_VAL_U64s_MAX) { prt_printf(err, "value too big (%zu > %zu)",
@@ -371,7 +373,8 @@ bool bch2_extent_merge(struct bch_fs *c, struct bkey_s l, struct bkey_s_c r)/* KEY_TYPE_reservation: */ int bch2_reservation_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_reservation r =bkey_s_c_to_reservation (k);
@@ -1103,7 +1106,8 @@ static int extent_ptr_invalid(const struct bch_fs *c, }int bch2_bkey_ptrs_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_ptrs_c ptrs =bch2_bkey_ptrs_c (k);const union bch_extent_entry *entry;
fs/bcachefs/extents.h +6 -3
@@ -8,6 +8,7 @@struct bch_fs ;struct btree_trans ; +enum bkey_invalid_flags ;/* extent entries: */
@@ -382,11 +383,13 @@ int bch2_bkey_pick_read_device(struct bch_fs *, struct bkey_s_c,/* KEY_TYPE_btree_ptr: */ -int bch2_btree_ptr_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_btree_ptr_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_btree_ptr_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c ); -int bch2_btree_ptr_v2_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_btree_ptr_v2_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_btree_ptr_v2_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );void bch2_btree_ptr_v2_compat (enum btree_id ,unsigned ,unsigned ,int ,struct bkey_s );
@@ -426,7 +429,7 @@ bool bch2_extent_merge(struct bch_fs *, struct bkey_s, struct bkey_s_c);/* KEY_TYPE_reservation: */ int bch2_reservation_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_reservation_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );bool bch2_reservation_merge (struct bch_fs *,struct bkey_s ,struct bkey_s_c );
fs/bcachefs/inode.c +8 -4
@@ -432,7 +432,8 @@ static int __bch2_inode_invalid(struct bkey_s_c k, struct printbuf *err) }int bch2_inode_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_inode inode =bkey_s_c_to_inode (k);
@@ -446,7 +447,8 @@ int bch2_inode_invalid(const struct bch_fs *c, struct bkey_s_c k, }int bch2_inode_v2_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_inode_v2 inode =bkey_s_c_to_inode_v2 (k);
@@ -460,7 +462,8 @@ int bch2_inode_v2_invalid(const struct bch_fs *c, struct bkey_s_c k, }int bch2_inode_v3_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_inode_v3 inode =bkey_s_c_to_inode_v3 (k);
@@ -517,7 +520,8 @@ void bch2_inode_to_text(struct printbuf *out, struct bch_fs *c, struct bkey_s_c }int bch2_inode_generation_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (k.k->p.inode) {prt_printf (err,"nonzero k.p.inode" );
fs/bcachefs/inode.h +8 -4
@@ -5,11 +5,15 @@#include "bkey.h" #include "opts.h" +enum bkey_invalid_flags ;extern const char *const bch2_inode_opts[]; -int bch2_inode_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); -int bch2_inode_v2_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); -int bch2_inode_v3_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_inode_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *); +int bch2_inode_v2_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *); +int bch2_inode_v3_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_inode_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );#define bch2_bkey_ops_inode ((struct bkey_ops) { \
@@ -44,7 +48,7 @@ static inline bool bkey_is_inode(const struct bkey *k) }int bch2_inode_generation_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_inode_generation_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );#define bch2_bkey_ops_inode_generation ((struct bkey_ops) { \
fs/bcachefs/journal_io.c +2 -1
@@ -340,7 +340,8 @@ static int journal_entry_btree_keys_validate(struct bch_fs *c,int ret =journal_validate_key (c, jset, entry, entry->level, entry->btree_id, - k, version, big_endian, write|BKEY_INVALID_FROM_JOURNAL ); + k, version, big_endian, + write|BKEY_INVALID_JOURNAL );if (ret ==FSCK_DELETED_KEY )continue ;
fs/bcachefs/lru.c +2 -1
@@ -11,7 +11,8 @@/* KEY_TYPE_lru is obsolete: */ int bch2_lru_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (!lru_pos_time (k.k->p)) {prt_printf (err,"lru entry at time=0" );
fs/bcachefs/lru.h +2 -1
@@ -43,7 +43,8 @@ static inline enum bch_lru_type lru_type(struct bkey_s_c l)return BCH_LRU_read; } -int bch2_lru_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_lru_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_lru_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );void bch2_lru_pos_to_text (struct printbuf *,struct bpos );
fs/bcachefs/quota.c +2 -1
@@ -60,7 +60,8 @@ const struct bch_sb_field_ops bch_sb_field_ops_quota = { };int bch2_quota_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (k.k->p.inode >=QTYP_NR ) { prt_printf(err, "invalid quota type (%llu >= %u)",
fs/bcachefs/quota.h +3 -1
@@ -5,9 +5,11 @@#include "inode.h" #include "quota_types.h" +enum bkey_invalid_flags ;extern const struct bch_sb_field_ops bch_sb_field_ops_quota; -int bch2_quota_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_quota_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_quota_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );#define bch2_bkey_ops_quota ((struct bkey_ops) { \
fs/bcachefs/reflink.c +6 -3
@@ -26,7 +26,8 @@ static inline unsigned bkey_type_to_indirect(const struct bkey *k)/* reflink pointers */ int bch2_reflink_p_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_reflink_p p =bkey_s_c_to_reflink_p (k);
@@ -72,7 +73,8 @@ bool bch2_reflink_p_merge(struct bch_fs *c, struct bkey_s _l, struct bkey_s_c _r/* indirect extents */ int bch2_reflink_v_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {return bch2_bkey_ptrs_invalid (c, k, flags, err); }
@@ -117,7 +119,8 @@ int bch2_trans_mark_reflink_v(struct btree_trans *trans,/* indirect inline data */ int bch2_indirect_inline_data_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {return 0 ; }
fs/bcachefs/reflink.h +5 -3
@@ -2,8 +2,10 @@#ifndef _BCACHEFS_REFLINK_H#define _BCACHEFS_REFLINK_H +enum bkey_invalid_flags ; +int bch2_reflink_p_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_reflink_p_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );bool bch2_reflink_p_merge (struct bch_fs *,struct bkey_s ,struct bkey_s_c );
@@ -18,7 +20,7 @@ bool bch2_reflink_p_merge(struct bch_fs *, struct bkey_s, struct bkey_s_c); })int bch2_reflink_v_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_reflink_v_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c ); int bch2_trans_mark_reflink_v(struct btree_trans *, enum btree_id, unsigned,
@@ -34,7 +36,7 @@ int bch2_trans_mark_reflink_v(struct btree_trans *, enum btree_id, unsigned, })int bch2_indirect_inline_data_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);void bch2_indirect_inline_data_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c ); int bch2_trans_mark_indirect_inline_data(struct btree_trans *,
fs/bcachefs/subvolume.c +4 -2
@@ -23,7 +23,8 @@ void bch2_snapshot_tree_to_text(struct printbuf *out, struct bch_fs *c, }int bch2_snapshot_tree_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {if (bkey_gt (k.k->p,POS (0 ,U32_MAX )) ||bkey_lt (k.k->p,POS (0 ,1 ))) {
@@ -97,7 +98,8 @@ void bch2_snapshot_to_text(struct printbuf *out, struct bch_fs *c, }int bch2_snapshot_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {struct bkey_s_c_snapshot s;u32 i, id;
fs/bcachefs/subvolume.h +4 -2
@@ -5,9 +5,11 @@#include "darray.h" #include "subvolume_types.h" +enum bkey_invalid_flags ; +void bch2_snapshot_tree_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );int bch2_snapshot_tree_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);#define bch2_bkey_ops_snapshot_tree ((struct bkey_ops) { \ .key_invalid = bch2_snapshot_tree_invalid, \
@@ -19,7 +21,7 @@ int bch2_snapshot_tree_lookup(struct btree_trans *, u32, struct bch_snapshot_trevoid bch2_snapshot_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );int bch2_snapshot_invalid (const struct bch_fs *,struct bkey_s_c , -unsigned ,struct printbuf *); +enum bkey_invalid_flags ,struct printbuf *);int bch2_mark_snapshot (struct btree_trans *,enum btree_id ,unsigned ,struct bkey_s_c ,struct bkey_s_c ,unsigned );
fs/bcachefs/xattr.c +2 -1
@@ -70,7 +70,8 @@ const struct bch_hash_desc bch2_xattr_hash_desc = { };int bch2_xattr_invalid (const struct bch_fs *c,struct bkey_s_c k, -unsigned flags,struct printbuf *err) +enum bkey_invalid_flags flags, +struct printbuf *err) {const struct xattr_handler *handler;struct bkey_s_c_xattr xattr =bkey_s_c_to_xattr (k);
fs/bcachefs/xattr.h +2 -1
@@ -6,7 +6,8 @@extern const struct bch_hash_desc bch2_xattr_hash_desc; -int bch2_xattr_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *); +int bch2_xattr_invalid (const struct bch_fs *,struct bkey_s_c , +enum bkey_invalid_flags ,struct printbuf *);void bch2_xattr_to_text (struct printbuf *,struct bch_fs *,struct bkey_s_c );#define bch2_bkey_ops_xattr ((struct bkey_ops) { \
Signature · 1 line
2.40.1
PATCH 5/10 bcachefs: BCH_SB_VERSION_UPGRADE_COMPLETE()
In reply to Kent OverstreetVersion upgrades are not atomic operations: when we do a version upgrade we need to update the superblock before we start using new features, and then when the upgrade completes we need to update the superblock again. This adds a new superblock field so we can detect and handle incomplete version upgrades.
--- fs/bcachefs/alloc_background.c | 3 +- fs/bcachefs/bcachefs.h | 7 +++++ fs/bcachefs/bcachefs_format.h | 5 ++++ fs/bcachefs/recovery.c | 54 +++++++++++++++++++--------------- fs/bcachefs/super-io.c | 18 ++++++++++++ 5 files changed, 62 insertions(+), 25 deletions(-)
fs/bcachefs/alloc_background.c +1 -2
@@ -1232,8 +1232,7 @@ int bch2_check_alloc_hole_bucket_gens(struct btree_trans *trans,unsigned i, gens_offset, gens_end_offset;int ret; -if (c->sb.version < bcachefs_metadata_version_bucket_gens && - !c->opts.version_upgrade) +if (c->sb.version < bcachefs_metadata_version_bucket_gens)return 0 ;bch2_btree_iter_set_pos (bucket_gens_iter,alloc_gens_pos (start, &gens_offset));
fs/bcachefs/bcachefs.h +7 -0
@@ -712,6 +712,7 @@ struct bch_fs {u16 version;u16 version_min; +u16 version_upgrade_complete;u8 nr_devices;u8 clean;
@@ -1134,6 +1135,12 @@ static inline bool bch2_dev_exists2(const struct bch_fs *c, unsigned dev)return dev < c->sb.nr_devices && c->devs[dev]; } +static inline bool bch2_version_upgrading_to (const struct bch_fs *c,unsigned new_version) +{ +return c->sb.version_upgrade_complete < new_version && + c->sb.version >= new_version; +} +#define BKEY_PADDED_ONSTACK (key, pad) \ struct { struct bkey_i key; __u64 key ## _pad[pad]; }
fs/bcachefs/bcachefs_format.h +5 -0
@@ -1748,6 +1748,11 @@ LE64_BITMASK(BCH_SB_JOURNAL_TRANSACTION_NAMES,struct bch_sb, flags[4], 32, 33); LE64_BITMASK(BCH_SB_NOCOW, struct bch_sb, flags[4], 33, 34); LE64_BITMASK(BCH_SB_WRITE_BUFFER_SIZE, struct bch_sb, flags[4], 34, 54); +/* flags[4] 56-64 unused: */ + +LE64_BITMASK (BCH_SB_VERSION_UPGRADE_COMPLETE , + struct bch_sb, flags[5], 0, 16); +/* * Features: *
fs/bcachefs/recovery.c +31 -23
84 lines not on this page · raw message
fs/bcachefs/super-io.c +18 -0
@@ -445,6 +445,7 @@ static void bch2_sb_update(struct bch_fs *c) c->sb.user_uuid = src->user_uuid; c->sb.version =le16_to_cpu (src->version); c->sb.version_min =le16_to_cpu (src->version_min); + c->sb.version_upgrade_complete =BCH_SB_VERSION_UPGRADE_COMPLETE (src) ?: c->sb.version; c->sb.nr_devices = src->nr_devices; c->sb.clean =BCH_SB_CLEAN (src); c->sb.encryption_type =BCH_SB_ENCRYPTION_TYPE (src);
@@ -1185,7 +1186,19 @@ int bch2_fs_mark_dirty(struct bch_fs *c)mutex_lock (&c->sb_lock);SET_BCH_SB_CLEAN (c->disk_sb.sb, false); + +if (BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb) > bcachefs_metadata_version_current) +SET_BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb, bcachefs_metadata_version_current); + +if (c->opts.version_upgrade || + c->sb.version > bcachefs_metadata_version_current) + c->disk_sb.sb->version =cpu_to_le16 (bcachefs_metadata_version_current); + +if (c->opts.version_upgrade) + c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALL ); + c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALWAYS ); + c->disk_sb.sb->compat[0 ] &=cpu_to_le64 ((1ULL <<BCH_COMPAT_NR ) -1 ); ret =bch2_write_super (c);mutex_unlock (&c->sb_lock);
@@ -1529,6 +1542,11 @@ void bch2_sb_to_text(struct printbuf *out, struct bch_sb *sb,bch2_version_to_text (out,le16_to_cpu (sb->version));prt_newline (out); +prt_str (out,"Version upgrade complete:" ); +prt_tab (out); +bch2_version_to_text (out,BCH_SB_VERSION_UPGRADE_COMPLETE (sb)); +prt_newline (out); +prt_printf (out,"Oldest version on disk:" );prt_tab (out);bch2_version_to_text (out,le16_to_cpu (sb->version_min));
Signature · 1 line
2.40.1
Re: [PATCH 05/10] bcachefs: BCH_SB_VERSION_UPGRADE_COMPLETE()
Quoted 42 lines · Kent Overstreet
Version upgrades are not atomic operations: when we do a version upgrade we need to update the superblock before we start using new features, and then when the upgrade completes we need to update the superblock again. This adds a new superblock field so we can detect and handle incomplete version upgrades.
Signed-off-by Kent Overstreet <[email protected]>--- fs/bcachefs/alloc_background.c | 3 +- fs/bcachefs/bcachefs.h | 7 +++++ fs/bcachefs/bcachefs_format.h | 5 ++++ fs/bcachefs/recovery.c | 54 +++++++++++++++++++--------------- fs/bcachefs/super-io.c | 18 ++++++++++++ 5 files changed, 62 insertions(+), 25 deletions(-)fs/bcachefs/alloc_background.c +1
-2@@ -1232,8 +1232,7 @@ int bch2_check_alloc_hole_bucket_gens(struct btree_trans *trans,unsigned i, gens_offset, gens_end_offset;int ret; -if (c->sb.version < bcachefs_metadata_version_bucket_gens && - !c->opts.version_upgrade) +if (c->sb.version < bcachefs_metadata_version_bucket_gens)return 0 ;bch2_btree_iter_set_pos (bucket_gens_iter,alloc_gens_pos (start, &gens_offset));fs/bcachefs/bcachefs.h +1
-0@@ -712,6 +712,7 @@ struct bch_fs {u16 version;u16 version_min; +u16 version_upgrade_complete;u8 nr_devices;u8 clean;
fs/bcachefs/bcachefs.h static inline bool bch2_dev_exists2(const struct bch_fs *c, unsigned dev) · line 1137
Quoted 3 lines
fs/bcachefs/bcachefs.h +0
-0@@ -1134,6 +1135,12 @@ static inline bool bch2_dev_exists2(const struct bch_fs *c, unsigned dev)return dev < c->sb.nr_devices && c->devs[dev]; }
So the commit log implies that when an upgrade starts, sb.version refers to the target version, and now version_upgrade_complete is updated to match upon completion of the upgrade. Am I following that correctly?
If so, I'm still missing something.. where exactly is sb.version updated to the target version?
Brian
Quoted 165 lines
fs/bcachefs/bcachefs.h +6
-0+static inline bool bch2_version_upgrading_to (const struct bch_fs *c,unsigned new_version) +{ +return c->sb.version_upgrade_complete < new_version && + c->sb.version >= new_version; +} +#define BKEY_PADDED_ONSTACK (key, pad) \ struct { struct bkey_i key; __u64 key ## _pad[pad]; }fs/bcachefs/bcachefs_format.h +5
-0@@ -1748,6 +1748,11 @@ LE64_BITMASK(BCH_SB_JOURNAL_TRANSACTION_NAMES,struct bch_sb, flags[4], 32, 33); LE64_BITMASK(BCH_SB_NOCOW, struct bch_sb, flags[4], 33, 34); LE64_BITMASK(BCH_SB_WRITE_BUFFER_SIZE, struct bch_sb, flags[4], 34, 54); +/* flags[4] 56-64 unused: */ + +LE64_BITMASK (BCH_SB_VERSION_UPGRADE_COMPLETE , + struct bch_sb, flags[5], 0, 16); +/* * Features: * fs/bcachefs/recovery.c +31
-23@@ -1107,6 +1107,31 @@ static int bch2_fs_upgrade_for_subvolumes(struct bch_fs *c)return ret; } +static void check_version_upgrade (struct bch_fs *c) +{ +unsigned version = c->sb.version_upgrade_complete ?: c->sb.version; + +if (version < bcachefs_metadata_required_upgrade_below) { +struct printbuf buf =PRINTBUF ; + +if (version != c->sb.version) +prt_str (&buf,"version upgrade incomplete:\n" ); + +prt_str (&buf,"version " ); +bch2_version_to_text (&buf, version); +prt_str (&buf," prior to " ); +bch2_version_to_text (&buf, bcachefs_metadata_required_upgrade_below); +prt_str (&buf,", upgrade and fsck required" ); + +bch_info (c,"%s" , buf.buf); +printbuf_exit (&buf); + + c->opts.version_upgrade = true; + c->opts.fsck = true; + c->opts.fix_errors =FSCK_OPT_YES ; + } +} +int bch2_fs_recovery (struct bch_fs *c) {struct bch_sb_field_clean *clean =NULL ;@@ -1146,23 +1171,8 @@ int bch2_fs_recovery(struct bch_fs *c) goto err; } -if (!c->opts.nochanges && - c->sb.version < bcachefs_metadata_required_upgrade_below) { -struct printbuf buf =PRINTBUF ; - -prt_str (&buf,"version " ); -bch2_version_to_text (&buf, c->sb.version); -prt_str (&buf," prior to " ); -bch2_version_to_text (&buf, bcachefs_metadata_required_upgrade_below); -prt_str (&buf,", upgrade and fsck required" ); - -bch_info (c,"%s" , buf.buf); -printbuf_exit (&buf); - - c->opts.version_upgrade = true; - c->opts.fsck = true; - c->opts.fix_errors =FSCK_OPT_YES ; - } +if (!c->opts.nochanges) +check_version_upgrade (c);if (c->opts.fsck && c->opts.norecovery) {bch_err (c,"cannot select both norecovery and fsck" );@@ -1406,8 +1416,7 @@ int bch2_fs_recovery(struct bch_fs *c)if (ret) goto err; -if (c->sb.version < bcachefs_metadata_version_bucket_gens && - c->opts.version_upgrade) { +if (bch2_version_upgrading_to (c, bcachefs_metadata_version_bucket_gens)) {bch_info (c,"initializing bucket_gens" ); ret =bch2_bucket_gens_init (c);if (ret)@@ -1415,7 +1424,7 @@ int bch2_fs_recovery(struct bch_fs *c)bch_verbose (c,"bucket_gens init done" ); } -if (c->sb.version < bcachefs_metadata_version_snapshot_2) { +if (bch2_version_upgrading_to (c, bcachefs_metadata_version_snapshot_2)) { ret =bch2_fs_upgrade_for_subvolumes (c);if (ret) goto err;@@ -1443,9 +1452,8 @@ int bch2_fs_recovery(struct bch_fs *c) }mutex_lock (&c->sb_lock); -if (c->opts.version_upgrade) { - c->disk_sb.sb->version =cpu_to_le16 (bcachefs_metadata_version_current); - c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALL ); +if (BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb) != c->sb.version) { +SET_BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb, c->sb.version); write_sb = true; }fs/bcachefs/super-io.c +18
-0@@ -445,6 +445,7 @@ static void bch2_sb_update(struct bch_fs *c) c->sb.user_uuid = src->user_uuid; c->sb.version =le16_to_cpu (src->version); c->sb.version_min =le16_to_cpu (src->version_min); + c->sb.version_upgrade_complete =BCH_SB_VERSION_UPGRADE_COMPLETE (src) ?: c->sb.version; c->sb.nr_devices = src->nr_devices; c->sb.clean =BCH_SB_CLEAN (src); c->sb.encryption_type =BCH_SB_ENCRYPTION_TYPE (src);@@ -1185,7 +1186,19 @@ int bch2_fs_mark_dirty(struct bch_fs *c)mutex_lock (&c->sb_lock);SET_BCH_SB_CLEAN (c->disk_sb.sb, false); + +if (BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb) > bcachefs_metadata_version_current) +SET_BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb, bcachefs_metadata_version_current); + +if (c->opts.version_upgrade || + c->sb.version > bcachefs_metadata_version_current) + c->disk_sb.sb->version =cpu_to_le16 (bcachefs_metadata_version_current); + +if (c->opts.version_upgrade) + c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALL ); + c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALWAYS ); + c->disk_sb.sb->compat[0 ] &=cpu_to_le64 ((1ULL <<BCH_COMPAT_NR ) -1 ); ret =bch2_write_super (c);mutex_unlock (&c->sb_lock);@@ -1529,6 +1542,11 @@ void bch2_sb_to_text(struct printbuf *out, struct bch_sb *sb,bch2_version_to_text (out,le16_to_cpu (sb->version));prt_newline (out); +prt_str (out,"Version upgrade complete:" ); +prt_tab (out); +bch2_version_to_text (out,BCH_SB_VERSION_UPGRADE_COMPLETE (sb)); +prt_newline (out); +prt_printf (out,"Oldest version on disk:" );prt_tab (out);bch2_version_to_text (out,le16_to_cpu (sb->version_min));Signature · 1 line
2.40.1
Re: [PATCH 05/10] bcachefs: BCH_SB_VERSION_UPGRADE_COMPLETE()
Quoted 51 lines · Brian Foster
Quoted 42 lines · Kent Overstreet
Version upgrades are not atomic operations: when we do a version upgrade we need to update the superblock before we start using new features, and then when the upgrade completes we need to update the superblock again. This adds a new superblock field so we can detect and handle incomplete version upgrades.
Signed-off-by Kent Overstreet <[email protected]>--- fs/bcachefs/alloc_background.c | 3 +- fs/bcachefs/bcachefs.h | 7 +++++ fs/bcachefs/bcachefs_format.h | 5 ++++ fs/bcachefs/recovery.c | 54 +++++++++++++++++++--------------- fs/bcachefs/super-io.c | 18 ++++++++++++ 5 files changed, 62 insertions(+), 25 deletions(-)fs/bcachefs/alloc_background.c +1
-2@@ -1232,8 +1232,7 @@ int bch2_check_alloc_hole_bucket_gens(struct btree_trans *trans,unsigned i, gens_offset, gens_end_offset;int ret; -if (c->sb.version < bcachefs_metadata_version_bucket_gens && - !c->opts.version_upgrade) +if (c->sb.version < bcachefs_metadata_version_bucket_gens)return 0 ;bch2_btree_iter_set_pos (bucket_gens_iter,alloc_gens_pos (start, &gens_offset));fs/bcachefs/bcachefs.h +1
-0@@ -712,6 +712,7 @@ struct bch_fs {u16 version;u16 version_min; +u16 version_upgrade_complete;u8 nr_devices;u8 clean;fs/bcachefs/bcachefs.h static inline bool bch2_dev_exists2(const struct bch_fs *c, unsigned dev) · line 1137
Quoted 3 lines
fs/bcachefs/bcachefs.h +0
-0@@ -1134,6 +1135,12 @@ static inline bool bch2_dev_exists2(const struct bch_fs *c, unsigned dev)return dev < c->sb.nr_devices && c->devs[dev]; }So the commit log implies that when an upgrade starts, sb.version refers to the target version, and now
version_upgrade_completeis updated to match upon completion of the upgrade. Am I following that correctly?
Quoted 2 lines
If so, I'm still missing something.. where exactly is sb.version updated to the target version?
I don't think that was at its final destination in this patch - it's check_version_uprade() in recovery.c post major minor versioning.
PATCH 6/10 bcachefs: version_upgrade is now an enum
In reply to Kent OverstreetThe version_upgrade parameter is now an enum, not a bool, and it's persistent in the superblock:
- compatible (default): upgrade to the latest compatible version
- incompatible: upgrade to latest incompatible version
- none
Currently all upgrades are incompatible upgrades, but the next release will introduce major:minor versions.
--- fs/bcachefs/bcachefs.h | 1 + fs/bcachefs/bcachefs_format.h | 12 ++++++++++++ fs/bcachefs/opts.c | 5 +++++ fs/bcachefs/opts.h | 5 +++-- fs/bcachefs/recovery.c | 19 +++++++++++-------- fs/bcachefs/super-io.c | 6 +++--- 6 files changed, 35 insertions(+), 13 deletions(-)
fs/bcachefs/bcachefs.h +1 -0
@@ -573,6 +573,7 @@ enum {BCH_FS_INITIAL_GC_UNFIXED ,/* kill when we enumerate fsck errors */ BCH_FS_NEED_ANOTHER_GC , +BCH_FS_VERSION_UPGRADE , BCH_FS_HAVE_DELETED_SNAPSHOTS,/* errors: */
fs/bcachefs/bcachefs_format.h +12 -0
@@ -1747,6 +1747,7 @@ LE64_BITMASK(BCH_SB_JOURNAL_RECLAIM_DELAY,struct bch_sb, flags[4], 0, 32); LE64_BITMASK(BCH_SB_JOURNAL_TRANSACTION_NAMES,struct bch_sb, flags[4], 32, 33); LE64_BITMASK(BCH_SB_NOCOW, struct bch_sb, flags[4], 33, 34); LE64_BITMASK(BCH_SB_WRITE_BUFFER_SIZE, struct bch_sb, flags[4], 34, 54); +LE64_BITMASK(BCH_SB_VERSION_UPGRADE, struct bch_sb, flags[4], 54, 56);/* flags[4] 56-64 unused: */
@@ -1819,6 +1820,17 @@ enum bch_sb_compat {/* options: */ +#define BCH_VERSION_UPGRADE_OPTS () \ + x(compatible, 0) \ + x(incompatible, 1) \ + x(none, 2) + +enum bch_version_upgrade_opts { +#define x(t, n) BCH_VERSION_UPGRADE_##t = n, + BCH_VERSION_UPGRADE_OPTS() +#undef x +}; +#define BCH_REPLICAS_MAX 4U#define BCH_BKEY_PTRS_MAX 16U
fs/bcachefs/opts.c +5 -0
@@ -16,6 +16,11 @@ const char * const bch2_error_actions[] = {NULL }; +const char *const bch2_version_upgrade_opts[] = { +BCH_VERSION_UPGRADE_OPTS () + NULL +}; +const char *const bch2_sb_features[] = {BCH_SB_FEATURES () NULL
fs/bcachefs/opts.h +3 -2
@@ -9,6 +9,7 @@#include "bcachefs_format.h" extern const char *const bch2_error_actions[]; +extern const char *const bch2_version_upgrade_opts[];extern const char *const bch2_sb_features[];extern const char *const bch2_sb_compat[];extern const char *const bch2_btree_ids[];
@@ -388,8 +389,8 @@ enum opt_type { NULL, "Reconstruct alloc btree") \ x(version_upgrade, u8, \ OPT_FS|OPT_MOUNT, \ - OPT_BOOL(), \ - BCH2_NO_SB_OPT, false, \ + OPT_STR(bch2_version_upgrade_opts), \ + BCH_SB_VERSION_UPGRADE, BCH_VERSION_UPGRADE_compatible, \ NULL, "Set superblock to latest version,\n" \ "allowing any new features to be used") \ x(buckets_nouse, u8, \
fs/bcachefs/recovery.c +11 -8
@@ -1111,11 +1111,16 @@ static void check_version_upgrade(struct bch_fs *c) {unsigned version = c->sb.version_upgrade_complete ?: c->sb.version; -if (version < bcachefs_metadata_required_upgrade_below) { +if (version < bcachefs_metadata_required_upgrade_below || + (version < bcachefs_metadata_version_current && + c->opts.version_upgrade != BCH_VERSION_UPGRADE_none)) {struct printbuf buf =PRINTBUF ; -if (version != c->sb.version) -prt_str (&buf,"version upgrade incomplete:\n" ); +if (version != c->sb.version) { +prt_str (&buf,"version upgrade to " ); +bch2_version_to_text (&buf, c->sb.version); +prt_str (&buf," incomplete:\n" ); + }prt_str (&buf,"version " );bch2_version_to_text (&buf, version);
@@ -1126,9 +1131,9 @@ static void check_version_upgrade(struct bch_fs *c)bch_info (c,"%s" , buf.buf);printbuf_exit (&buf); - c->opts.version_upgrade = true; c->opts.fsck = true; c->opts.fix_errors =FSCK_OPT_YES ; +set_bit (BCH_FS_VERSION_UPGRADE , &c->flags); } }
@@ -1534,11 +1539,9 @@ int bch2_fs_initialize(struct bch_fs *c) c->disk_sb.sb->compat[0 ] |=cpu_to_le64 (1ULL << BCH_COMPAT_extents_above_btree_updates_done); c->disk_sb.sb->compat[0 ] |=cpu_to_le64 (1ULL << BCH_COMPAT_bformat_overflow_done); -if (c->sb.version < bcachefs_metadata_version_inode_v3) - c->opts.version_upgrade = true; - -if (c->opts.version_upgrade) { +if (c->opts.version_upgrade != BCH_VERSION_UPGRADE_none) { c->disk_sb.sb->version =cpu_to_le16 (bcachefs_metadata_version_current); +SET_BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb, bcachefs_metadata_version_current); c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALL );bch2_write_super (c); }
fs/bcachefs/super-io.c +3 -3
@@ -809,7 +809,7 @@ int bch2_write_super(struct bch_fs *c)closure_init_stack (cl);memset (&sb_written,0 ,sizeof (sb_written)); -if (c->opts.version_upgrade) { +if (test_bit (BCH_FS_VERSION_UPGRADE , &c->flags)) { c->disk_sb.sb->magic =BCHFS_MAGIC ; c->disk_sb.sb->layout.magic =BCHFS_MAGIC ; }
@@ -1190,11 +1190,11 @@ int bch2_fs_mark_dirty(struct bch_fs *c)if (BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb) > bcachefs_metadata_version_current)SET_BCH_SB_VERSION_UPGRADE_COMPLETE (c->disk_sb.sb, bcachefs_metadata_version_current); -if (c->opts.version_upgrade || +if (test_bit (BCH_FS_VERSION_UPGRADE , &c->flags) || c->sb.version > bcachefs_metadata_version_current) c->disk_sb.sb->version =cpu_to_le16 (bcachefs_metadata_version_current); -if (c->opts.version_upgrade) +if (test_bit (BCH_FS_VERSION_UPGRADE , &c->flags)) c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALL ); c->disk_sb.sb->features[0 ] |=cpu_to_le64 (BCH_SB_FEATURES_ALWAYS );
Signature · 1 line
2.40.1
PATCH 7/10 bcachefs: Kill bch2_bucket_gens_read()
In reply to Kent OverstreetThis folds bch2_bucket_gens_read() into bch2_alloc_read(), doing the version check there.
This is prep work for enumarating all recovery passes: we need some cleanup first to make calling all the recovery passes consistent.
--- fs/bcachefs/alloc_background.c | 100 +++++++++++++++------------------ fs/bcachefs/alloc_background.h | 1 - fs/bcachefs/recovery.c | 6 +- 3 files changed, 45 insertions(+), 62 deletions(-)
fs/bcachefs/alloc_background.c +44 -56
@@ -551,40 +551,6 @@ bch2_trans_start_alloc_update(struct btree_trans *trans, struct btree_iter *iterreturn ERR_PTR (ret); } -int bch2_alloc_read (struct bch_fs *c) -{ -struct btree_trans trans; -struct btree_iter iter; -struct bkey_s_c k; -struct bch_alloc_v4 a; -struct bch_dev *ca; -int ret; - -bch2_trans_init (&trans, c,0 ,0 ); - -for_each_btree_key (&trans, iter, BTREE_ID_alloc,POS_MIN , -BTREE_ITER_PREFETCH , k, ret) { -/* -* Not a fsck error because this is checked/repaired by -* bch2_check_alloc_key() which runs later: -*/ -if (!bch2_dev_bucket_exists (c, k.k->p)) -continue ; - - ca =bch_dev_bkey_exists (c, k.k->p.inode); - - *bucket_gen (ca, k.k->p.offset) =bch2_alloc_to_v4 (k, &a)->gen; - } -bch2_trans_iter_exit (&trans, &iter); - -bch2_trans_exit (&trans); - -if (ret) -bch_err_fn (c, ret); - -return ret; -} -static struct bpos alloc_gens_pos (struct bpos pos,unsigned *offset) { *offset = pos.offset &KEY_TYPE_BUCKET_GENS_MASK ;
@@ -692,45 +658,67 @@ int bch2_bucket_gens_init(struct bch_fs *c)return ret; } -int bch2_bucket_gens_read (struct bch_fs *c) +int bch2_alloc_read (struct bch_fs *c) {struct btree_trans trans;struct btree_iter iter;struct bkey_s_c k; -const struct bch_bucket_gens *g;struct bch_dev *ca; -u64 b;int ret; +down_read (&c->gc_lock);bch2_trans_init (&trans, c,0 ,0 ); -for_each_btree_key (&trans, iter, BTREE_ID_bucket_gens,POS_MIN , -BTREE_ITER_PREFETCH , k, ret) { -u64 start =bucket_gens_pos_to_alloc (k.k->p,0 ).offset; -u64 end =bucket_gens_pos_to_alloc (bpos_nosnap_successor (k.k->p),0 ).offset; +if (c->sb.version_upgrade_complete >= bcachefs_metadata_version_bucket_gens) { +const struct bch_bucket_gens *g; +u64 b; -if (k.k->type != KEY_TYPE_bucket_gens) -continue ; +for_each_btree_key (&trans, iter, BTREE_ID_bucket_gens,POS_MIN , +BTREE_ITER_PREFETCH , k, ret) { +u64 start =bucket_gens_pos_to_alloc (k.k->p,0 ).offset; +u64 end =bucket_gens_pos_to_alloc (bpos_nosnap_successor (k.k->p),0 ).offset; - g =bkey_s_c_to_bucket_gens (k).v; +if (k.k->type != KEY_TYPE_bucket_gens) +continue ; -/* -* Not a fsck error because this is checked/repaired by -* bch2_check_alloc_key() which runs later: -*/ -if (!bch2_dev_exists2 (c, k.k->p.inode)) -continue ; + g =bkey_s_c_to_bucket_gens (k).v; + +/* +* Not a fsck error because this is checked/repaired by +* bch2_check_alloc_key() which runs later: +*/ +if (!bch2_dev_exists2 (c, k.k->p.inode)) +continue ; - ca =bch_dev_bkey_exists (c, k.k->p.inode); + ca =bch_dev_bkey_exists (c, k.k->p.inode); + +for (b =max_t (u64, ca->mi.first_bucket, start); + b <min_t (u64, ca->mi.nbuckets, end); + b++) + *bucket_gen (ca, b) = g->gens[b &KEY_TYPE_BUCKET_GENS_MASK ]; + } +bch2_trans_iter_exit (&trans, &iter); + }else { +struct bch_alloc_v4 a; -for (b =max_t (u64, ca->mi.first_bucket, start); - b <min_t (u64, ca->mi.nbuckets, end); - b++) - *bucket_gen (ca, b) = g->gens[b &KEY_TYPE_BUCKET_GENS_MASK ]; +for_each_btree_key (&trans, iter, BTREE_ID_alloc,POS_MIN , +BTREE_ITER_PREFETCH , k, ret) { +/* +* Not a fsck error because this is checked/repaired by +* bch2_check_alloc_key() which runs later: +*/ +if (!bch2_dev_bucket_exists (c, k.k->p)) +continue ; + + ca =bch_dev_bkey_exists (c, k.k->p.inode); + + *bucket_gen (ca, k.k->p.offset) =bch2_alloc_to_v4 (k, &a)->gen; + } +bch2_trans_iter_exit (&trans, &iter); } -bch2_trans_iter_exit (&trans, &iter);bch2_trans_exit (&trans); +up_read (&c->gc_lock);if (ret)bch_err_fn (c, ret);
fs/bcachefs/alloc_background.h +0 -1
@@ -212,7 +212,6 @@ static inline bool bkey_is_alloc(const struct bkey *k) }int bch2_alloc_read (struct bch_fs *); -int bch2_bucket_gens_read (struct bch_fs *);int bch2_trans_mark_alloc (struct btree_trans *,enum btree_id ,unsigned ,struct bkey_s_c ,struct bkey_i *,unsigned );
fs/bcachefs/recovery.c +1 -5
@@ -1309,11 +1309,7 @@ int bch2_fs_recovery(struct bch_fs *c) goto err;bch_verbose (c,"starting alloc read" ); -down_read (&c->gc_lock); - ret = c->sb.version < bcachefs_metadata_version_bucket_gens - ?bch2_alloc_read (c) - :bch2_bucket_gens_read (c); -up_read (&c->gc_lock); + ret =bch2_alloc_read (c);if (ret) goto err;bch_verbose (c,"alloc read done" );
Signature · 1 line
2.40.1
PATCH 8/10 bcachefs: Stash journal replay params in bch_fs
In reply to Kent OverstreetFor the upcoming enumeration of recovery passes, we need all recovery passes to be called the same way - including journal replay.
--- fs/bcachefs/bcachefs.h | 3 +++ fs/bcachefs/recovery.c | 11 ++++++++--- 2 files changed, 11 insertions(+), 3 deletions(-)
fs/bcachefs/bcachefs.h +3 -0
@@ -993,6 +993,9 @@ struct bch_fs {/* QUOTAS */ struct bch_memquota_type quotas[QTYP_NR ]; +/* RECOVERY */ +u64 journal_replay_seq_start; +u64 journal_replay_seq_end;/* DEBUG JUNK */ struct dentry *fs_debug_dir;struct dentry *btree_debug_dir;
fs/bcachefs/recovery.c +8 -3
@@ -624,11 +624,13 @@ static int journal_sort_seq_cmp(const void *_l, const void *_r)return cmp_int (l->journal_seq, r->journal_seq); } -static int bch2_journal_replay (struct bch_fs *c,u64 start_seq,u64 end_seq) +static int bch2_journal_replay (struct bch_fs *c) {struct journal_keys *keys = &c->journal_keys;struct journal_key **keys_sorted, *k;struct journal *j = &c->journal; +u64 start_seq = c->journal_replay_seq_start; +u64 end_seq = c->journal_replay_seq_start;size_t i;int ret;
@@ -1256,6 +1258,9 @@ int bch2_fs_recovery(struct bch_fs *c) blacklist_seq = journal_seq =le64_to_cpu (clean->journal_seq) +1 ; } + c->journal_replay_seq_start = last_seq; + c->journal_replay_seq_end = blacklist_seq -1 ;; +if (c->opts.reconstruct_alloc) { c->sb.compat &= ~(1ULL << BCH_COMPAT_alloc_info);drop_alloc_keys (&c->journal_keys);
@@ -1346,7 +1351,7 @@ int bch2_fs_recovery(struct bch_fs *c)set_bit (BCH_FS_MAY_GO_RW , &c->flags);bch_info (c,"starting journal replay, %zu keys" , c->journal_keys.nr); - ret =bch2_journal_replay (c, last_seq, blacklist_seq -1 ); + ret =bch2_journal_replay (c);if (ret) goto err;if (c->opts.verbose || !c->sb.clean)
@@ -1406,7 +1411,7 @@ int bch2_fs_recovery(struct bch_fs *c)set_bit (BCH_FS_MAY_GO_RW , &c->flags);bch_verbose (c,"starting journal replay, %zu keys" , c->journal_keys.nr); - ret =bch2_journal_replay (c, last_seq, blacklist_seq -1 ); + ret =bch2_journal_replay (c);if (ret) goto err;if (c->opts.verbose || !c->sb.clean)
Signature · 1 line
2.40.1
PATCH 9/10 bcachefs: Enumerate recovery passes
In reply to Kent OverstreetRecovery and fsck have many different passes/jobs to do, which always run in the same order - but not all of them run all the time. Some are for fsck, some for unclean shutdown, some for version upgrades.
This adds some new structure: a defined list of recovery passes that we can run in a loop, as well as consolidating the log messages.
The main benefit is consolidating the "should run this recovery pass" logic, as well as cleaning up the "this recovery pass has finished" state; instead of having a bunch of ad-hoc state bits in c->flags, we've now got c->curr_recovery_pass.
By consolidating the "should run this recovery pass" logic, in the future on disk format upgrades will be able to say "upgrading to this version requires x passes to run", instead of forcing all of fsck to run.
--- fs/bcachefs/alloc_background.c | 12 +- fs/bcachefs/alloc_foreground.c | 9 +- fs/bcachefs/backpointers.c | 6 +- fs/bcachefs/bcachefs.h | 49 ++++++- fs/bcachefs/btree_cache.c | 2 +- fs/bcachefs/btree_gc.c | 4 +- fs/bcachefs/fsck.c | 77 ++--------- fs/bcachefs/fsck.h | 10 +- fs/bcachefs/recovery.c | 227 +++++++++++++-------------------- fs/bcachefs/subvolume.c | 10 +- fs/bcachefs/subvolume.h | 8 +- 11 files changed, 175 insertions(+), 239 deletions(-)
fs/bcachefs/alloc_background.c +6 -6
@@ -286,7 +286,7 @@ int bch2_alloc_v4_invalid(const struct bch_fs *c, struct bkey_s_c k,if (rw ==WRITE && !(flags &BKEY_INVALID_JOURNAL ) && -test_bit (BCH_FS_CHECK_BACKPOINTERS_DONE , &c->flags)) { + c->curr_recovery_pass > BCH_RECOVERY_PASS_check_btree_backpointers) {unsigned i, bp_len =0 ;for (i =0 ; i <BCH_ALLOC_V4_NR_BACKPOINTERS (a.v); i++)
@@ -336,7 +336,7 @@ int bch2_alloc_v4_invalid(const struct bch_fs *c, struct bkey_s_c k, }if (!a.v->io_time[READ ] && -test_bit (BCH_FS_CHECK_ALLOC_TO_LRU_REFS_DONE , &c->flags)) { + c->curr_recovery_pass > BCH_RECOVERY_PASS_check_alloc_to_lru_refs) {prt_printf (err,"cached bucket with read_time == 0" );return -BCH_ERR_invalid_bkey; }
@@ -777,7 +777,7 @@ static int bch2_bucket_do_index(struct btree_trans *trans,return ret;if (ca->mi.freespace_initialized && -test_bit (BCH_FS_CHECK_ALLOC_DONE , &c->flags) && + c->curr_recovery_pass > BCH_RECOVERY_PASS_check_alloc_info && bch2_trans_inconsistent_on(old.k->type != old_type, trans, "incorrect key when %s %s:%llu:%llu:0 (got %s should be %s)\n" " for %s",
@@ -1663,7 +1663,7 @@ static int bch2_discard_one_bucket(struct btree_trans *trans, }if (a->v.journal_seq > c->journal.flushed_seq_ondisk) { -if (test_bit (BCH_FS_CHECK_ALLOC_DONE , &c->flags)) { +if (c->curr_recovery_pass > BCH_RECOVERY_PASS_check_alloc_info) { bch2_trans_inconsistent(trans, "clearing need_discard but journal_seq %llu > flushed_seq %llu\n" "%s",
@@ -1676,7 +1676,7 @@ static int bch2_discard_one_bucket(struct btree_trans *trans, }if (a->v.data_type != BCH_DATA_need_discard) { -if (test_bit (BCH_FS_CHECK_ALLOC_DONE , &c->flags)) { +if (c->curr_recovery_pass > BCH_RECOVERY_PASS_check_alloc_info) { bch2_trans_inconsistent(trans, "bucket incorrectly set in need_discard btree\n" "%s",
@@ -1844,7 +1844,7 @@ static int invalidate_one_bucket(struct btree_trans *trans,bch2_bkey_val_to_text (&buf, c,bkey_i_to_s_c (&a->k_i));bch_err (c,"%s" , buf.buf); -if (test_bit (BCH_FS_CHECK_LRUS_DONE , &c->flags)) { +if (c->curr_recovery_pass > BCH_RECOVERY_PASS_check_lrus) {bch2_inconsistent_error (c); ret = -EINVAL ; }
fs/bcachefs/alloc_foreground.c +4 -5
@@ -324,7 +324,7 @@ static struct open_bucket *try_alloc_bucket(struct btree_trans *trans, struct bc a =bch2_alloc_to_v4 (k, &a_convert);if (a->data_type != BCH_DATA_free) { -if (!test_bit (BCH_FS_CHECK_ALLOC_DONE , &c->flags)) { +if (c->curr_recovery_pass <= BCH_RECOVERY_PASS_check_alloc_info) { ob =NULL ; goto err; }
@@ -340,7 +340,7 @@ static struct open_bucket *try_alloc_bucket(struct btree_trans *trans, struct bc }if (genbits != (alloc_freespace_genbits (*a) >>56 ) && -test_bit (BCH_FS_CHECK_ALLOC_DONE , &c->flags)) { + c->curr_recovery_pass > BCH_RECOVERY_PASS_check_alloc_info) {prt_printf (&buf,"bucket in freespace btree with wrong genbits (got %u should be %llu)\n" " freespace key " , genbits,alloc_freespace_genbits (*a) >>56 );
@@ -350,10 +350,9 @@ static struct open_bucket *try_alloc_bucket(struct btree_trans *trans, struct bcbch2_trans_inconsistent (trans,"%s" , buf.buf); ob =ERR_PTR (-EIO ); goto err; - } -if (!test_bit (BCH_FS_CHECK_BACKPOINTERS_DONE , &c->flags)) { +if (c->curr_recovery_pass <= BCH_RECOVERY_PASS_check_extents_to_backpointers) {struct bch_backpointer bp;struct bpos bp_pos =POS_MIN ;
@@ -556,7 +555,7 @@ static struct open_bucket *bch2_bucket_alloc_trans(struct btree_trans *trans,if (s.skipped_need_journal_commit *2 > avail)bch2_journal_flush_async (&c->journal,NULL ); -if (!ob && freespace && !test_bit (BCH_FS_CHECK_ALLOC_DONE , &c->flags)) { +if (!ob && freespace && c->curr_recovery_pass <= BCH_RECOVERY_PASS_check_alloc_info) { freespace = false; goto alloc; }
fs/bcachefs/backpointers.c +3 -3
@@ -104,7 +104,7 @@ static noinline int backpointer_mod_err(struct btree_trans *trans,bch2_bkey_val_to_text (&buf, c, orig_k);bch_err (c,"%s" , buf.buf); - }else if (test_bit (BCH_FS_CHECK_BACKPOINTERS_DONE , &c->flags)) { + }else if (c->curr_recovery_pass > BCH_RECOVERY_PASS_check_extents_to_backpointers) {prt_printf (&buf,"backpointer not found when deleting" );prt_newline (&buf);printbuf_indent_add (&buf,2 );
@@ -125,7 +125,7 @@ static noinline int backpointer_mod_err(struct btree_trans *trans,printbuf_exit (&buf); -if (test_bit (BCH_FS_CHECK_BACKPOINTERS_DONE , &c->flags)) { +if (c->curr_recovery_pass > BCH_RECOVERY_PASS_check_extents_to_backpointers) {bch2_inconsistent_error (c);return -EIO ; }else {
@@ -258,7 +258,7 @@ static void backpointer_not_found(struct btree_trans *trans,bch2_backpointer_to_text (&buf, &bp);prt_printf (&buf,"\n " );bch2_bkey_val_to_text (&buf, c, k); -if (!test_bit (BCH_FS_CHECK_BACKPOINTERS_DONE , &c->flags)) +if (c->curr_recovery_pass >= BCH_RECOVERY_PASS_check_extents_to_backpointers)bch_err_ratelimited (c,"%s" , buf.buf);else bch2_trans_inconsistent (trans,"%s" , buf.buf);
fs/bcachefs/bcachefs.h +44 -5
67 lines not on this page · raw message
fs/bcachefs/btree_cache.c +1 -1
@@ -825,7 +825,7 @@ static noinline void btree_bad_header(struct bch_fs *c, struct btree *b) {struct printbuf buf =PRINTBUF ; -if (!test_bit (BCH_FS_INITIAL_GC_DONE , &c->flags)) +if (c->curr_recovery_pass <= BCH_RECOVERY_PASS_check_allocations)return ; prt_printf(&buf,
fs/bcachefs/btree_gc.c +2 -2
@@ -1810,7 +1810,7 @@ int bch2_gc(struct bch_fs *c, bool initial, bool metadata_only)if (IS_ENABLED (CONFIG_BCACHEFS_DEBUG ) || (BCH_SB_HAS_TOPOLOGY_ERRORS (c->disk_sb.sb) && - !test_bit (BCH_FS_INITIAL_GC_DONE , &c->flags) && + c->curr_recovery_pass <= BCH_RECOVERY_PASS_check_allocations && c->opts.fix_errors !=FSCK_OPT_NO )) {bch_info (c,"Starting topology repair pass" ); ret =bch2_repair_topology (c);
@@ -1825,7 +1825,7 @@ int bch2_gc(struct bch_fs *c, bool initial, bool metadata_only)if (ret == -BCH_ERR_need_topology_repair && !test_bit (BCH_FS_TOPOLOGY_REPAIR_DONE , &c->flags) && - !test_bit (BCH_FS_INITIAL_GC_DONE , &c->flags)) { + c->curr_recovery_pass <= BCH_RECOVERY_PASS_check_allocations) {set_bit (BCH_FS_NEED_ANOTHER_GC , &c->flags);SET_BCH_SB_HAS_TOPOLOGY_ERRORS (c->disk_sb.sb, true); ret =0 ;
fs/bcachefs/fsck.c +11 -66
169 lines not on this page · raw message
fs/bcachefs/fsck.h +8 -2
@@ -2,7 +2,13 @@#ifndef _BCACHEFS_FSCK_H#define _BCACHEFS_FSCK_H -int bch2_fsck_full (struct bch_fs *); -int bch2_fsck_walk_inodes_only (struct bch_fs *); +int bch2_check_inodes (struct bch_fs *); +int bch2_check_extents (struct bch_fs *); +int bch2_check_dirents (struct bch_fs *); +int bch2_check_xattrs (struct bch_fs *); +int bch2_check_root (struct bch_fs *); +int bch2_check_directory_structure (struct bch_fs *); +int bch2_check_nlinks (struct bch_fs *); +int bch2_fix_reflink_p (struct bch_fs *);#endif /* _BCACHEFS_FSCK_H */
fs/bcachefs/recovery.c +87 -140
263 lines not on this page · raw message
fs/bcachefs/subvolume.c +5 -5
@@ -408,7 +408,7 @@ static int check_snapshot_tree(struct btree_trans *trans,* And, make sure it points to a subvolume within that snapshot tree, or correct * it to point to the oldest subvolume within that snapshot tree. */ -int bch2_fs_check_snapshot_trees (struct bch_fs *c) +int bch2_check_snapshot_trees (struct bch_fs *c) {struct btree_iter iter;struct bkey_s_c k;
@@ -612,7 +612,7 @@ static int check_snapshot(struct btree_trans *trans,return ret; } -int bch2_fs_check_snapshots (struct bch_fs *c) +int bch2_check_snapshots (struct bch_fs *c) {struct btree_iter iter;struct bkey_s_c k;
@@ -692,7 +692,7 @@ static int check_subvol(struct btree_trans *trans,return ret; } -int bch2_fs_check_subvols (struct bch_fs *c) +int bch2_check_subvols (struct bch_fs *c) {struct btree_iter iter;struct bkey_s_c k;
@@ -713,7 +713,7 @@ void bch2_fs_snapshots_exit(struct bch_fs *c)genradix_free (&c->snapshots); } -int bch2_fs_snapshots_start (struct bch_fs *c) +int bch2_snapshots_read (struct bch_fs *c) {struct btree_iter iter;struct bkey_s_c k;
@@ -1151,7 +1151,7 @@ static int bch2_delete_dead_snapshots_hook(struct btree_trans *trans,set_bit (BCH_FS_HAVE_DELETED_SNAPSHOTS , &c->flags); -if (!test_bit (BCH_FS_FSCK_DONE , &c->flags)) +if (c->curr_recovery_pass <= BCH_RECOVERY_PASS_delete_dead_snapshots)return 0 ;bch2_delete_dead_snapshots_async (c);
fs/bcachefs/subvolume.h +4 -4
@@ -130,12 +130,12 @@ static inline int snapshot_list_add(struct bch_fs *c, snapshot_id_list *s, u32 ireturn ret; } -int bch2_fs_check_snapshot_trees (struct bch_fs *); -int bch2_fs_check_snapshots (struct bch_fs *); -int bch2_fs_check_subvols (struct bch_fs *); +int bch2_check_snapshot_trees (struct bch_fs *); +int bch2_check_snapshots (struct bch_fs *); +int bch2_check_subvols (struct bch_fs *);void bch2_fs_snapshots_exit (struct bch_fs *); -int bch2_fs_snapshots_start (struct bch_fs *); +int bch2_snapshots_read (struct bch_fs *);int bch2_subvolume_invalid (const struct bch_fs *,struct bkey_s_c ,unsigned ,struct printbuf *);
Signature · 1 line
2.40.1
PATCH 10/10 bcachefs: bcachefs_metadata_version_major_minor
In reply to Kent OverstreetThis introduces major/minor versioning to the superblock version number. Major version number changes indicate incompatible releases; we can move forward to a new major version number, but not backwards. Minor version numbers indicate compatible changes - these add features, but can still be mounted and used by old versions.
With the recent patches that make it possible to roll out new btrees and key types without breaking compatibility, we should be able to roll out most new features without incompatible changes.
--- fs/bcachefs/bcachefs.h | 1 - fs/bcachefs/bcachefs_format.h | 46 +++++++++++++----------- fs/bcachefs/btree_io.c | 7 ++-- fs/bcachefs/journal_io.c | 12 ++++--- fs/bcachefs/recovery.c | 66 +++++++++++++++++++++++++++-------- fs/bcachefs/super-io.c | 58 ++++++++++++++++++++---------- fs/bcachefs/super-io.h | 3 +- 7 files changed, 132 insertions(+), 61 deletions(-)
fs/bcachefs/bcachefs.h +0 -1
@@ -568,7 +568,6 @@ enum {BCH_FS_INITIAL_GC_UNFIXED ,/* kill when we enumerate fsck errors */ BCH_FS_NEED_ANOTHER_GC , -BCH_FS_VERSION_UPGRADE , BCH_FS_HAVE_DELETED_SNAPSHOTS,/* errors: */
fs/bcachefs/bcachefs_format.h +25 -21
60 lines not on this page · raw message
fs/bcachefs/btree_io.c +5 -2
@@ -701,7 +701,9 @@ static int validate_bset(struct bch_fs *c, struct bch_dev *ca,btree_err_on (!bch2_version_compatible (version),BTREE_ERR_INCOMPATIBLE , c, ca, b, i, -"unsupported bset version %u" , version); +"unsupported bset version %u.%u" , +BCH_VERSION_MAJOR (version), +BCH_VERSION_MINOR (version)); if (btree_err_on(version < c->sb.version_min, BTREE_ERR_FIXABLE, c, NULL, b, i,
@@ -713,7 +715,8 @@ static int validate_bset(struct bch_fs *c, struct bch_dev *ca,mutex_unlock (&c->sb_lock); } -if (btree_err_on (version > c->sb.version, +if (btree_err_on (BCH_VERSION_MAJOR (version) > +BCH_VERSION_MAJOR (c->sb.version),BTREE_ERR_FIXABLE , c,NULL , b, i,"bset version %u newer than superblock version %u" , version, c->sb.version)) {
fs/bcachefs/journal_io.c +8 -4
@@ -747,9 +747,11 @@ static int jset_validate(struct bch_fs *c, version =le32_to_cpu (jset->version);if (journal_entry_err_on (!bch2_version_compatible (version), c, jset,NULL , -"%s sector %llu seq %llu: incompatible journal entry version %u" , +"%s sector %llu seq %llu: incompatible journal entry version %u.%u" , ca ? ca->name : c->name, - sector,le64_to_cpu (jset->seq), version)) { + sector,le64_to_cpu (jset->seq), +BCH_VERSION_MAJOR (version), +BCH_VERSION_MINOR (version))) {/* don't try to continue: */ return -EINVAL ; }
@@ -794,9 +796,11 @@ static int jset_validate_early(struct bch_fs *c, version =le32_to_cpu (jset->version);if (journal_entry_err_on (!bch2_version_compatible (version), c, jset,NULL , -"%s sector %llu seq %llu: unknown journal entry version %u" , +"%s sector %llu seq %llu: unknown journal entry version %u.%u" , ca ? ca->name : c->name, - sector,le64_to_cpu (jset->seq), version)) { + sector,le64_to_cpu (jset->seq), +BCH_VERSION_MAJOR (version), +BCH_VERSION_MINOR (version))) {/* don't try to continue: */ return -EINVAL ; }
fs/bcachefs/recovery.c +52 -14
83 lines not on this page · raw message
fs/bcachefs/super-io.c +40 -18
88 lines not on this page · raw message
fs/bcachefs/super-io.h +2 -1
@@ -11,11 +11,12 @@static inline bool bch2_version_compatible (u16 version) { -return version <= bcachefs_metadata_version_current && +return BCH_VERSION_MAJOR (version) <=BCH_VERSION_MAJOR (bcachefs_metadata_version_current) && version >= bcachefs_metadata_version_min; }void bch2_version_to_text (struct printbuf *,unsigned ); +unsigned bch2_latest_compatible_version (unsigned );struct bch_sb_field *bch2_sb_field_get (struct bch_sb *,enum bch_sb_field_type ); struct bch_sb_field *bch2_sb_field_resize(struct bch_sb_handle *,
Signature · 1 line
2.40.1
Re: [PATCH 10/10] bcachefs: bcachefs_metadata_version_major_minor
Hi Kent,
the subject for this patch looks a bit like a placeholder, in contrast with the other patches of the series.
More below.
Quoted 28 lines · Kent Overstreet
This introduces major/minor versioning to the superblock version number. Major version number changes indicate incompatible releases; we can move forward to a new major version number, but not backwards. Minor version numbers indicate compatible changes - these add features, but can still be mounted and used by old versions.
With the recent patches that make it possible to roll out new btrees and key types without breaking compatibility, we should be able to roll out most new features without incompatible changes.
Signed-off-by Kent Overstreet <[email protected]>--- fs/bcachefs/bcachefs.h | 1 - fs/bcachefs/bcachefs_format.h | 46 +++++++++++++----------- fs/bcachefs/btree_io.c | 7 ++-- fs/bcachefs/journal_io.c | 12 ++++--- fs/bcachefs/recovery.c | 66 +++++++++++++++++++++++++++-------- fs/bcachefs/super-io.c | 58 ++++++++++++++++++++---------- fs/bcachefs/super-io.h | 3 +- 7 files changed, 132 insertions(+), 61 deletions(-)[..]
fs/bcachefs/bcachefs_format.h +0
-0
fs/bcachefs/bcachefs_format.h struct bch_sb_field_journal_seq_blacklist { · line 1577
Quoted 4 lines
fs/bcachefs/bcachefs_format.h +1
-0@@ -1574,28 +1574,32 @@ struct bch_sb_field_journal_seq_blacklist {* One common version number for all on disk data structures - superblock, btree * nodes, journal entries */ +#define BCH_VERSION_MAJOR (_v) ((__u8) ((_v) >> 10))
This "wastes" two bits of the major version that do not fit into the u8 value. This is problematic in two cases:
* If in the future there is a major version so high that it will overflow the 8 bits it would be interpreted as major version "0" and inadvertedly accepted by older/ancient implementations.
* If this logic is used in other derived projects (libblkid/util-linux) it is likely that those will miss if the in-kernel logic is expanded to the full ten bits.
If eight bits are supposed to be enough for the major version, why not split it into 8 bits major, 8 bits minor?
Or make the major version an u16.
fs/bcachefs/bcachefs_format.h struct bch_sb_field_journal_seq_blacklist { · line 1579
Quoted 2 lines
fs/bcachefs/bcachefs_format.h +2
-0+#define BCH_VERSION_MINOR (_v) ((__u8) ((_v) >> 0)) +#define BCH_VERSION (_major, _minor) (((_major) << 10)|(_minor) << 0)
This changes the user-visible data format.
For example for libblkid/util-linux parses and exposes it to users.
I don't think it's a big problem and I will adapt blkid for this format.
But I think it would be good opportunity to have some sort of official statement about the stability of the current (pre-mainline) superblock.
Do you expect or plan any incompatibilities?
Quoted 1 line
[..]
Thomas
Re: [PATCH 10/10] bcachefs: bcachefs_metadata_version_major_minor
Quoted 48 lines · Thomas Weißschuh
Hi Kent,
the subject for this patch looks a bit like a placeholder, in contrast with the other patches of the series.
More below.
Quoted 28 lines · Kent Overstreet
This introduces major/minor versioning to the superblock version number. Major version number changes indicate incompatible releases; we can move forward to a new major version number, but not backwards. Minor version numbers indicate compatible changes - these add features, but can still be mounted and used by old versions.
With the recent patches that make it possible to roll out new btrees and key types without breaking compatibility, we should be able to roll out most new features without incompatible changes.
Signed-off-by Kent Overstreet <[email protected]>--- fs/bcachefs/bcachefs.h | 1 - fs/bcachefs/bcachefs_format.h | 46 +++++++++++++----------- fs/bcachefs/btree_io.c | 7 ++-- fs/bcachefs/journal_io.c | 12 ++++--- fs/bcachefs/recovery.c | 66 +++++++++++++++++++++++++++-------- fs/bcachefs/super-io.c | 58 ++++++++++++++++++++---------- fs/bcachefs/super-io.h | 3 +- 7 files changed, 132 insertions(+), 61 deletions(-)[..]
fs/bcachefs/bcachefs_format.h +0
-0fs/bcachefs/bcachefs_format.h struct bch_sb_field_journal_seq_blacklist { · line 1577
Quoted 4 lines
fs/bcachefs/bcachefs_format.h +1
-0@@ -1574,28 +1574,32 @@ struct bch_sb_field_journal_seq_blacklist {* One common version number for all on disk data structures - superblock, btree * nodes, journal entries */ +#define BCH_VERSION_MAJOR (_v) ((__u8) ((_v) >> 10))This "wastes" two bits of the major version that do not fit into the u8 value. This is problematic in two cases:
* If in the future there is a major version so high that it will overflow the 8 bits it would be interpreted as major version "0" and inadvertedly accepted by older/ancient implementations.
Quoted 7 lines
* If this logic is used in other derived projects (libblkid/util-linux) it is likely that those will miss if the in-kernel logic is expanded to the full ten bits.
If eight bits are supposed to be enough for the major version, why not split it into 8 bits major, 8 bits minor?
Or make the major version an u16.
Minor releases are going to be much more common than major releases:
thus, I'm using 10 bits for the minor number and 6 bits for the major.
It would've been better if the version number was a u32, and then the major and minor could've both been u16s, but alas I was not that foresightful :)
Quoted 6 lines
fs/bcachefs/bcachefs_format.h struct bch_sb_field_journal_seq_blacklist { · line 1579
Quoted 2 lines
fs/bcachefs/bcachefs_format.h +2
-0+#define BCH_VERSION_MINOR (_v) ((__u8) ((_v) >> 0)) +#define BCH_VERSION (_major, _minor) (((_major) << 10)|(_minor) << 0)This changes the user-visible data format.
For example for libblkid/util-linux parses and exposes it to users.
Quoted 6 lines
I don't think it's a big problem and I will adapt blkid for this format.
But I think it would be good opportunity to have some sort of official statement about the stability of the current (pre-mainline) superblock.
Do you expect or plan any incompatibilities?
I think it's pretty unlikely that there'll ever be a majorly incompatible superblock change. I recently switched to a new magic UUID, different from the one bcache uses - that and this change are the biggest incompatibilities there have been since I implemented the "tagged variable-length sections" scheme years ago.
If we can't fit something in the main part of the superblock, we'll just create new tagged sections, and those are much easier from a compatibility point of view.
Re: [PATCH 10/10] bcachefs: bcachefs_metadata_version_major_minor
Quoted 53 lines · Kent Overstreet
Quoted 52 lines · Thomas Weißschuh
Hi Kent,
the subject for this patch looks a bit like a placeholder, in contrast with the other patches of the series.
More below.
Quoted 28 lines · Kent Overstreet
This introduces major/minor versioning to the superblock version number. Major version number changes indicate incompatible releases; we can move forward to a new major version number, but not backwards. Minor version numbers indicate compatible changes - these add features, but can still be mounted and used by old versions.
With the recent patches that make it possible to roll out new btrees and key types without breaking compatibility, we should be able to roll out most new features without incompatible changes.
Signed-off-by Kent Overstreet <[email protected]>--- fs/bcachefs/bcachefs.h | 1 - fs/bcachefs/bcachefs_format.h | 46 +++++++++++++----------- fs/bcachefs/btree_io.c | 7 ++-- fs/bcachefs/journal_io.c | 12 ++++--- fs/bcachefs/recovery.c | 66 +++++++++++++++++++++++++++-------- fs/bcachefs/super-io.c | 58 ++++++++++++++++++++---------- fs/bcachefs/super-io.h | 3 +- 7 files changed, 132 insertions(+), 61 deletions(-)[..]
fs/bcachefs/bcachefs_format.h +0
-0fs/bcachefs/bcachefs_format.h struct bch_sb_field_journal_seq_blacklist { · line 1577
Quoted 4 lines
fs/bcachefs/bcachefs_format.h +1
-0@@ -1574,28 +1574,32 @@ struct bch_sb_field_journal_seq_blacklist {* One common version number for all on disk data structures - superblock, btree * nodes, journal entries */ +#define BCH_VERSION_MAJOR (_v) ((__u8) ((_v) >> 10))This "wastes" two bits of the major version that do not fit into the u8 value. This is problematic in two cases:
* If in the future there is a major version so high that it will overflow the 8 bits it would be interpreted as major version "0" and inadvertedly accepted by older/ancient implementations.
* If this logic is used in other derived projects (libblkid/util-linux) it is likely that those will miss if the in-kernel logic is expanded to the full ten bits.
Quoted 6 lines
Quoted 3 lines
If eight bits are supposed to be enough for the major version, why not split it into 8 bits major, 8 bits minor?
Or make the major version an u16.Minor releases are going to be much more common than major releases:
thus, I'm using 10 bits for the minor number and 6 bits for the major.
Sounds good. But can this be reflected in the definition of BCH_VERSION_MINOR?
Currently it only supports 8 bits.
Something like:
#define BCH_VERSION_MINOR(_v) ((__u16) ((_v) & 0x03ff))
Quoted 23 lines
It would've been better if the version number was a u32, and then the major and minor could've both been u16s, but alas I was not that foresightful :)
Quoted 6 lines
fs/bcachefs/bcachefs_format.h struct bch_sb_field_journal_seq_blacklist { · line 1579
Quoted 2 lines
fs/bcachefs/bcachefs_format.h +2
-0+#define BCH_VERSION_MINOR (_v) ((__u8) ((_v) >> 0)) +#define BCH_VERSION (_major, _minor) (((_major) << 10)|(_minor) << 0)This changes the user-visible data format.
For example for libblkid/util-linux parses and exposes it to users.Quoted 6 lines
I don't think it's a big problem and I will adapt blkid for this format.
But I think it would be good opportunity to have some sort of official statement about the stability of the current (pre-mainline) superblock.
Do you expect or plan any incompatibilities?
I think it's pretty unlikely that there'll ever be a majorly incompatible superblock change. I recently switched to a new magic UUID, different from the one bcache uses - that and this change are the biggest incompatibilities there have been since I implemented the "tagged variable-length sections" scheme years ago.
Quoted 3 lines
If we can't fit something in the main part of the superblock, we'll just create new tagged sections, and those are much easier from a compatibility point of view.
Thanks for the clarification! Sounds good to me.
PS: Is the source for "bcachefs: Principles of Operations" available?
While reading it I noticed some typos.
Re: [PATCH 10/10] bcachefs: bcachefs_metadata_version_major_minor
Quoted 11 lines · Thomas Weißschuh
Quoted 2 lines · Kent Overstreet
Minor releases are going to be much more common than major releases:
thus, I'm using 10 bits for the minor number and 6 bits for the major.Sounds good. But can this be reflected in the definition of
BCH_VERSION_MINOR?
Currently it only supports 8 bits.Something like:
#define
BCH_VERSION_MINOR(_v)((__u16) ((_v) &0x03ff))
Ah, good catch. Changing it to:
#define BCH_VERSION_MAJOR(_v) ((__u16) ((_v) >> 10))
#define BCH_VERSION_MINOR(_v) ((__u16) ((_v) & ~(~0U << 10)))