3- and 4- copy RAID1
David Sterba
maintainercover 3- and 4- copy RAID1
Hi,
I have some goodies that go into the RAID56 problem, although not implementing all the remaining features, it can be useful independently.
This time my hackweek project
https://hackweek.suse.com/17/projects/do-something-about-btrfs-and-raid56
aimed to implement the fix for the write hole problem but I spent more time with analysis and design of the solution and don't have a working prototype for that yet.
This patchset brings a feature that will be used by the raid56 log, the log has to be on the same redundancy level and thus we need a 3-copy replication for raid6. As it was easy to extend to higher replication, I've added a 4-copy replication, that would allow triple copy raid (that does not have a standardized name).
The number of copies is fixed, so it's not N-copy for an arbitrary N. This would complicate the implementation too much, though I'd be willing to add a 5-copy replication for a small bribe.
The new raid profiles and covered by an incompatibility bit, called extended_raid, the (idealistic) plan is to stuff as many new raid-related features as possible. The patch 4/4 mentions the 3- 4- copy raid1, configurable stripe length, write hole log and triple parity. If the plan turns out to be too ambitious, the ready and implemented features will be split and merged.
An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
Regarding the missing raid56 features, I'll continue working on them as time permits in the following weeks/months, as I'm not aware of anybody working on that actively enough so to speak.
Anyway, git branches with the patches:
kernel: git://github.com/kdave/btrfs-devel dev/extended-raid-ncopies progs: git://github.com/kdave/btrfs-progs dev/extended-raid-ncopies
David Sterba (4):
btrfs: refactor block group replication factor calculation to a helper
btrfs: add support for 3-copy replication (raid1c3)
btrfs: add support for 4-copy replication (raid1c4)
btrfs: add incompatibility bit for extended raid features
fs/btrfs/ctree.h | 1 + fs/btrfs/extent-tree.c | 45 +++++++----------- fs/btrfs/relocation.c | 1 + fs/btrfs/scrub.c | 4 +- fs/btrfs/super.c | 17 +++---- fs/btrfs/sysfs.c | 2 + fs/btrfs/volumes.c | 84 ++++++++++++++++++++++++++++++--- fs/btrfs/volumes.h | 6 +++ include/uapi/linux/btrfs.h | 12 ++++- include/uapi/linux/btrfs_tree.h | 6 +++ 10 files changed, 134 insertions(+), 44 deletions(-)
Signature · 1 line
2.18.0
David Sterba
maintainerPATCH 1/1 btrfs-progs: add support for raid1c3 and raid1c4
$ ./mkfs.btrfs -m raid1c4 -d raid1c3 /dev/sd[abcd]
Label: (null)
UUID: f1f988ab-6750-4bc2-957b-98a4ebe98631
Node size: 16384
Sector size: 4096
Filesystem size: 8.00GiB
Block group profiles:
Data: RAID1C3 273.06MiB
Metadata: RAID1C4 204.75MiB
System: RAID1C4 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata, extraid
Number of devices: 4
Devices:
ID SIZE PATH
1 2.00GiB /dev/sda
2 2.00GiB /dev/sdb
3 2.00GiB /dev/sdc
4 2.00GiB /dev/sdd--- chunk-recover.c | 4 ++++ cmds-balance.c | 4 ++++ cmds-fi-usage.c | 8 +++++++ cmds-inspect-dump-super.c | 3 ++- ctree.h | 8 +++++++ extent-tree.c | 4 ++++ fsfeatures.c | 6 +++++ ioctl.h | 3 ++- mkfs/main.c | 11 ++++++++- print-tree.c | 6 +++++ utils.c | 13 +++++++++-- volumes.c | 48 +++++++++++++++++++++++++++++++++++++-- volumes.h | 4 ++++ 13 files changed, 115 insertions(+), 7 deletions(-)
chunk-recover.c +4 -0
@@ -1569,6 +1569,10 @@ static int calc_num_stripes(u64 type)else if (type & (BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_DUP ))return 2 ; +else if (type & (BTRFS_BLOCK_GROUP_RAID1C3 )) +return 3 ; +else if (type & (BTRFS_BLOCK_GROUP_RAID1C4 )) +return 4 ;else return 1 ; }
cmds-balance.c +4 -0
@@ -46,6 +46,10 @@ static int parse_one_profile(const char *profile, u64 *flags) *flags |=BTRFS_BLOCK_GROUP_RAID0 ; }else if (!strcmp (profile,"raid1" )) { *flags |=BTRFS_BLOCK_GROUP_RAID1 ; + }else if (!strcmp (profile,"raid1c3" )) { + *flags |=BTRFS_BLOCK_GROUP_RAID1C3 ; + }else if (!strcmp (profile,"raid1c4" )) { + *flags |=BTRFS_BLOCK_GROUP_RAID1C4 ; }else if (!strcmp (profile,"raid10" )) { *flags |=BTRFS_BLOCK_GROUP_RAID10 ; }else if (!strcmp (profile,"raid5" )) {
cmds-fi-usage.c +8 -0
@@ -373,6 +373,10 @@ static int print_filesystem_usage_overall(int fd, struct chunk_info *chunkinfo, ratio =1 ; else if (flags & BTRFS_BLOCK_GROUP_RAID1) ratio =2 ; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C3) + ratio =3 ; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C4) + ratio =4 ; else if (flags & BTRFS_BLOCK_GROUP_RAID5) ratio =0 ; else if (flags & BTRFS_BLOCK_GROUP_RAID6)
@@ -653,6 +657,10 @@ static u64 calc_chunk_size(struct chunk_info *ci)return ci->size / ci->num_stripes; else if (ci->type & BTRFS_BLOCK_GROUP_RAID1) return ci->size ; + else if (ci->type & BTRFS_BLOCK_GROUP_RAID1C3) + return ci->size; + else if (ci->type & BTRFS_BLOCK_GROUP_RAID1C4) + return ci->size; else if (ci->type & BTRFS_BLOCK_GROUP_DUP) return ci->size ; else if (ci->type & BTRFS_BLOCK_GROUP_RAID5)
cmds-inspect-dump-super.c +2 -1
@@ -228,7 +228,8 @@ static struct readable_flag_entry incompat_flags_array[] = {DEF_INCOMPAT_FLAG_ENTRY (EXTENDED_IREF ),DEF_INCOMPAT_FLAG_ENTRY (RAID56 ),DEF_INCOMPAT_FLAG_ENTRY (SKINNY_METADATA ), - DEF_INCOMPAT_FLAG_ENTRY(NO_HOLES) +DEF_INCOMPAT_FLAG_ENTRY (NO_HOLES ), + DEF_INCOMPAT_FLAG_ENTRY(EXTENDED_RAID), };static const int incompat_flags_num =sizeof (incompat_flags_array) /sizeof (struct readable_flag_entry );
ctree.h +8 -0
@@ -489,6 +489,7 @@ struct btrfs_super_block {#define BTRFS_FEATURE_INCOMPAT_RAID56 (1ULL << 7)#define BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA (1ULL << 8)#define BTRFS_FEATURE_INCOMPAT_NO_HOLES (1ULL << 9) +#define BTRFS_FEATURE_INCOMPAT_EXTENDED_RAID (1ULL << 10)#define BTRFS_FEATURE_COMPAT_SUPP 0ULL
@@ -509,6 +510,7 @@ struct btrfs_super_block {BTRFS_FEATURE_INCOMPAT_RAID56 | \BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS | \BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | \ +BTRFS_FEATURE_INCOMPAT_EXTENDED_RAID | \ BTRFS_FEATURE_INCOMPAT_NO_HOLES)/*
@@ -958,6 +960,8 @@ struct btrfs_csum_item {#define BTRFS_BLOCK_GROUP_RAID10 (1ULL << 6)#define BTRFS_BLOCK_GROUP_RAID5 (1ULL << 7)#define BTRFS_BLOCK_GROUP_RAID6 (1ULL << 8) +#define BTRFS_BLOCK_GROUP_RAID1C3 (1ULL << 9) +#define BTRFS_BLOCK_GROUP_RAID1C4 (1ULL << 10)#define BTRFS_BLOCK_GROUP_RESERVED BTRFS_AVAIL_ALLOC_BIT_SINGLEenum btrfs_raid_types {
@@ -968,6 +972,8 @@ enum btrfs_raid_types {BTRFS_RAID_SINGLE ,BTRFS_RAID_RAID5 ,BTRFS_RAID_RAID6 , +BTRFS_RAID_RAID1C3 , +BTRFS_RAID_RAID1C4 , BTRFS_NR_RAID_TYPES };
@@ -979,6 +985,8 @@ enum btrfs_raid_types {BTRFS_BLOCK_GROUP_RAID1 | \BTRFS_BLOCK_GROUP_RAID5 | \BTRFS_BLOCK_GROUP_RAID6 | \ +BTRFS_BLOCK_GROUP_RAID1C3 | \ +BTRFS_BLOCK_GROUP_RAID1C4 | \BTRFS_BLOCK_GROUP_DUP | \ BTRFS_BLOCK_GROUP_RAID10)
extent-tree.c +4 -0
@@ -1848,6 +1848,8 @@ static void set_avail_alloc_bits(struct btrfs_fs_info *fs_info, u64 flags) { u64 extra_flags = flags & (BTRFS_BLOCK_GROUP_RAID0 | BTRFS_BLOCK_GROUP_RAID1 | + BTRFS_BLOCK_GROUP_RAID1C3 | + BTRFS_BLOCK_GROUP_RAID1C4 | BTRFS_BLOCK_GROUP_RAID10 | BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_RAID6 |
@@ -3629,6 +3631,8 @@ static u64 get_dev_extent_len(struct map_lookup *map)case 0 :/* Single */ case BTRFS_BLOCK_GROUP_DUP :case BTRFS_BLOCK_GROUP_RAID1 : +case BTRFS_BLOCK_GROUP_RAID1C3 : +case BTRFS_BLOCK_GROUP_RAID1C4 : div =1 ;break ;case BTRFS_BLOCK_GROUP_RAID5 :
fsfeatures.c +6 -0
@@ -86,6 +86,12 @@ static const struct btrfs_fs_feature { VERSION_TO_STRING2(4,0), NULL, 0, "no explicit hole extents for files" }, + {"extraid" ,BTRFS_FEATURE_INCOMPAT_EXTENDED_RAID , +"extended_raid" , +VERSION_TO_STRING2 (4 ,17 ), +NULL ,0 , +NULL ,0 , + "extended raid features: raid1c3, raid1c4" },/* Keep this one last */ { "list-all", BTRFS_FEATURE_LIST_ALL, NULL } };
ioctl.h +2 -1
@@ -682,7 +682,8 @@ enum btrfs_err_code {BTRFS_ERROR_DEV_TGT_REPLACE ,BTRFS_ERROR_DEV_MISSING_NOT_FOUND ,BTRFS_ERROR_DEV_ONLY_WRITABLE , - BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS +BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS , + BTRFS_ERROR_DEV_RAID1c3_MIN_NOT_MET, };/* An error code to error string mapping for the kernel
mkfs/main.c +10 -1
@@ -346,7 +346,7 @@ static void print_usage(int ret)printf ("Usage: mkfs.btrfs [options] dev [ dev ... ]\n" );printf ("Options:\n" );printf (" allocation profiles:\n" ); -printf ("\t-d|--data PROFILE data profile, raid0, raid1, raid5, raid6, raid10, dup or single\n" ); +printf ("\t-d|--data PROFILE data profile, raid0, raid1, raid1c3, raid1c4, raid5, raid6, raid10, dup or single\n" );printf ("\t-m|--metadata PROFILE metadata profile, values like for data profile\n" );printf ("\t-M|--mixed mix metadata and data together\n" );printf (" features:\n" );
@@ -377,6 +377,10 @@ static u64 parse_profile(const char *s)return BTRFS_BLOCK_GROUP_RAID0 ; }else if (strcasecmp (s,"raid1" ) ==0 ) {return BTRFS_BLOCK_GROUP_RAID1 ; + }else if (strcasecmp (s,"raid1c3" ) ==0 ) { +return BTRFS_BLOCK_GROUP_RAID1C3 ; + }else if (strcasecmp (s,"raid1c4" ) ==0 ) { +return BTRFS_BLOCK_GROUP_RAID1C4 ; }else if (strcasecmp (s,"raid5" ) ==0 ) {return BTRFS_BLOCK_GROUP_RAID5 ; }else if (strcasecmp (s,"raid6" ) ==0 ) {
@@ -958,6 +962,11 @@ int main(int argc, char **argv) features |=BTRFS_FEATURE_INCOMPAT_RAID56 ; } +if ((data_profile | metadata_profile) & + (BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID1C4 )) { + features |=BTRFS_FEATURE_INCOMPAT_EXTENDED_RAID ; + } +if (btrfs_check_nodesize (nodesize, sectorsize, features)) goto error;
print-tree.c +6 -0
@@ -163,6 +163,12 @@ static void bg_flags_to_str(u64 flags, char *ret)case BTRFS_BLOCK_GROUP_RAID1 :strcat (ret,"|RAID1" );break ; +case BTRFS_BLOCK_GROUP_RAID1C3 : +strcat (ret,"|RAID1C3" ); +break ; +case BTRFS_BLOCK_GROUP_RAID1C4 : +strcat (ret,"|RAID1C4" ); +break ;case BTRFS_BLOCK_GROUP_DUP :strcat (ret,"|DUP" );break ;
utils.c +11 -2
@@ -1884,8 +1884,10 @@ static int group_profile_devs_min(u64 flag)case BTRFS_BLOCK_GROUP_RAID5 :return 2 ;case BTRFS_BLOCK_GROUP_RAID6 : +case BTRFS_BLOCK_GROUP_RAID1C3 :return 3 ;case BTRFS_BLOCK_GROUP_RAID10 : +case BTRFS_BLOCK_GROUP_RAID1C4 :return 4 ;default :return -1 ;
@@ -1901,9 +1903,9 @@ int test_num_disk_vs_raid(u64 metadata_profile, u64 data_profile,switch (dev_cnt) {default :case 4 : - allowed |=BTRFS_BLOCK_GROUP_RAID10 ; + allowed |=BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID1C4 ;case 3 : - allowed |=BTRFS_BLOCK_GROUP_RAID6 ; + allowed |=BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_RAID1C3 ;case 2 : allowed |=BTRFS_BLOCK_GROUP_RAID0 |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID5 ;
@@ -1955,7 +1957,10 @@ int group_profile_max_safe_loss(u64 flags)case BTRFS_BLOCK_GROUP_RAID10 :return 1 ;case BTRFS_BLOCK_GROUP_RAID6 : +case BTRFS_BLOCK_GROUP_RAID1C3 :return 2 ; +case BTRFS_BLOCK_GROUP_RAID1C4 : +return 3 ;default :return -1 ; }
@@ -2170,6 +2175,10 @@ const char* btrfs_group_profile_str(u64 flag)return "RAID0" ;case BTRFS_BLOCK_GROUP_RAID1 :return "RAID1" ; +case BTRFS_BLOCK_GROUP_RAID1C3 : +return "RAID1C3" ; +case BTRFS_BLOCK_GROUP_RAID1C4 : +return "RAID1C4" ;case BTRFS_BLOCK_GROUP_RAID5 :return "RAID5" ;case BTRFS_BLOCK_GROUP_RAID6 :
volumes.c +46 -2
102 lines not on this page · raw message
volumes.h +4 -0
@@ -126,6 +126,10 @@ static inline enum btrfs_raid_types btrfs_bg_flags_to_raid_index(u64 flags)return BTRFS_RAID_RAID10 ; else if (flags & BTRFS_BLOCK_GROUP_RAID1) return BTRFS_RAID_RAID1; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C3) + return BTRFS_RAID_RAID1C3; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C4) + return BTRFS_RAID_RAID1C4; else if (flags & BTRFS_BLOCK_GROUP_DUP) return BTRFS_RAID_DUP; else if (flags & BTRFS_BLOCK_GROUP_RAID0)
Signature · 1 line
2.18.0
David Sterba
maintainerPATCH 1/4 btrfs: refactor block group replication factor calculation to a helper
In reply to David SterbaThere are many places that open code the duplicity factor of the block group profiles, create a common helper. This can be easily extended for more copies.
--- fs/btrfs/extent-tree.c | 36 ++++++++---------------------------- fs/btrfs/super.c | 11 +++-------- fs/btrfs/volumes.c | 11 +++++++++++ fs/btrfs/volumes.h | 2 ++ 4 files changed, 24 insertions(+), 36 deletions(-)
fs/btrfs/extent-tree.c +8 -28
@@ -4060,11 +4060,7 @@ static void update_space_info(struct btrfs_fs_info *info, u64 flags,struct btrfs_space_info *found;int factor; -if (flags & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID10 )) - factor =2 ; -else - factor =1 ; + factor =btrfs_bg_type_to_factor (flags); found =__find_space_info (info, flags);ASSERT (found);
@@ -4703,6 +4699,7 @@ static int can_overcommit(struct btrfs_fs_info *fs_info,u64 space_size;u64 avail;u64 used; +int factor;/* Don't overcommit when in mixed mode. */ if (space_info->flags &BTRFS_BLOCK_GROUP_DATA )
@@ -4737,10 +4734,8 @@ static int can_overcommit(struct btrfs_fs_info *fs_info,* doesn't include the parity drive, so we don't have to * change the math */ -if (profile & (BTRFS_BLOCK_GROUP_DUP | -BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID10 )) - avail >>=1 ; + factor =btrfs_bg_type_to_factor (profile); + avail /= factor;/* * If we aren't flushing all things, let us overcommit up to
@@ -6219,12 +6214,8 @@ static int update_block_group(struct btrfs_trans_handle *trans, cache =btrfs_lookup_block_group (info, bytenr);if (!cache)return -ENOENT ; -if (cache->flags & (BTRFS_BLOCK_GROUP_DUP | -BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID10 )) - factor =2 ; -else - factor =1 ; + factor =btrfs_bg_type_to_factor (cache->flags); +/* * If this block group has free space cache written out, we * need to make sure to load it if we are removing space. This
@@ -9520,13 +9511,7 @@ u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo)continue ; } -if (block_group->flags & (BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID10 | -BTRFS_BLOCK_GROUP_DUP )) - factor =2 ; -else - factor =1 ; - + factor =btrfs_bg_type_to_factor (block_group->flags); free_bytes += (block_group->key.offset -btrfs_block_group_used (&block_group->item)) * factor;
@@ -10343,12 +10328,7 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans,memcpy (&key, &block_group->key,sizeof (key)); index =btrfs_bg_flags_to_raid_index (block_group->flags); -if (block_group->flags & (BTRFS_BLOCK_GROUP_DUP | -BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID10 )) - factor =2 ; -else - factor =1 ; + factor =btrfs_bg_type_to_factor (block_group->flags);/* make sure this block group isn't part of an allocation cluster */ cluster = &fs_info->data_alloc_cluster;
fs/btrfs/super.c +3 -8
@@ -2098,14 +2098,9 @@ static int btrfs_statfs(struct dentry *dentry, struct kstatfs *buf)btrfs_account_ro_block_groups_free_space (found);for (i =0 ; i <BTRFS_NR_RAID_TYPES ; i++) { -if (!list_empty (&found->block_groups[i])) { -switch (i) { -case BTRFS_RAID_DUP : -case BTRFS_RAID_RAID1 : -case BTRFS_RAID_RAID10 : - factor =2 ; - } - } +if (!list_empty (&found->block_groups[i])) + factor =btrfs_bg_type_to_factor ( + btrfs_raid_array[i].bg_flag); } }
fs/btrfs/volumes.c +11 -0
@@ -7380,3 +7380,14 @@ void btrfs_reset_fs_info_ptr(struct btrfs_fs_info *fs_info) fs_devices = fs_devices->seed; } } + +/* +* Multiplicity factor for simple profiles: DUP, RAID1-like and RAID10. +*/ +int btrfs_bg_type_to_factor (u64 flags) +{ +if (flags & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID10 )) +return 2 ; +return 1 ; +}
fs/btrfs/volumes.h +2 -0
@@ -560,4 +560,6 @@ void btrfs_reset_fs_info_ptr(struct btrfs_fs_info *fs_info);bool btrfs_check_rw_degradable (struct btrfs_fs_info *fs_info,struct btrfs_device *failing_dev); +int btrfs_bg_type_to_factor (u64 flags); +#endif
Signature · 1 line
2.18.0
David Sterba
maintainerPATCH 2/4 btrfs: add support for 3-copy replication (raid1c3)
In reply to David SterbaAdd new block group profile to store 3 copies in a simliar way that current RAID1 does. The profile name is temporary and may change in the future.
--- fs/btrfs/extent-tree.c | 6 +++++ fs/btrfs/relocation.c | 1 + fs/btrfs/scrub.c | 3 ++- fs/btrfs/super.c | 3 +++ fs/btrfs/volumes.c | 40 ++++++++++++++++++++++++++++----- fs/btrfs/volumes.h | 2 ++ include/uapi/linux/btrfs.h | 3 ++- include/uapi/linux/btrfs_tree.h | 3 +++ 8 files changed, 53 insertions(+), 8 deletions(-)
fs/btrfs/extent-tree.c +6 -0
@@ -7527,6 +7527,7 @@ static noinline int find_free_extent(struct btrfs_fs_info *fs_info,if (!block_group_bits (block_group, flags)) {u64 extra =BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_RAID10 ;
@@ -9330,6 +9331,8 @@ static u64 update_block_group_flags(struct btrfs_fs_info *fs_info, u64 flags) num_devices = fs_info->fs_devices->rw_devices; +ASSERT (!(flags &BTRFS_BLOCK_GROUP_RAID1C3 )); + stripped =BTRFS_BLOCK_GROUP_RAID0 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 ;
@@ -9647,6 +9650,8 @@ int btrfs_can_relocate(struct btrfs_fs_info *fs_info, u64 bytenr) min_free >>=1 ; }else if (index ==BTRFS_RAID_RAID1 ) { dev_min =2 ; + }else if (index ==BTRFS_RAID_RAID1C3 ) { + dev_min =3 ; }else if (index ==BTRFS_RAID_DUP ) {/* Multiply by 2 */ min_free <<=1 ;
@@ -10141,6 +10146,7 @@ int btrfs_read_block_groups(struct btrfs_fs_info *info)if (!(get_alloc_profile (info, space_info->flags) & (BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_DUP )))
fs/btrfs/relocation.c +1 -0
@@ -4339,6 +4339,7 @@ static void describe_relocation(struct btrfs_fs_info *fs_info,DESCRIBE_FLAG (METADATA ,"metadata" );DESCRIBE_FLAG (RAID0 ,"raid0" );DESCRIBE_FLAG (RAID1 ,"raid1" ); +DESCRIBE_FLAG (RAID1C3 ,"raid1c3" );DESCRIBE_FLAG (DUP ,"dup" );DESCRIBE_FLAG (RAID10 ,"raid10" );DESCRIBE_FLAG (RAID5 ,"raid5" );
fs/btrfs/scrub.c +2 -1
@@ -3388,7 +3388,8 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, offset = map->stripe_len * (num / map->sub_stripes); increment = map->stripe_len * factor; mirror_num = num % map->sub_stripes +1 ; - }else if (map->type &BTRFS_BLOCK_GROUP_RAID1 ) { + }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 )) { increment = map->stripe_len; mirror_num = num % map->num_stripes +1 ; }else if (map->type &BTRFS_BLOCK_GROUP_DUP ) {
fs/btrfs/super.c +3 -0
@@ -1977,6 +1977,9 @@ static int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info, }else if (type &BTRFS_BLOCK_GROUP_RAID1 ) { min_stripes =2 ; num_stripes =2 ; + }else if (type &BTRFS_BLOCK_GROUP_RAID1C3 ) { + min_stripes =3 ; + num_stripes =3 ; }else if (type &BTRFS_BLOCK_GROUP_RAID10 ) { min_stripes =4 ; num_stripes =4 ;
fs/btrfs/volumes.c +34 -6
@@ -116,6 +116,18 @@ const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = { .bg_flag = BTRFS_BLOCK_GROUP_RAID6, .mindev_error = BTRFS_ERROR_DEV_RAID6_MIN_NOT_MET, }, + [BTRFS_RAID_RAID1C3] = { + .sub_stripes = 1, + .dev_stripes = 1, + .devs_max = 0, + .devs_min = 3, + .tolerated_failures = 2, + .devs_increment = 3, + .ncopies = 3, + .raid_name = "raid1c3", + .bg_flag = BTRFS_BLOCK_GROUP_RAID1C3, + .mindev_error = BTRFS_ERROR_DEV_RAID1C3_MIN_NOT_MET, + }, };const char *get_raid_name (enum btrfs_raid_types type)
@@ -3336,6 +3348,8 @@ static int chunk_drange_filter(struct extent_buffer *leaf,if (btrfs_chunk_type (leaf, chunk) & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 )) { factor = num_stripes /2 ; + }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID1C3 ) { + factor = num_stripes /3 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID5 ) { factor = num_stripes -1 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID6 ) {
@@ -3822,7 +3836,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,if (num_devices >1 ) allowed |= (BTRFS_BLOCK_GROUP_RAID0 |BTRFS_BLOCK_GROUP_RAID1 );if (num_devices >2 ) - allowed |=BTRFS_BLOCK_GROUP_RAID5 ; + allowed |=BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID1C3 ;if (num_devices >3 ) allowed |= (BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID6 );
@@ -3856,6 +3870,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,/* allow to reduce meta or sys integrity only if force set */ allowed =BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 ;
@@ -4787,8 +4802,11 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,sort (devices_info, ndevs,sizeof (struct btrfs_device_info ), btrfs_cmp_device_info,NULL ); -/* round down to number of usable stripes */ - ndevs =round_down (ndevs, devs_increment); +/* +* Round down to number of usable stripes, devs_increment can be any +* number so we can't use round_down() +*/ + ndevs -= ndevs % devs_increment;if (ndevs < devs_min) { ret = -ENOSPC ;
@@ -5075,6 +5093,8 @@ static inline int btrfs_chunk_max_errors(struct map_lookup *map) BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_DUP)) { max_errors =1 ; + } else if (map->type & BTRFS_BLOCK_GROUP_RAID1C3) { + max_errors =2 ; } else if (map->type & BTRFS_BLOCK_GROUP_RAID6) { max_errors =2 ; } else {
@@ -5163,7 +5183,8 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len)return 1 ; map = em->map_lookup; -if (map->type & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 )) +if (map->type & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 )) ret = map->num_stripes;else if (map->type &BTRFS_BLOCK_GROUP_RAID10 ) ret = map->sub_stripes;
@@ -5237,7 +5258,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,struct btrfs_device *srcdev;ASSERT ((map->type & - (BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 ))); + (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID10 )));if (map->type &BTRFS_BLOCK_GROUP_RAID10 ) num_stripes = map->sub_stripes;
@@ -5427,6 +5450,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info,div_u64_rem (stripe_nr_end -1 , factor, &last_stripe); last_stripe *= sub_stripes; }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_DUP )) { num_stripes = map->num_stripes; }else {
@@ -5792,7 +5816,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, &stripe_index);if (!need_full_stripe (op)) mirror_num =1 ; - }else if (map->type &BTRFS_BLOCK_GROUP_RAID1 ) { + }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 )) {if (need_full_stripe (op)) num_stripes = map->num_stripes;else if (mirror_num)
@@ -6441,6 +6466,7 @@ static int btrfs_check_chunk_valid(struct btrfs_fs_info *fs_info, } if ((type & BTRFS_BLOCK_GROUP_RAID10 && sub_stripes != 2) || (type & BTRFS_BLOCK_GROUP_RAID1 && num_stripes < 1) || + (type & BTRFS_BLOCK_GROUP_RAID1C3 && num_stripes < 3) || (type & BTRFS_BLOCK_GROUP_RAID5 && num_stripes < 2) || (type & BTRFS_BLOCK_GROUP_RAID6 && num_stripes < 3) || (type & BTRFS_BLOCK_GROUP_DUP && num_stripes > 2) ||
@@ -7389,5 +7415,7 @@ int btrfs_bg_type_to_factor(u64 flags)if (flags & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 ))return 2 ; +if (flags &BTRFS_BLOCK_GROUP_RAID1C3 ) +return 3 ;return 1 ; }
fs/btrfs/volumes.h +2 -0
@@ -537,6 +537,8 @@ static inline enum btrfs_raid_types btrfs_bg_flags_to_raid_index(u64 flags)return BTRFS_RAID_RAID10 ; else if (flags & BTRFS_BLOCK_GROUP_RAID1) return BTRFS_RAID_RAID1; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C3) + return BTRFS_RAID_RAID1C3; else if (flags & BTRFS_BLOCK_GROUP_DUP) return BTRFS_RAID_DUP; else if (flags & BTRFS_BLOCK_GROUP_RAID0)
include/uapi/linux/btrfs.h +2 -1
@@ -825,7 +825,8 @@ enum btrfs_err_code {BTRFS_ERROR_DEV_TGT_REPLACE ,BTRFS_ERROR_DEV_MISSING_NOT_FOUND ,BTRFS_ERROR_DEV_ONLY_WRITABLE , - BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS +BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS , + BTRFS_ERROR_DEV_RAID1C3_MIN_NOT_MET, };#define BTRFS_IOC_SNAP_CREATE _IOW(BTRFS_IOCTL_MAGIC, 1, \
include/uapi/linux/btrfs_tree.h +3 -0
@@ -836,6 +836,7 @@ struct btrfs_dev_replace_item {#define BTRFS_BLOCK_GROUP_RAID10 (1ULL << 6)#define BTRFS_BLOCK_GROUP_RAID5 (1ULL << 7)#define BTRFS_BLOCK_GROUP_RAID6 (1ULL << 8) +#define BTRFS_BLOCK_GROUP_RAID1C3 (1ULL << 9)#define BTRFS_BLOCK_GROUP_RESERVED (BTRFS_AVAIL_ALLOC_BIT_SINGLE | \ BTRFS_SPACE_INFO_GLOBAL_RSV)
@@ -847,6 +848,7 @@ enum btrfs_raid_types {BTRFS_RAID_SINGLE ,BTRFS_RAID_RAID5 ,BTRFS_RAID_RAID6 , +BTRFS_RAID_RAID1C3 , BTRFS_NR_RAID_TYPES };
@@ -856,6 +858,7 @@ enum btrfs_raid_types {#define BTRFS_BLOCK_GROUP_PROFILE_MASK (BTRFS_BLOCK_GROUP_RAID0 | \ BTRFS_BLOCK_GROUP_RAID1 | \ + BTRFS_BLOCK_GROUP_RAID1C3 | \ BTRFS_BLOCK_GROUP_RAID5 | \ BTRFS_BLOCK_GROUP_RAID6 | \ BTRFS_BLOCK_GROUP_DUP | \
Signature · 1 line
2.18.0
Re: [PATCH 2/4] btrfs: add support for 3-copy replication (raid1c3)
As general comment, good to hear that something is moving around raid5/6 + write hole and multiple mirroring. However I am guessing if this is time to simplify the RAID code. There are a lot of "if" which could be avoided using the values stored in the array "btrfs_raid_array[]".
Below some comments:
Quoted 21 lines · David Sterba
Add new block group profile to store 3 copies in a simliar way that current RAID1 does. The profile name is temporary and may change in the future.
Signed-off-by David Sterba <[email protected]>--- fs/btrfs/extent-tree.c | 6 +++++ fs/btrfs/relocation.c | 1 + fs/btrfs/scrub.c | 3 ++- fs/btrfs/super.c | 3 +++ fs/btrfs/volumes.c | 40 ++++++++++++++++++++++++++++----- fs/btrfs/volumes.h | 2 ++ include/uapi/linux/btrfs.h | 3 ++- include/uapi/linux/btrfs_tree.h | 3 +++ 8 files changed, 53 insertions(+), 8 deletions(-)fs/btrfs/extent-tree.c +0
-0
fs/btrfs/extent-tree.c static noinline int find_free_extent(struct btrfs_fs_info *fs_info, · line 7533
Quoted 7 lines
fs/btrfs/extent-tree.c +1
-0@@ -7527,6 +7527,7 @@ static noinline int find_free_extent(struct btrfs_fs_info *fs_info,if (!block_group_bits (block_group, flags)) {u64 extra =BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_RAID10 ;
"extra" could be created iterating on btrfs_raid_array[] and considering only the item with ncopies > 1; or we could add
#define BTRFS_BLOCK_GROUP_REDUNDANCY (BTRFS_BLOCK_GROUP_DUP| .....)
This constant could be used also below
Quoted 10 lines
fs/btrfs/extent-tree.c +2
-0@@ -9330,6 +9331,8 @@ static u64 update_block_group_flags(struct btrfs_fs_info *fs_info, u64 flags) num_devices = fs_info->fs_devices->rw_devices; +ASSERT (!(flags &BTRFS_BLOCK_GROUP_RAID1C3 )); + stripped =BTRFS_BLOCK_GROUP_RAID0 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 ;
fs/btrfs/extent-tree.c int btrfs_can_relocate(struct btrfs_fs_info *fs_info, u64 bytenr) · line 9657
Quoted 8 lines
fs/btrfs/extent-tree.c +2
-0@@ -9647,6 +9650,8 @@ int btrfs_can_relocate(struct btrfs_fs_info *fs_info, u64 bytenr) min_free >>=1 ; }else if (index ==BTRFS_RAID_RAID1 ) { dev_min =2 ; + }else if (index ==BTRFS_RAID_RAID1C3 ) { + dev_min =3 ; }else if (index ==BTRFS_RAID_DUP ) {/* Multiply by 2 */ min_free <<=1 ;
The "if"s above could be simplified as:
dev_min = btrfs_raid_array[index].devs_min; if (index == BTRFS_RAID_DUP) min_free <<= 1; else if (index == BTRFS_RAID_RAID0) min_free = div64_u64(min_free, dev_min); else if (index == BTRFS_RAID_RAID10) min_free >>= 1;
fs/btrfs/extent-tree.c int btrfs_read_block_groups(struct btrfs_fs_info *info) · line 10152
Quoted 8 lines
fs/btrfs/extent-tree.c +1
-0@@ -10141,6 +10146,7 @@ int btrfs_read_block_groups(struct btrfs_fs_info *info)if (!(get_alloc_profile (info, space_info->flags) & (BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_DUP )))
See above about BTRFS_BLOCK_GROUP_REDUNDANCY
fs/btrfs/relocation.c static void describe_relocation(struct btrfs_fs_info *fs_info, · line 4339
Quoted 6 lines
fs/btrfs/relocation.c +0
-0@@ -4339,6 +4339,7 @@ static void describe_relocation(struct btrfs_fs_info *fs_info,DESCRIBE_FLAG (METADATA ,"metadata" );
The code below, could be performed searching in the array btrfs_raid_array[] instead of checking each possibility
Quoted 62 lines
fs/btrfs/relocation.c +1
-0DESCRIBE_FLAG (RAID0 ,"raid0" );DESCRIBE_FLAG (RAID1 ,"raid1" ); +DESCRIBE_FLAG (RAID1C3 ,"raid1c3" );DESCRIBE_FLAG (DUP ,"dup" );DESCRIBE_FLAG (RAID10 ,"raid10" );DESCRIBE_FLAG (RAID5 ,"raid5" );fs/btrfs/scrub.c +2
-1@@ -3388,7 +3388,8 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, offset = map->stripe_len * (num / map->sub_stripes); increment = map->stripe_len * factor; mirror_num = num % map->sub_stripes +1 ; - }else if (map->type &BTRFS_BLOCK_GROUP_RAID1 ) { + }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 )) { increment = map->stripe_len; mirror_num = num % map->num_stripes +1 ; }else if (map->type &BTRFS_BLOCK_GROUP_DUP ) {fs/btrfs/super.c +3
-0@@ -1977,6 +1977,9 @@ static int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info, }else if (type &BTRFS_BLOCK_GROUP_RAID1 ) { min_stripes =2 ; num_stripes =2 ; + }else if (type &BTRFS_BLOCK_GROUP_RAID1C3 ) { + min_stripes =3 ; + num_stripes =3 ; }else if (type &BTRFS_BLOCK_GROUP_RAID10 ) { min_stripes =4 ; num_stripes =4 ;fs/btrfs/volumes.c +12
-0@@ -116,6 +116,18 @@ const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = { .bg_flag = BTRFS_BLOCK_GROUP_RAID6, .mindev_error = BTRFS_ERROR_DEV_RAID6_MIN_NOT_MET, }, + [BTRFS_RAID_RAID1C3] = { + .sub_stripes = 1, + .dev_stripes = 1, + .devs_max = 0, + .devs_min = 3, + .tolerated_failures = 2, + .devs_increment = 3, + .ncopies = 3, + .raid_name = "raid1c3", + .bg_flag = BTRFS_BLOCK_GROUP_RAID1C3, + .mindev_error = BTRFS_ERROR_DEV_RAID1C3_MIN_NOT_MET, + }, };const char *get_raid_name (enum btrfs_raid_types type)
fs/btrfs/volumes.c static int chunk_drange_filter(struct extent_buffer *leaf, · line 3355
Quoted 8 lines
fs/btrfs/volumes.c +2
-0@@ -3336,6 +3348,8 @@ static int chunk_drange_filter(struct extent_buffer *leaf,if (btrfs_chunk_type (leaf, chunk) & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 )) { factor = num_stripes /2 ; + }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID1C3 ) { + factor = num_stripes /3 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID5 ) { factor = num_stripes -1 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID6 ) {
May be time to add two other fields to the btrfs_raid_array[] array ? I.e.:
factor_div and factor_sub
so factor could be computed as
index = btrfs_bg_flags_to_raid_index(btrfs_chunk_type(leaf, chunk))
factor = num_stripes / btrfs_raid_array[index].factor_div - btrfs_raid_array[index].factor_sub;
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3842
Quoted 9 lines
fs/btrfs/volumes.c +1
-1@@ -3822,7 +3836,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,if (num_devices >1 ) allowed |= (BTRFS_BLOCK_GROUP_RAID0 |BTRFS_BLOCK_GROUP_RAID1 );if (num_devices >2 ) - allowed |=BTRFS_BLOCK_GROUP_RAID5 ; + allowed |=BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID1C3 ;if (num_devices >3 ) allowed |= (BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID6 );
The "if"s below could be replaced by a search in the btrfs_raid_array[]
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3876
Quoted 8 lines
fs/btrfs/volumes.c +1
-0@@ -3856,6 +3870,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,/* allow to reduce meta or sys integrity only if force set */ allowed =BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 ;
See above about BTRFS_BLOCK_GROUP_REDUNDANCY
Quoted 15 lines
fs/btrfs/volumes.c +5
-2@@ -4787,8 +4802,11 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,sort (devices_info, ndevs,sizeof (struct btrfs_device_info ), btrfs_cmp_device_info,NULL ); -/* round down to number of usable stripes */ - ndevs =round_down (ndevs, devs_increment); +/* +* Round down to number of usable stripes, devs_increment can be any +* number so we can't use round_down() +*/ + ndevs -= ndevs % devs_increment;if (ndevs < devs_min) { ret = -ENOSPC ;
fs/btrfs/volumes.c static inline int btrfs_chunk_max_errors(struct map_lookup *map) · line 5100
Quoted 8 lines
fs/btrfs/volumes.c +2
-0@@ -5075,6 +5093,8 @@ static inline int btrfs_chunk_max_errors(struct map_lookup *map) BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_DUP)) { max_errors =1 ; + } else if (map->type & BTRFS_BLOCK_GROUP_RAID1C3) { + max_errors =2 ; } else if (map->type & BTRFS_BLOCK_GROUP_RAID6) { max_errors =2 ; } else {
Even in this case the ifs above could be replaced with something like:
index = btrfs_bg_flags_to_raid_index(map->type) max_errors = btrfs_raid_array[index].ncopies-1;
fs/btrfs/volumes.c int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5190
Quoted 10 lines
fs/btrfs/volumes.c +2
-1@@ -5163,7 +5183,8 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len)return 1 ; map = em->map_lookup; -if (map->type & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 )) +if (map->type & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 )) ret = map->num_stripes;else if (map->type &BTRFS_BLOCK_GROUP_RAID10 ) ret = map->sub_stripes;
With the exception of RAID6 case (which I don't understand), the ifs above could be replaced with
index = btrfs_bg_flags_to_raid_index(map->type) ret = btrfs_raid_array[index].ncopies
Quoted 30 lines
fs/btrfs/volumes.c +6
-2@@ -5237,7 +5258,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,struct btrfs_device *srcdev;ASSERT ((map->type & - (BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 ))); + (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID10 )));if (map->type &BTRFS_BLOCK_GROUP_RAID10 ) num_stripes = map->sub_stripes;@@ -5427,6 +5450,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info,div_u64_rem (stripe_nr_end -1 , factor, &last_stripe); last_stripe *= sub_stripes; }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_DUP )) { num_stripes = map->num_stripes; }else {@@ -5792,7 +5816,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, &stripe_index);if (!need_full_stripe (op)) mirror_num =1 ; - }else if (map->type &BTRFS_BLOCK_GROUP_RAID1 ) { + }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | +BTRFS_BLOCK_GROUP_RAID1C3 )) {if (need_full_stripe (op)) num_stripes = map->num_stripes;else if (mirror_num)
fs/btrfs/volumes.c static int btrfs_check_chunk_valid(struct btrfs_fs_info *fs_info, · line 6472
Quoted 7 lines
fs/btrfs/volumes.c +1
-0@@ -6441,6 +6466,7 @@ static int btrfs_check_chunk_valid(struct btrfs_fs_info *fs_info, } if ((type & BTRFS_BLOCK_GROUP_RAID10 && sub_stripes != 2) || (type & BTRFS_BLOCK_GROUP_RAID1 && num_stripes < 1) || + (type & BTRFS_BLOCK_GROUP_RAID1C3 && num_stripes < 3) || (type & BTRFS_BLOCK_GROUP_RAID5 && num_stripes < 2) || (type & BTRFS_BLOCK_GROUP_RAID6 && num_stripes < 3) || (type & BTRFS_BLOCK_GROUP_DUP && num_stripes > 2) ||
The check above could be translate in specific check for BTRFS_BLOCK_GROUP_DUP and BTRFS_BLOCK_GROUP_RAID10; for the other cases we could check that num_stripes is >= devs_min
fs/btrfs/volumes.c int btrfs_bg_type_to_factor(u64 flags) · line 7421
Quoted 8 lines
fs/btrfs/volumes.c +2
-0@@ -7389,5 +7415,7 @@ int btrfs_bg_type_to_factor(u64 flags)if (flags & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID10 ))return 2 ; +if (flags &BTRFS_BLOCK_GROUP_RAID1C3 ) +return 3 ;return 1 ; }
Even the function above could be replaced with something more general
int btrfs_bg_type_to_factor(u64 flags) {
if (flags & (BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_RAID6))
return 1;
return btrfs_raid_array[btrfs_bg_flags_to_raid_index(flags)]
}fs/btrfs/volumes.h static inline enum btrfs_raid_types btrfs_bg_flags_to_raid_index(u64 flags) · line 544
Quoted 13 lines
fs/btrfs/volumes.h +2
-0@@ -537,6 +537,8 @@ static inline enum btrfs_raid_types btrfs_bg_flags_to_raid_index(u64 flags)return BTRFS_RAID_RAID10 ; else if (flags & BTRFS_BLOCK_GROUP_RAID1) return BTRFS_RAID_RAID1; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C3) + return BTRFS_RAID_RAID1C3; else if (flags & BTRFS_BLOCK_GROUP_DUP) return BTRFS_RAID_DUP; else if (flags & BTRFS_BLOCK_GROUP_RAID0)
What about iterating on btrfs_raid_array[] in the above function
Quoted 43 lines
include/uapi/linux/btrfs.h +2
-1@@ -825,7 +825,8 @@ enum btrfs_err_code {BTRFS_ERROR_DEV_TGT_REPLACE ,BTRFS_ERROR_DEV_MISSING_NOT_FOUND ,BTRFS_ERROR_DEV_ONLY_WRITABLE , - BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS +BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS , + BTRFS_ERROR_DEV_RAID1C3_MIN_NOT_MET, };#define BTRFS_IOC_SNAP_CREATE _IOW(BTRFS_IOCTL_MAGIC, 1, \include/uapi/linux/btrfs_tree.h +3
-0@@ -836,6 +836,7 @@ struct btrfs_dev_replace_item {#define BTRFS_BLOCK_GROUP_RAID10 (1ULL << 6)#define BTRFS_BLOCK_GROUP_RAID5 (1ULL << 7)#define BTRFS_BLOCK_GROUP_RAID6 (1ULL << 8) +#define BTRFS_BLOCK_GROUP_RAID1C3 (1ULL << 9)#define BTRFS_BLOCK_GROUP_RESERVED (BTRFS_AVAIL_ALLOC_BIT_SINGLE | \ BTRFS_SPACE_INFO_GLOBAL_RSV)@@ -847,6 +848,7 @@ enum btrfs_raid_types {BTRFS_RAID_SINGLE ,BTRFS_RAID_RAID5 ,BTRFS_RAID_RAID6 , +BTRFS_RAID_RAID1C3 , BTRFS_NR_RAID_TYPES };@@ -856,6 +858,7 @@ enum btrfs_raid_types {#define BTRFS_BLOCK_GROUP_PROFILE_MASK (BTRFS_BLOCK_GROUP_RAID0 | \ BTRFS_BLOCK_GROUP_RAID1 | \ + BTRFS_BLOCK_GROUP_RAID1C3 | \ BTRFS_BLOCK_GROUP_RAID5 | \ BTRFS_BLOCK_GROUP_RAID6 | \ BTRFS_BLOCK_GROUP_DUP | \
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Re: [PATCH 2/4] btrfs: add support for 3-copy replication (raid1c3)
Quoted 3 lines · Goffredo Baroncelli
As general comment, good to hear that something is moving around raid5/6 + write hole and multiple mirroring. However I am guessing if this is time to simplify the RAID code. There are a lot of "if" which could be avoided using the values stored in the array "btrfs_raid_array[]".
I absolutely agree and had the same impression during implementing the feature. For this patchset I did only a minimal prep work, the suggestions you give below make sense to me.
Enhancing the table would make a lot of code go away and just use one formula to calculate the results that are now opencoded. I'll be going through the raid code so I'll get to the cleanups eventually.
Quoted 14 lines
Below some comments:
fs/btrfs/volumes.c static inline int btrfs_chunk_max_errors(struct map_lookup *map) · line 5100
Quoted 9 lines
fs/btrfs/volumes.c +2
-0@@ -5075,6 +5093,8 @@ static inline int btrfs_chunk_max_errors(struct map_lookup *map) BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_DUP)) { max_errors =1 ; + } else if (map->type & BTRFS_BLOCK_GROUP_RAID1C3) { + max_errors =2 ; } else if (map->type & BTRFS_BLOCK_GROUP_RAID6) { max_errors =2 ; } else {Even in this case the ifs above could be replaced with something like:
Quoted 2 lines
index = btrfs_bg_flags_to_raid_index(map->type) max_errors = btrfs_raid_array[index].ncopies-1;
There's .tolerated_failures that should equal ncopies - 1 in general, but does not for DUP so the semantics of the function and caller needs to be verified.
David Sterba
maintainerPATCH 3/4 btrfs: add support for 4-copy replication (raid1c4)
In reply to David SterbaAdd new block group profile to store 4 copies in a simliar way that current RAID1 does. The profile name is temporary and may change in the future.
--- fs/btrfs/extent-tree.c | 3 +++ fs/btrfs/scrub.c | 3 ++- fs/btrfs/super.c | 3 +++ fs/btrfs/volumes.c | 28 ++++++++++++++++++++++++++-- fs/btrfs/volumes.h | 2 ++ include/uapi/linux/btrfs.h | 1 + include/uapi/linux/btrfs_tree.h | 3 +++ 7 files changed, 40 insertions(+), 3 deletions(-)
fs/btrfs/extent-tree.c +3 -0
@@ -7528,6 +7528,7 @@ static noinline int find_free_extent(struct btrfs_fs_info *fs_info,u64 extra =BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 |BTRFS_BLOCK_GROUP_RAID10 ;
@@ -9332,6 +9333,7 @@ static u64 update_block_group_flags(struct btrfs_fs_info *fs_info, u64 flags) num_devices = fs_info->fs_devices->rw_devices;ASSERT (!(flags &BTRFS_BLOCK_GROUP_RAID1C3 )); +ASSERT (!(flags &BTRFS_BLOCK_GROUP_RAID1C4 )); stripped =BTRFS_BLOCK_GROUP_RAID0 | BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_RAID6 |
@@ -10147,6 +10149,7 @@ int btrfs_read_block_groups(struct btrfs_fs_info *info) (BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 | BTRFS_BLOCK_GROUP_DUP)))
fs/btrfs/scrub.c +2 -1
@@ -3389,7 +3389,8 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx, increment = map->stripe_len * factor; mirror_num = num % map->sub_stripes +1 ; }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID1C3 )) { +BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 )) { increment = map->stripe_len; mirror_num = num % map->num_stripes +1 ; }else if (map->type &BTRFS_BLOCK_GROUP_DUP ) {
fs/btrfs/super.c +3 -0
@@ -1980,6 +1980,9 @@ static int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info, }else if (type &BTRFS_BLOCK_GROUP_RAID1C3 ) { min_stripes =3 ; num_stripes =3 ; + }else if (type &BTRFS_BLOCK_GROUP_RAID1C4 ) { + min_stripes =4 ; + num_stripes =4 ; }else if (type &BTRFS_BLOCK_GROUP_RAID10 ) { min_stripes =4 ; num_stripes =4 ;
fs/btrfs/volumes.c +26 -2
@@ -128,6 +128,18 @@ const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = { .bg_flag = BTRFS_BLOCK_GROUP_RAID1C3, .mindev_error = BTRFS_ERROR_DEV_RAID1C3_MIN_NOT_MET, }, + [BTRFS_RAID_RAID1C4] = { + .sub_stripes = 1, + .dev_stripes = 1, + .devs_max = 0, + .devs_min = 4, + .tolerated_failures = 3, + .devs_increment = 4, + .ncopies = 4, + .raid_name = "raid1c4", + .bg_flag = BTRFS_BLOCK_GROUP_RAID1C4, + .mindev_error = BTRFS_ERROR_DEV_RAID1C4_MIN_NOT_MET, + }, };const char *get_raid_name (enum btrfs_raid_types type)
@@ -3350,6 +3362,8 @@ static int chunk_drange_filter(struct extent_buffer *leaf, factor = num_stripes /2 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID1C3 ) { factor = num_stripes /3 ; + }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID1C4 ) { + factor = num_stripes /4 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID5 ) { factor = num_stripes -1 ; }else if (btrfs_chunk_type (leaf, chunk) &BTRFS_BLOCK_GROUP_RAID6 ) {
@@ -3839,6 +3853,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info, allowed |=BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID1C3 ;if (num_devices >3 ) allowed |= (BTRFS_BLOCK_GROUP_RAID10 | +BTRFS_BLOCK_GROUP_RAID1C4 |BTRFS_BLOCK_GROUP_RAID6 );if (validate_convert_profile (&bctl->data, allowed)) {int index =btrfs_bg_flags_to_raid_index (bctl->data.target);
@@ -3871,6 +3886,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,/* allow to reduce meta or sys integrity only if force set */ allowed =BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 |BTRFS_BLOCK_GROUP_RAID10 |BTRFS_BLOCK_GROUP_RAID5 |BTRFS_BLOCK_GROUP_RAID6 ;
@@ -5095,6 +5111,8 @@ static inline int btrfs_chunk_max_errors(struct map_lookup *map) max_errors =1 ; }else if (map->type &BTRFS_BLOCK_GROUP_RAID1C3 ) { max_errors =2 ; + }else if (map->type &BTRFS_BLOCK_GROUP_RAID1C4 ) { + max_errors =3 ; }else if (map->type &BTRFS_BLOCK_GROUP_RAID6 ) { max_errors =2 ; }else {
@@ -5184,7 +5202,7 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len) map = em->map_lookup;if (map->type & (BTRFS_BLOCK_GROUP_DUP |BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID1C3 )) +BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID1C4 )) ret = map->num_stripes;else if (map->type &BTRFS_BLOCK_GROUP_RAID10 ) ret = map->sub_stripes;
@@ -5260,6 +5278,7 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,ASSERT ((map->type & (BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 |BTRFS_BLOCK_GROUP_RAID10 )));if (map->type &BTRFS_BLOCK_GROUP_RAID10 )
@@ -5451,6 +5470,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, last_stripe *= sub_stripes; }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 |BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 |BTRFS_BLOCK_GROUP_DUP )) { num_stripes = map->num_stripes; }else {
@@ -5817,7 +5837,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info,if (!need_full_stripe (op)) mirror_num =1 ; }else if (map->type & (BTRFS_BLOCK_GROUP_RAID1 | -BTRFS_BLOCK_GROUP_RAID1C3 )) { +BTRFS_BLOCK_GROUP_RAID1C3 | +BTRFS_BLOCK_GROUP_RAID1C4 )) {if (need_full_stripe (op)) num_stripes = map->num_stripes;else if (mirror_num)
@@ -6467,6 +6488,7 @@ static int btrfs_check_chunk_valid(struct btrfs_fs_info *fs_info, if ((type & BTRFS_BLOCK_GROUP_RAID10 && sub_stripes != 2) || (type & BTRFS_BLOCK_GROUP_RAID1 && num_stripes < 1) || (type & BTRFS_BLOCK_GROUP_RAID1C3 && num_stripes < 3) || + (type & BTRFS_BLOCK_GROUP_RAID1C4 && num_stripes < 4) || (type & BTRFS_BLOCK_GROUP_RAID5 && num_stripes < 2) || (type & BTRFS_BLOCK_GROUP_RAID6 && num_stripes < 3) || (type & BTRFS_BLOCK_GROUP_DUP && num_stripes > 2) ||
@@ -7417,5 +7439,7 @@ int btrfs_bg_type_to_factor(u64 flags)return 2 ;if (flags &BTRFS_BLOCK_GROUP_RAID1C3 )return 3 ; +if (flags &BTRFS_BLOCK_GROUP_RAID1C4 ) +return 4 ;return 1 ; }
fs/btrfs/volumes.h +2 -0
@@ -539,6 +539,8 @@ static inline enum btrfs_raid_types btrfs_bg_flags_to_raid_index(u64 flags)return BTRFS_RAID_RAID1 ; else if (flags & BTRFS_BLOCK_GROUP_RAID1C3) return BTRFS_RAID_RAID1C3; + else if (flags & BTRFS_BLOCK_GROUP_RAID1C4) + return BTRFS_RAID_RAID1C4; else if (flags & BTRFS_BLOCK_GROUP_DUP) return BTRFS_RAID_DUP; else if (flags & BTRFS_BLOCK_GROUP_RAID0)
include/uapi/linux/btrfs.h +1 -0
@@ -827,6 +827,7 @@ enum btrfs_err_code {BTRFS_ERROR_DEV_ONLY_WRITABLE ,BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS ,BTRFS_ERROR_DEV_RAID1C3_MIN_NOT_MET , + BTRFS_ERROR_DEV_RAID1C4_MIN_NOT_MET, };#define BTRFS_IOC_SNAP_CREATE _IOW(BTRFS_IOCTL_MAGIC, 1, \
include/uapi/linux/btrfs_tree.h +3 -0
@@ -837,6 +837,7 @@ struct btrfs_dev_replace_item {#define BTRFS_BLOCK_GROUP_RAID5 (1ULL << 7)#define BTRFS_BLOCK_GROUP_RAID6 (1ULL << 8)#define BTRFS_BLOCK_GROUP_RAID1C3 (1ULL << 9) +#define BTRFS_BLOCK_GROUP_RAID1C4 (1ULL << 10)#define BTRFS_BLOCK_GROUP_RESERVED (BTRFS_AVAIL_ALLOC_BIT_SINGLE | \ BTRFS_SPACE_INFO_GLOBAL_RSV)
@@ -849,6 +850,7 @@ enum btrfs_raid_types {BTRFS_RAID_RAID5 ,BTRFS_RAID_RAID6 ,BTRFS_RAID_RAID1C3 , +BTRFS_RAID_RAID1C4 , BTRFS_NR_RAID_TYPES };
@@ -859,6 +861,7 @@ enum btrfs_raid_types {#define BTRFS_BLOCK_GROUP_PROFILE_MASK (BTRFS_BLOCK_GROUP_RAID0 | \ BTRFS_BLOCK_GROUP_RAID1 | \ BTRFS_BLOCK_GROUP_RAID1C3 | \ + BTRFS_BLOCK_GROUP_RAID1C4 | \ BTRFS_BLOCK_GROUP_RAID5 | \ BTRFS_BLOCK_GROUP_RAID6 | \ BTRFS_BLOCK_GROUP_DUP | \
Signature · 1 line
2.18.0
David Sterba
maintainerPATCH 4/4 btrfs: add incompatibility bit for extended raid features
In reply to David SterbaThis bit will cover all newly added features of RAID:
- 3 copy replication
- 4 copy replication
- configurable stripe length
- triple parity
- raid56 log
--- fs/btrfs/ctree.h | 1 + fs/btrfs/sysfs.c | 2 ++ fs/btrfs/volumes.c | 9 +++++++++ include/uapi/linux/btrfs.h | 8 ++++++++ 4 files changed, 20 insertions(+)
fs/btrfs/ctree.h +1 -0
@@ -265,6 +265,7 @@ struct btrfs_super_block {BTRFS_FEATURE_INCOMPAT_RAID56 | \BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | \BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | \ +BTRFS_FEATURE_INCOMPAT_EXTENDED_RAID | \ BTRFS_FEATURE_INCOMPAT_NO_HOLES) #define BTRFS_FEATURE_INCOMPAT_SAFE_SET \
fs/btrfs/sysfs.c +2 -0
@@ -194,6 +194,7 @@ BTRFS_FEAT_ATTR_INCOMPAT(raid56, RAID56);BTRFS_FEAT_ATTR_INCOMPAT (skinny_metadata,SKINNY_METADATA );BTRFS_FEAT_ATTR_INCOMPAT (no_holes,NO_HOLES );BTRFS_FEAT_ATTR_COMPAT_RO (free_space_tree,FREE_SPACE_TREE ); +BTRFS_FEAT_ATTR_INCOMPAT (extended_raid,EXTENDED_RAID );static struct attribute *btrfs_supported_feature_attrs[] = {BTRFS_FEAT_ATTR_PTR (mixed_backref),
@@ -207,6 +208,7 @@ static struct attribute *btrfs_supported_feature_attrs[] = {BTRFS_FEAT_ATTR_PTR (skinny_metadata),BTRFS_FEAT_ATTR_PTR (no_holes),BTRFS_FEAT_ATTR_PTR (free_space_tree), +BTRFS_FEAT_ATTR_PTR (extended_raid), NULL };
fs/btrfs/volumes.c +9 -0
@@ -4663,6 +4663,14 @@ static void check_raid56_incompat_flag(struct btrfs_fs_info *info, u64 type)btrfs_set_fs_incompat (info,RAID56 ); } +static void check_extended_raid_incompat_flag (struct btrfs_fs_info *info,u64 type) +{ +if (!(type & (BTRFS_BLOCK_GROUP_RAID1C3 |BTRFS_BLOCK_GROUP_RAID1C4 ))) +return ; + +btrfs_set_fs_incompat (info,EXTENDED_RAID ); +} +#define BTRFS_MAX_DEVS (info) ((BTRFS_MAX_ITEM_SIZE(info) \ - sizeof(struct btrfs_chunk)) \ / sizeof(struct btrfs_stripe) + 1)
@@ -4945,6 +4953,7 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,free_extent_map (em);check_raid56_incompat_flag (info, type); +check_extended_raid_incompat_flag (info, type);kfree (devices_info);return 0 ;
include/uapi/linux/btrfs.h +8 -0
@@ -270,6 +270,14 @@ struct btrfs_ioctl_fs_info_args {#define BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA (1ULL << 8)#define BTRFS_FEATURE_INCOMPAT_NO_HOLES (1ULL << 9) +/* +* More RAID features: +* - RAID1C3 - 3-copy mirroring +* - RAID1C4 - 4-copy mirroring +* - ... +*/ +#define BTRFS_FEATURE_INCOMPAT_EXTENDED_RAID (1ULL << 10) +struct btrfs_ioctl_feature_flags {__u64 compat_flags;__u64 compat_ro_flags;
Signature · 1 line
2.18.0
Quoted 9 lines · David Sterba
An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
As just a humble BTRFS user I agree and really think it is about time to move far away from the RAID terminology. However adding some more descriptive profile names (or at least some aliases) would be much better for the commoners (such as myself).
For example:
Old format / New Format / My suggested alias
SINGLE / 1C / SINGLE DUP / 2CD / DUP (or even MIRRORLOCAL1) RAID0 / 1CmS / STRIPE RAID1 / 2C / MIRROR1 RAID1c3 / 3C / MIRROR2 RAID1c4 / 4C / MIRROR3 RAID10 / 2CmS / STRIPE.MIRROR1
RAID5 / 1CmS1P / STRIPE.PARITY1
RAID6 / 1CmS2P / STRIPE.PARITY2
I find that writing something like "btrfs balance start -dconvert=stripe5.parity2 /mnt" is far less confusing and therefore less error prone than writing "-dconvert=1C5S2P".
While Hugo's suggestion is compact and to the point I would call for expanding that so it is a bit more descriptive and human readable.
So for example : STRIPE<num> where <num> obviously is the same as Hugo proposed - the number of storage devices for the stripe and no <num> would be best to mean 'use max devices'. For PARITY then <num> is obviously required
Keep in mind that most people (...and I am willing to bet even Duncan which probably HAS backups ;) ) get a bit stressed when their storage system is degraded. With that in mind I hope for more elaborate, descriptive and human readable profile names to be used to avoid making mistakes using the "compact" layout.
...and yes, of course this could go both ways. A more compact (and dare I say cryptic) variant can cause people to stop and think before doing something and thus avoid errors,
Now that I made my point I can't help being a bit extra hash, obnoxious and possibly difficult so I would also suggest that Hugo's format could have been changed (dare I say improved?) from....
numCOPIESnumSTRIPESnumPARITY
to.....
REPLICASnum.STRIPESnum.PARITYnum
Which would make the above table look like so:
Old format / My Format / My suggested alias
SINGLE / R0.S0.P0 / SINGLE DUP / R1.S1.P0 / DUP (or even MIRRORLOCAL1) RAID0 / R0.Sm.P0 / STRIPE RAID1 / R1.S0.P0 / MIRROR1 RAID1c3 / R2.S0.P0 / MIRROR2 RAID1c4 / R3.S0.P0 / MIRROR3 RAID10 / R1.Sm.P0 / STRIPE.MIRROR1 RAID5 / R1.Sm.P1 / STRIPE.PARITY1 RAID6 / R1.Sm.P2 / STRIPE.PARITY2
And i think this is much more readable, but others may disagree. And as a side note... from a (hobby) coders perspective this is probably simpler to parse as well.
Quoted 20 lines · waxhead
Quoted 9 lines · David Sterba
An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
As just a humble BTRFS user I agree and really think it is about time to move far away from the RAID terminology. However adding some more descriptive profile names (or at least some aliases) would be much better for the commoners (such as myself).
For example:
Old format / New Format / My suggested alias
SINGLE / 1C / SINGLE
DUP / 2CD / DUP (or even MIRRORLOCAL1)
RAID0 / 1CmS / STRIPE
Quoted 4 lines
RAID1 / 2C / MIRROR1
RAID1c3 / 3C / MIRROR2
RAID1c4 / 4C / MIRROR3
RAID10 / 2CmS / STRIPE.MIRROR1
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive. What about
RAID1 -> MIRROR1
RAID10 -> MIRROR1S
RAID1c3 -> MIRROR2
RAID1c3+striping -> MIRROR2S
and so on...
Quoted 2 lines
RAID5 / 1CmS1P / STRIPE.PARITY1
RAID6 / 1CmS2P / STRIPE.PARITY2
To me these should be called something like
RAID5 -> PARITY1S
RAID6 -> PARITY2S
The S final is due to the fact that usually RAID5/6 spread the data on all available disks
Question #1: for "parity" profiles, does make sense to limit the maximum disks number where the data may be spread ? If the answer is not, we could omit the last S. IMHO it should.
Question #2: historically RAID10 is requires 4 disks. However I am guessing if the stripe could be done on a different number of disks: What about RAID1+Striping on 3 (or 5 disks) ? The key of striping is that every 64k, the data are stored on a different disk....
Quoted 39 lines
I find that writing something like "btrfs balance start -dconvert=stripe5.parity2 /mnt" is far less confusing and therefore less error prone than writing "-dconvert=1C5S2P".
While Hugo's suggestion is compact and to the point I would call for expanding that so it is a bit more descriptive and human readable.
So for example : STRIPE<num> where <num> obviously is the same as Hugo proposed - the number of storage devices for the stripe and no <num> would be best to mean 'use max devices'. For PARITY then <num> is obviously required
Keep in mind that most people (...and I am willing to bet even Duncan which probably HAS backups ;) ) get a bit stressed when their storage system is degraded. With that in mind I hope for more elaborate, descriptive and human readable profile names to be used to avoid making mistakes using the "compact" layout.
...and yes, of course this could go both ways. A more compact (and dare I say cryptic) variant can cause people to stop and think before doing something and thus avoid errors,
Now that I made my point I can't help being a bit extra hash, obnoxious and possibly difficult so I would also suggest that Hugo's format could have been changed (dare I say improved?) from....
numCOPIESnumSTRIPESnumPARITY
to.....
REPLICASnum.STRIPESnum.PARITYnum
Which would make the above table look like so:
Old format / My Format / My suggested alias
SINGLE / R0.S0.P0 / SINGLE
DUP / R1.S1.P0 / DUP (or even MIRRORLOCAL1)
RAID0 / R0.Sm.P0 / STRIPE
RAID1 / R1.S0.P0 / MIRROR1
RAID1c3 / R2.S0.P0 / MIRROR2
RAID1c4 / R3.S0.P0 / MIRROR3
RAID10 / R1.Sm.P0 / STRIPE.MIRROR1
RAID5 / R1.Sm.P1 / STRIPE.PARITY1
RAID6 / R1.Sm.P2 / STRIPE.PARITY2And i think this is much more readable, but others may disagree. And as a side note... from a (hobby) coders perspective this is probably simpler to parse as well.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 38 lines · Goffredo Baroncelli
Quoted 20 lines · waxhead
Quoted 9 lines · David Sterba
An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
As just a humble BTRFS user I agree and really think it is about time to move far away from the RAID terminology. However adding some more descriptive profile names (or at least some aliases) would be much better for the commoners (such as myself).
For example:
Old format / New Format / My suggested alias
SINGLE / 1C / SINGLE
DUP / 2CD / DUP (or even MIRRORLOCAL1)
RAID0 / 1CmS / STRIPEQuoted 4 lines
RAID1 / 2C / MIRROR1
RAID1c3 / 3C / MIRROR2
RAID1c4 / 4C / MIRROR3
RAID10 / 2CmS / STRIPE.MIRROR1Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive. What about
RAID1 -> MIRROR1
RAID10 -> MIRROR1S
RAID1c3 -> MIRROR2
RAID1c3+striping -> MIRROR2Sand so on...
Quoted 2 lines
RAID5 / 1CmS1P / STRIPE.PARITY1
RAID6 / 1CmS2P / STRIPE.PARITY2
Quoted 8 lines
To me these should be called something like
RAID5 -> PARITY1S
RAID6 -> PARITY2SThe S final is due to the fact that usually RAID5/6 spread the data on all available disks
Question #1: for "parity" profiles, does make sense to limit the maximum disks number where the data may be spread ? If the answer is not, we could omit the last S. IMHO it should.
Currently, there is no ability to cap the number of disks that striping can happen across. Ideally, that will change in the future, in which case not only the S will be needed, but also a number indicating how wide the stripe is.
Quoted 1 line
Question #2: historically RAID10 is requires 4 disks. However I am guessing if the stripe could be done on a different number of disks: What about RAID1+Striping on 3 (or 5 disks) ? The key of striping is that every 64k, the data are stored on a different disk....
This is what MD and LVM RAID10 do. They work somewhat differently from what BTRFS calls raid10 (actually, what we currently call raid1 works almost identically to MD and LVM RAID10 when more than 3 disks are involved, except that the chunk size is 1G or larger). Short of drastic internal changes to how that profile works, this isn't likely to happen.
In spite of both of these, there is practical need for indicating the stripe width. Depending on the configuration of the underlying storage, it's fully possible (and sometimes even certain) that you will see chunks with differing stripe widths, so properly reporting the stripe width (in devices, not bytes) is useful for monitoring purposes).
Consider for example a 6-device array using what's currently called a raid10 profile where 2 of the disks are smaller than the other four. On such an array, chunks will span all six disks (resulting in 2 copies striped across 3 disks each) until those two smaller disks are full, at which point new chunks will span only the remaining four disks (resulting in 2 copies striped across 2 disks each).
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense, or purely in the btrfs context (which I suspect), but either way I can't agree.
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
In the global context, just to complete things and mostly for others reading as I feel a bit like a simpleton explaining to the expert here, just as raid10 is shorthand for raid1+0, aka raid0 layered on top of raid1 (normally preferred to raid01 due to rebuild characteristics, and as opposed to raid01, aka raid0+1, aka raid1 on top of raid0, sometimes recommended as btrfs raid1 on top of whatever raid0 here due to btrfs' data integrity characteristics and less optimized performance), so there's also raid51 and raid15, raid61 and raid16, etc, with or without the + symbols, involving mirroring and parity conceptually at two different levels altho they can be combined in a single implementation just as raid10 and raid01 commonly are. These additional layered-raid levels can be used for higher reliability, with differing rebuild and performance characteristics between the two forms depending on which is the top layer.
Quoted 3 lines
Question #1: for "parity" profiles, does make sense to limit the maximum disks number where the data may be spread ? If the answer is not, we could omit the last S. IMHO it should.
As someone else already replied, btrfs doesn't currently have the ability to specify spread limit, but the idea if we're going to change the notation is to allow for the flexibility in the new notation so the feature can be added later without further notation changes.
Why might it make sense to specify spread? At least two possible reasons:
a) (stealing an already posted example) Consider a multi-device layout with two or more device sizes. Someone may want to limit the spread in ordered to keep performance and risk consistent as the smaller devices fill up, limiting further usage to a lower number of devices. If that lower number is specified as the spread originally it'll make things more consistent between the room on all devices case and the room on only some devices case.
b) Limiting spread can change the risk and rebuild performance profiles. Stripes of full width mean all stripes have a strip on each device, so knock a device out and (assuming parity or mirroring) replace it, and all stripes are degraded and must be rebuilt. With less than maximum spread, some stripes won't be stripped to the replaced device, and won't be degraded or need rebuilt, tho assuming the same overall fill, a larger percentage of stripes that /do/ need rebuilt will be on the replaced device. So the risk profile is more "objects" (stripes/chunks/files) affected but less of each object, or less of the total affected, but more of each affected object.
Quoted 4 lines
Question #2: historically RAID10 is requires 4 disks. However I am guessing if the stripe could be done on a different number of disks:
What about RAID1+Striping on 3 (or 5 disks) ? The key of striping is that every 64k, the data are stored on a different disk....
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 12 lines · Duncan
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense, or purely in the btrfs context (which I suspect), but either way I can't agree.
Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have
- striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)
However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
[...]
Quoted 9 lines
Quoted 5 lines
Question #2: historically RAID10 is requires 4 disks. However I am guessing if the stripe could be done on a different number of disks:
What about RAID1+Striping on 3 (or 5 disks) ? The key of striping is that every 64k, the data are stored on a different disk....As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
As implemented in BTRFS, raid1 doesn't have striping.
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 27 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)
Quoted 3 lines
However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
While it's not possible today on (pure) btrfs (it's possible today with md/dm-raid or hardware-raid handling one layer), it's theoretically possible both for btrfs and in general, and it could be added to btrfs in the future, so a notation with the flexibility to allow parity and mirroring together does make sense, and having just that sort of flexibility is exactly why Hugo made the notation proposal he did.
Tho a sensible use-case for mirroring+parity is a different question. I can see a case being made for it if one layer is hardware/firmware raid, but I'm not entirely sure what the use-case for pure-btrfs raid16 or 61 (or 15 or 51) might be, where pure mirroring or pure parity wouldn't arguably be a at least as good a match to the use-case. Perhaps one of the other experts in such things here might help with that.
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Question #2: historically RAID10 is requires 4 disks. However I am guessing if the stripe could be done on a different number of disks:
What about RAID1+Striping on 3 (or 5 disks) ? The key of striping is that every 64k, the data are stored on a different disk....As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Duncan posted on Wed, 18 Jul 2018 07:20:09 +0000 as excerpted:
Quoted 11 lines
Quoted 1 line
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
I forgot this bit...
Similarly, multi-device single is regarded by some to be conceptually equivalent to raid0 with really huge GiB strips/chunks.
(As you may note, "the argument is" and "regarded by some" are distancing phrases. I've seen the argument made on-list, but while I understand the argument and agree with it to some extent, I'm still a bit uncomfortable with it and don't normally make it myself, this thread being a noted exception tho originally I simply repeated what someone else already said in-thread, because I too agree it's stretching things a bit. But it does appear to be a useful conceptual equivalency for some, and I do see the similarity.
Perhaps it's a case of coder's view (no code doing it that way, it's just a coincidental oddity conditional on equal sizes), vs. sysadmin's view (code or not, accidental or not, it's a reasonably accurate high-level description of how it ends up working most of the time with equivalent sized devices).)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 19 lines · Duncan
Duncan posted on Wed, 18 Jul 2018 07:20:09 +0000 as excerpted:
Quoted 11 lines
Quoted 1 line
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
I forgot this bit...
Similarly, multi-device single is regarded by some to be conceptually equivalent to raid0 with really huge GiB strips/chunks.
Quoted 8 lines
(As you may note, "the argument is" and "regarded by some" are distancing phrases. I've seen the argument made on-list, but while I understand the argument and agree with it to some extent, I'm still a bit uncomfortable with it and don't normally make it myself, this thread being a noted exception tho originally I simply repeated what someone else already said in-thread, because I too agree it's stretching things a bit. But it does appear to be a useful conceptual equivalency for some, and I do see the similarity.
If the file is larger than the data chunk size, it _is_ striped, because it spans multiple chunks which are on separate devices. Otherwise, it's more similar to what in GlusterFS is called a 'distributed volume'. In such a Gluster volume, each file is entirely stored on one node (or you have a complete copy on N nodes where N is the number of replicas), with the selection of what node is used for the next file created being based on which node has the most free space.
That said, the main reason I explain single and raid1 the way I do is that I've found it's a much simpler way to explain generically how they work to people who already have storage background but may not care about the specifics.
Quoted 7 lines
Perhaps it's a case of coder's view (no code doing it that way, it's just a coincidental oddity conditional on equal sizes), vs. sysadmin's view (code or not, accidental or not, it's a reasonably accurate high-level description of how it ends up working most of the time with equivalent sized devices).)
Quoted 28 lines · Duncan
Duncan posted on Wed, 18 Jul 2018 07:20:09 +0000 as excerpted:
Quoted 11 lines
Quoted 1 line
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
I forgot this bit...
Similarly, multi-device single is regarded by some to be conceptually equivalent to raid0 with really huge GiB strips/chunks.
(As you may note, "the argument is" and "regarded by some" are distancing phrases. I've seen the argument made on-list, but while I understand the argument and agree with it to some extent, I'm still a bit uncomfortable with it and don't normally make it myself, this thread being a noted exception tho originally I simply repeated what someone else already said in-thread, because I too agree it's stretching things a bit. But it does appear to be a useful conceptual equivalency for some, and I do see the similarity.
Quoted 5 lines
Perhaps it's a case of coder's view (no code doing it that way, it's just a coincidental oddity conditional on equal sizes), vs. sysadmin's view (code or not, accidental or not, it's a reasonably accurate high-level description of how it ends up working most of the time with equivalent sized devices).)
Well, it's an *accurate* observation. It's just not a particularly
*useful* one. :)
Hugo.
Signature · 4 lines
Hugo Mills | I gave up smoking, drinking and sex once. It was the hugo@... carfax.org.uk | scariest 20 minutes of my life. http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 9 lines · Hugo Mills
Quoted 7 lines · Duncan
Duncan posted on Wed, 18 Jul 2018 07:20:09 +0000 as excerpted:
Perhaps it's a case of coder's view (no code doing it that way, it's just a coincidental oddity conditional on equal sizes), vs. sysadmin's view (code or not, accidental or not, it's a reasonably accurate high-level description of how it ends up working most of the time with equivalent sized devices).)
Quoted 5 lines
Well, it's an *accurate* observation. It's just not a particularly
*useful* one. :)Hugo.
A bit off topic perhaps - but I've got to give it a go:
Pretty please with sugar, nuts, a cherry and chocolate sprinkles dipped in syrup and coated with icecream on top , would it not be about time to update your online btrfs-usage calculator (which is insanely useful in so many ways) to support the new modes!?
In fact it would have been a great- / even better as a- cli-tool.
And yes, a while ago I toyed about porting it to C for own use mostly, but never got that far.
Quoted 62 lines · Duncan
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 27 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)Quoted 3 lines
However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
While it's not possible today on (pure) btrfs (it's possible today with md/dm-raid or hardware-raid handling one layer), it's theoretically possible both for btrfs and in general, and it could be added to btrfs in the future, so a notation with the flexibility to allow parity and mirroring together does make sense, and having just that sort of flexibility is exactly why Hugo made the notation proposal he did.
Tho a sensible use-case for mirroring+parity is a different question. I can see a case being made for it if one layer is hardware/firmware raid, but I'm not entirely sure what the use-case for pure-btrfs raid16 or 61 (or 15 or 51) might be, where pure mirroring or pure parity wouldn't arguably be a at least as good a match to the use-case. Perhaps one of the other experts in such things here might help with that.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Question #2: historically RAID10 is requires 4 disks. However I am guessing if the stripe could be done on a different number of disks:
What about RAID1+Striping on 3 (or 5 disks) ? The key of striping is that every 64k, the data are stored on a different disk....As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
Quoted 7 lines
Quoted 1 line
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
Actually, it also behaves like LVM and MD RAID10 for any number of devices greater than 2, though the exact placement may diverge because of BTRFS's concept of different chunk types. In LVM and MD RAID10, each block is stored as two copies, and what disks it ends up on is dependent on the block number modulo the number of disks (so, for 3 disks A, B, and C, block 0 is on A and B, block 1 is on C and A, and block 2 is on B and C, with subsequent blocks following the same pattern). In an idealized model of BTRFS with only one chunk type, you get exactly the same behavior (because BTRFS allocates chunks based on disk utilization, and prefers lower numbered disks to higher ones in the event of a tie).
Quoted 5 lines
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Quoted 30 lines · Duncan
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 27 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)
Quoted 8 lines
Quoted 3 lines
However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
The fact that you can combine striping and mirroring (or pairing) makes sense because you could have a speed gain (see below). [....]
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
Quoted 7 lines
Quoted 1 line
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
The striping concept is based to the fact that if the "stripe size" is small enough you have a speed benefit because the reads may be performed in parallel from different disks. With a "stripe size" of 1GB, it is very unlikely that this would happens.
Quoted 3 lines
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 35 lines · Goffredo Baroncelli
Quoted 34 lines · Duncan
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 30 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Quoted 7 lines
Quoted 4 lines
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
No, there are cases where other configurations make sense.
RAID05 and RAID06 are very widely used, especially on NAS systems where you have lots of disks. The RAID5/6 lower layer mitigates the data loss risk of RAID0, and the RAID0 upper-layer mitigates the rebuild scalability issues of RAID5/6. In fact, this is pretty much the standard recommended configuration for large ZFS arrays that want to use parity RAID. This could be reasonably easily supported to a rudimentary degree in BTRFS by providing the ability to limit the stripe width for the parity profiles.
Some people use RAID50 or RAID60, although they are strictly speaking inferior in almost all respects to RAID05 and RAID06.
RAID01 is also used on occasion, it ends up having the same storage capacity as RAID10, but for some RAID implementations it has a different performance envelope and different rebuild characteristics. Usually, when it is used though, it's software RAID0 on top of hardware RAID1.
RAID51 and RAID61 used to be used, but aren't much now. They provided an easy way to have proper data verification without always having the rebuild overhead of RAID5/6 and without needing to do checksumming. They are pretty much useless for BTRFS, as it can already tell which copy is correct.
RAID15 and RAID16 are a similar case to RAID51 and RAID61, except they might actually make sense in BTRFS to provide a backup means of rebuilding blocks that fail checksum validation if both copies fail.
Quoted 10 lines
The fact that you can combine striping and mirroring (or pairing) makes sense because you could have a speed gain (see below). [....]
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
As implemented in BTRFS, raid1 doesn't have striping.
Quoted 7 lines
Quoted 5 lines
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
The striping concept is based to the fact that if the "stripe size" is small enough you have a speed benefit because the reads may be performed in parallel from different disks.
That's not the only benefit of striping though. The other big one is that you now have one volume that's the combined size of both of the original devices. Striping is arguably better for this even if you're using a large stripe size because it better balances the wear across the devices than simple concatenation.
Quoted 1 line
With a "stripe size" of 1GB, it is very unlikely that this would happens.
That's a pretty big assumption. There are all kinds of access patterns that will still distribute the load reasonably evenly across the constituent devices, even if they don't parallelize things.
If, for example, all your files are 64k or less, and you only read whole files, there's no functional difference between RAID0 with 1GB blocks and RAID0 with 64k blocks. Such a workload is not unusual on a very busy mail-server.
Quoted 5 lines
Quoted 5 lines
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Quoted 9 lines · Austin S. Hemmelgarn
Quoted 8 lines · Goffredo Baroncelli
Quoted 7 lines · Duncan
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 4 lines · Duncan
Quoted 3 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
[...]
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Quoted 7 lines
Quoted 4 lines
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
No, there are cases where other configurations make sense.
Quoted 7 lines
RAID05 and RAID06 are very widely used, especially on NAS systems where you have lots of disks. The RAID5/6 lower layer mitigates the data loss risk of RAID0, and the RAID0 upper-layer mitigates the rebuild scalability issues of RAID5/6. In fact, this is pretty much the standard recommended configuration for large ZFS arrays that want to use parity RAID. This could be reasonably easily supported to a rudimentary degree in BTRFS by providing the ability to limit the stripe width for the parity profiles.
Some people use RAID50 or RAID60, although they are strictly speaking inferior in almost all respects to RAID05 and RAID06.
RAID01 is also used on occasion, it ends up having the same storage capacity as RAID10, but for some RAID implementations it has a different performance envelope and different rebuild characteristics. Usually, when it is used though, it's software RAID0 on top of hardware RAID1.
RAID51 and RAID61 used to be used, but aren't much now. They provided an easy way to have proper data verification without always having the rebuild overhead of RAID5/6 and without needing to do checksumming. They are pretty much useless for BTRFS, as it can already tell which copy is correct.
So until now you are repeating what I told: the only useful raid profile are
- striping
- mirroring
- striping+paring (even limiting the number of disk involved)
- striping+mirroring
Quoted 2 lines
RAID15 and RAID16 are a similar case to RAID51 and RAID61, except they might actually make sense in BTRFS to provide a backup means of rebuilding blocks that fail checksum validation if both copies fail.
If you need further redundancy, it is easy to implement a parity3 and parity4 raid profile instead of stacking a raid6+raid1
Quoted 10 lines
Quoted 10 lines
The fact that you can combine striping and mirroring (or pairing) makes sense because you could have a speed gain (see below). [....]
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
As implemented in BTRFS, raid1 doesn't have striping.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
The striping concept is based to the fact that if the "stripe size" is small enough you have a speed benefit because the reads may be performed in parallel from different disks.
That's not the only benefit of striping though. The other big one is that you now have one volume that's the combined size of both of the original devices. Striping is arguably better for this even if you're using a large stripe size because it better balances the wear across the devices than simple concatenation.
Striping means that the data is interleaved between the disks with a reasonable "block unit". Otherwise which would be the difference between btrfs-raid0 and btrfs-single ?
Quoted 5 lines
Quoted 2 lines
With a "stripe size" of 1GB, it is very unlikely that this would happens.
That's a pretty big assumption. There are all kinds of access patterns that will still distribute the load reasonably evenly across the constituent devices, even if they don't parallelize things.
If, for example, all your files are 64k or less, and you only read whole files, there's no functional difference between RAID0 with 1GB blocks and RAID0 with 64k blocks. Such a workload is not unusual on a very busy mail-server.
I fully agree that 64K may be too much for some workload, however I have to point out that I still find difficult to imagine that you can take advantage of parallel read from multiple disks with a 1GB stripe unit for a *common workload*. Pay attention that btrfs inline in the metadata the small files, so even if the file is smaller than 64k, a 64k read (or more) will be required in order to access it.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 38 lines · Goffredo Baroncelli
Quoted 9 lines · Austin S. Hemmelgarn
Quoted 8 lines · Goffredo Baroncelli
Quoted 7 lines · Duncan
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 4 lines · Duncan
Quoted 3 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
[...]
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Quoted 7 lines
Quoted 4 lines
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
No, there are cases where other configurations make sense.
RAID05 and RAID06 are very widely used, especially on NAS systems where you have lots of disks. The RAID5/6 lower layer mitigates the data loss risk of RAID0, and the RAID0 upper-layer mitigates the rebuild scalability issues of RAID5/6. In fact, this is pretty much the standard recommended configuration for large ZFS arrays that want to use parity RAID. This could be reasonably easily supported to a rudimentary degree in BTRFS by providing the ability to limit the stripe width for the parity profiles.
Some people use RAID50 or RAID60, although they are strictly speaking inferior in almost all respects to RAID05 and RAID06.
RAID01 is also used on occasion, it ends up having the same storage capacity as RAID10, but for some RAID implementations it has a different performance envelope and different rebuild characteristics. Usually, when it is used though, it's software RAID0 on top of hardware RAID1.
Quoted 7 lines
Quoted 1 line
RAID51 and RAID61 used to be used, but aren't much now. They provided an easy way to have proper data verification without always having the rebuild overhead of RAID5/6 and without needing to do checksumming. They are pretty much useless for BTRFS, as it can already tell which copy is correct.
So until now you are repeating what I told: the only useful raid profile are
- striping
- mirroring
- striping+paring (even limiting the number of disk involved)
- striping+mirroring
No, not quite. At least, not in the combinations you're saying make sense if you are using standard terminology. RAID05 and RAID06 are not the same thing as 'striping+parity' as BTRFS implements that case, and can be significantly more optimized than the trivial implementation of just limiting the number of disks involved in each chunk (by, you know, actually striping just like what we currently call raid10 mode in BTRFS does).
Quoted 4 lines
Quoted 3 lines
RAID15 and RAID16 are a similar case to RAID51 and RAID61, except they might actually make sense in BTRFS to provide a backup means of rebuilding blocks that fail checksum validation if both copies fail.
If you need further redundancy, it is easy to implement a parity3 and parity4 raid profile instead of stacking a raid6+raid1
I think you're misunderstanding what I mean here.
RAID15/16 consist of two layers:
* The top layer is regular RAID1, usually limited to two copies.
* The lower layer is RAID5 or RAID6.
This means that the lower layer can validate which of the two copies in the upper layer is correct when they don't agree. It doesn't really provide significantly better redundancy (they can technically sustain more disk failures without failing completely than simple two-copy RAID1 can, but just like BTRFS raid10, they can't reliably survive more than one (or two if you're using RAID6 as the lower layer) disk failure), so it does not do the same thing that higher-order parity does.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
The fact that you can combine striping and mirroring (or pairing) makes sense because you could have a speed gain (see below). [....]
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
The striping concept is based to the fact that if the "stripe size" is small enough you have a speed benefit because the reads may be performed in parallel from different disks.
That's not the only benefit of striping though. The other big one is that you now have one volume that's the combined size of both of the original devices. Striping is arguably better for this even if you're using a large stripe size because it better balances the wear across the devices than simple concatenation.
Striping means that the data is interleaved between the disks with a reasonable "block unit". Otherwise which would be the difference between btrfs-raid0 and btrfs-single ?
Single mode guarantees that any file less than the chunk size in length will either be completely present or completely absent if one of the devices fails. BTRFS raid0 mode does not provide any such guarantee, and in fact guarantees that all files that are larger than the stripe unit size (however much gets put on one disk before moving to the next) will all lose data if a device fails.
Stupid as it sounds, this matters for some people.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
With a "stripe size" of 1GB, it is very unlikely that this would happens.
That's a pretty big assumption. There are all kinds of access patterns that will still distribute the load reasonably evenly across the constituent devices, even if they don't parallelize things.
If, for example, all your files are 64k or less, and you only read whole files, there's no functional difference between RAID0 with 1GB blocks and RAID0 with 64k blocks. Such a workload is not unusual on a very busy mail-server.
I fully agree that 64K may be too much for some workload, however I have to point out that I still find difficult to imagine that you can take advantage of parallel read from multiple disks with a 1GB stripe unit for a *common workload*. Pay attention that btrfs inline in the metadata the small files, so even if the file is smaller than 64k, a 64k read (or more) will be required in order to access it.
Again, mail servers. Each file should be written out as a single extent, which means it's all in one chunk. Delivery and processing need to access _LOTS_ of files on a busy mail server, and the good ones do this with userspace parallelization. BTRFS doesn't parallelize disk accesses from the same userspace execution context (thread if threads are being used, process if not), but it does parallelize access for separate contexts, so if userspace is doing things from multiple threads, so will BTRFS.
FWIW, I actually tested this back when the company I work for still ran their own internal mail server. BTRFS was significantly less optimized back then, but there was no measurable performance difference from userspace between using single profile for data or raid0 profile for data.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Quoted 1 line · Austin S. Hemmelgarn
On 2018-07-19 13:29, Goffredo Baroncelli wrote:
[...]
Quoted 8 lines
Quoted 6 lines
So until now you are repeating what I told: the only useful raid profile are
- striping
- mirroring
- striping+paring (even limiting the number of disk involved)
- striping+mirroringNo, not quite. At least, not in the combinations you're saying make sense if you are using standard terminology. RAID05 and RAID06 are not the same thing as 'striping+parity' as BTRFS implements that case, and can be significantly more optimized than the trivial implementation of just limiting the number of disks involved in each chunk (by, you know, actually striping just like what we currently call raid10 mode in BTRFS does).
Could you provide more information ?
Quoted 11 lines
Quoted 4 lines
Quoted 3 lines
RAID15 and RAID16 are a similar case to RAID51 and RAID61, except they might actually make sense in BTRFS to provide a backup means of rebuilding blocks that fail checksum validation if both copies fail.
If you need further redundancy, it is easy to implement a parity3 and parity4 raid profile instead of stacking a raid6+raid1
I think you're misunderstanding what I mean here.
RAID15/16 consist of two layers:
* The top layer is regular RAID1, usually limited to two copies.
* The lower layer is RAID5 or RAID6.This means that the lower layer can validate which of the two copies in the upper layer is correct when they don't agree.
This happens only because there is a redundancy greater than 1. Anyway BTRFS has the checksum, which helps a lot in this area
Quoted 18 lines
It doesn't really provide significantly better redundancy (they can technically sustain more disk failures without failing completely than simple two-copy RAID1 can, but just like BTRFS raid10, they can't reliably survive more than one (or two if you're using RAID6 as the lower layer) disk failure), so it does not do the same thing that higher-order parity does.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
The fact that you can combine striping and mirroring (or pairing) makes sense because you could have a speed gain (see below). [....]
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
The striping concept is based to the fact that if the "stripe size" is small enough you have a speed benefit because the reads may be performed in parallel from different disks.
That's not the only benefit of striping though. The other big one is that you now have one volume that's the combined size of both of the original devices. Striping is arguably better for this even if you're using a large stripe size because it better balances the wear across the devices than simple concatenation.
Striping means that the data is interleaved between the disks with a reasonable "block unit". Otherwise which would be the difference between btrfs-raid0 and btrfs-single ?
Single mode guarantees that any file less than the chunk size in length will either be completely present or completely absent if one of the devices fails. BTRFS raid0 mode does not provide any such guarantee, and in fact guarantees that all files that are larger than the stripe unit size (however much gets put on one disk before moving to the next) will all lose data if a device fails.
Stupid as it sounds, this matters for some people.
I think that even better would be having different filesystems.
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
With a "stripe size" of 1GB, it is very unlikely that this would happens.
That's a pretty big assumption. There are all kinds of access patterns that will still distribute the load reasonably evenly across the constituent devices, even if they don't parallelize things.
If, for example, all your files are 64k or less, and you only read whole files, there's no functional difference between RAID0 with 1GB blocks and RAID0 with 64k blocks. Such a workload is not unusual on a very busy mail-server.
I fully agree that 64K may be too much for some workload, however I have to point out that I still find difficult to imagine that you can take advantage of parallel read from multiple disks with a 1GB stripe unit for a *common workload*. Pay attention that btrfs inline in the metadata the small files, so even if the file is smaller than 64k, a 64k read (or more) will be required in order to access it.
Again, mail servers. Each file should be written out as a single extent, which means it's all in one chunk. Delivery and processing need to access _LOTS_ of files on a busy mail server, and the good ones do this with userspace parallelization. BTRFS doesn't parallelize disk accesses from the same userspace execution context (thread if threads are being used, process if not), but it does parallelize access for separate contexts, so if userspace is doing things from multiple threads, so will BTRFS.
The parallelization matters only if it is distributed across different disks. So more disks are involved more parallelization is possible. As extreme example, whit a stripe unit of 1GB, until the filesystem is smaller than 1GB, no parallelizzation is possible[*] because all data is in the same disk. And when the filesystem increases its size, the data must be "distant" more than 1GB to be parallelized.
[*] Of course it is possible to perform parallel read on the same disk, but the throughput would decrease; may be that the average latency would perform better.
Quoted 2 lines
FWIW, I actually tested this back when the company I work for still ran their own internal mail server. BTRFS was significantly less optimized back then, but there was no measurable performance difference from userspace between using single profile for data or raid0 profile for data.
Despite the btrfs optimization, having a stripe unit of 1GB reduces the likelihood of parallelizing the reads. This because the data to read to be parallelized must be "distant" more than the "stripe unit": having a stripe unit smaller increase the likelihood of a parallel reads.
Of course this is not sufficient. In any case BTRFS should improve its I/O scheduler
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
At 1 GiB strip size it doesn't have the typical performance advantage of striping, but conceptually, it's equivalent to raid10 with huge 1 GiB strips/chunks.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 10 lines · Goffredo Baroncelli
Quoted 1 line · Austin S. Hemmelgarn
On 2018-07-19 13:29, Goffredo Baroncelli wrote:
[...]
Quoted 7 lines
Quoted 6 lines
So until now you are repeating what I told: the only useful raid profile are
- striping
- mirroring
- striping+paring (even limiting the number of disk involved)
- striping+mirroring
Quoted 3 lines
Quoted 1 line
No, not quite. At least, not in the combinations you're saying make sense if you are using standard terminology. RAID05 and RAID06 are not the same thing as 'striping+parity' as BTRFS implements that case, and can be significantly more optimized than the trivial implementation of just limiting the number of disks involved in each chunk (by, you know, actually striping just like what we currently call raid10 mode in BTRFS does).
Could you provide more information ?
Just parity by itself is functionally equivalent to a really stupid implementation of 2 or more copies of the data. Setups with only one disk more than the number of parities in RAID5 and RAID6 are called degenerate for this very reason. All sane RAID5/6 implementations do striping across multiple devices internally, and that's almost always what people mean when talking about striping plus parity.
What I'm referring to is different though. Just like RAID10 used to be implemented as RAID1 on top of RAID0, RAID05 is RAID0 on top of RAID5. That is, you're striping your data across multiple RAID5 arrays instead of using one big RAID5 array to store it all. As I mentioned, this mitigates the scaling issues inherent in RAID5 when it comes to rebuilds (namely, the fact that device failure rates go up faster for larger arrays than rebuild times do).
Functionally, such a setup can be implemented in BTRFS by limiting RAID5/6 stripe width, but that will have all kinds of performance limitations compared to actually striping across all of the underlying RAID5 chunks. In fact, it will have the exact same performance limitations you're calling out BTRFS single mode for below.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
RAID15 and RAID16 are a similar case to RAID51 and RAID61, except they might actually make sense in BTRFS to provide a backup means of rebuilding blocks that fail checksum validation if both copies fail.
If you need further redundancy, it is easy to implement a parity3 and parity4 raid profile instead of stacking a raid6+raid1
I think you're misunderstanding what I mean here.
Quoted 7 lines
Quoted 5 lines
RAID15/16 consist of two layers:
* The top layer is regular RAID1, usually limited to two copies.
* The lower layer is RAID5 or RAID6.This means that the lower layer can validate which of the two copies in the upper layer is correct when they don't agree.
This happens only because there is a redundancy greater than 1. Anyway BTRFS has the checksum, which helps a lot in this area
The checksum helps, but what do you do when all copies fail the checksum? Or, worse yet, what do you do with both copies have the 'right' checksum, but different data? Yes, you could have one more copy, but that just reduces the chances of those cases happening, it doesn't eliminate them.
Note that I'm not necessarily saying it makes sense to have support for this in BTRFS, just that it's a real-world counter-example to your statement that only those combinations make sense. In the case of BTRFS, these would make more sense than RAID51 and RAID61, but they still aren't particularly practical. For classic RAID though, they're really important, because you don't have checksumming (unless you have T10 DIF capable hardware and a RAID implementation that understands how to work with it, but that's rare and expensive) and it makes it easier to resize an array than having three copies (you only need 2 new disks for RAID15 or RAID16 to increase the size of the array, but you need 3 for 3-copy RAID1 or RAID10).
Quoted 20 lines
Quoted 20 lines
It doesn't really provide significantly better redundancy (they can technically sustain more disk failures without failing completely than simple two-copy RAID1 can, but just like BTRFS raid10, they can't reliably survive more than one (or two if you're using RAID6 as the lower layer) disk failure), so it does not do the same thing that higher-order parity does.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
The fact that you can combine striping and mirroring (or pairing) makes sense because you could have a speed gain (see below). [....]
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
As someone else pointed out, md/lvm-raid10 already work like this. What btrfs calls raid10 is somewhat different, but btrfs raid1 pretty much works this way except with huge (gig size) chunks.
Quoted 7 lines
Quoted 1 line
As implemented in BTRFS, raid1 doesn't have striping.
The argument is that because there's only two copies, on multi-device btrfs raid1 with 4+ devices of equal size so chunk allocations tend to alternate device pairs, it's effectively striped at the macro level, with the 1 GiB device-level chunks effectively being huge individual device strips of 1 GiB.
The striping concept is based to the fact that if the "stripe size" is small enough you have a speed benefit because the reads may be performed in parallel from different disks.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
That's not the only benefit of striping though. The other big one is that you now have one volume that's the combined size of both of the original devices. Striping is arguably better for this even if you're using a large stripe size because it better balances the wear across the devices than simple concatenation.
Striping means that the data is interleaved between the disks with a reasonable "block unit". Otherwise which would be the difference between btrfs-raid0 and btrfs-single ?
Single mode guarantees that any file less than the chunk size in length will either be completely present or completely absent if one of the devices fails. BTRFS raid0 mode does not provide any such guarantee, and in fact guarantees that all files that are larger than the stripe unit size (however much gets put on one disk before moving to the next) will all lose data if a device fails.
Stupid as it sounds, this matters for some people.
I think that even better would be having different filesystems.
Not necessarily. In fact, quite the opposite in most cases, because having separate filesystems pushes the requirement to sort the files onto devices to userspace, which should not have to worry about that.
Put in cluster computing terms (where this kind of file layout is the norm), why exactly should the application software be the component responsible for figuring out what node a given file from a particular dataset is on? Why shouldn't the filesystem itself handle this?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
With a "stripe size" of 1GB, it is very unlikely that this would happens.
That's a pretty big assumption. There are all kinds of access patterns that will still distribute the load reasonably evenly across the constituent devices, even if they don't parallelize things.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
If, for example, all your files are 64k or less, and you only read whole files, there's no functional difference between RAID0 with 1GB blocks and RAID0 with 64k blocks. Such a workload is not unusual on a very busy mail-server.
I fully agree that 64K may be too much for some workload, however I have to point out that I still find difficult to imagine that you can take advantage of parallel read from multiple disks with a 1GB stripe unit for a *common workload*. Pay attention that btrfs inline in the metadata the small files, so even if the file is smaller than 64k, a 64k read (or more) will be required in order to access it.
Again, mail servers. Each file should be written out as a single extent, which means it's all in one chunk. Delivery and processing need to access _LOTS_ of files on a busy mail server, and the good ones do this with userspace parallelization. BTRFS doesn't parallelize disk accesses from the same userspace execution context (thread if threads are being used, process if not), but it does parallelize access for separate contexts, so if userspace is doing things from multiple threads, so will BTRFS.
The parallelization matters only if it is distributed across different disks. So more disks are involved more parallelization is possible. As extreme example, whit a stripe unit of 1GB, until the filesystem is smaller than 1GB, no parallelizzation is possible[*] because all data is in the same disk. And when the filesystem increases its size, the data must be "distant" more than 1GB to be parallelized.First, I think you have things slightly backwards, it should be 'until
the filesystem is _larger_ than 1GB' here.
That aside, the whole issue of data locality is not one to the degree you might think. for 64k files, that's 16384 files per chunk. That's a minuscule number for a really active mail-server (no, seriously, single subsidiary mail-servers in big companies may be handling queuing and delivery of more than twice that per-minute).
Quoted 2 lines
[*] Of course it is possible to perform parallel read on the same disk, but the throughput would decrease; may be that the average latency would perform better.
Raw throughput, measured simply as how many bytes you can read or write per second, would decrease. Actual effective throughput will not necessarily if you've got a storage device with very low seek times, because being able to load and process files in parallel may allow for much faster actual processing of the data compared to simple serial processing. Latency would be dependent on
Quoted 7 lines
Quoted 3 lines
FWIW, I actually tested this back when the company I work for still ran their own internal mail server. BTRFS was significantly less optimized back then, but there was no measurable performance difference from userspace between using single profile for data or raid0 profile for data.
Despite the btrfs optimization, having a stripe unit of 1GB reduces the likelihood of parallelizing the reads. This because the data to read to be parallelized must be "distant" more than the "stripe unit": having a stripe unit smaller increase the likelihood of a parallel reads.
Of course this is not sufficient. In any case BTRFS should improve its I/O scheduler
Agreed, we need actual parallel access to devices in BTRFS.
18.07.2018 22:42, Goffredo Baroncelli пишет:
Quoted 35 lines · Duncan
Quoted 34 lines
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 30 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Quoted 7 lines
Quoted 4 lines
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
RAID50 (striping across RAID5) is common.
Quoted 40 lines · Andrei Borzenkov
18.07.2018 22:42, Goffredo Baroncelli пишет:
Quoted 39 lines · Duncan
Quoted 38 lines
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 27 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)Quoted 3 lines
However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
Quoted 5 lines
Quoted 3 lines
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
RAID50 (striping across RAID5) is common.
Yeah someone else report that. But other than reducing the number of disk per raid5 (increasing the ration number of disks/number of parity disks), which other advantages has ? Limiting the number of disk per raid, in BTRFS would be quite simple to implement in the "chunk allocator"
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
20.07.2018 20:16, Goffredo Baroncelli пишет:
Quoted 41 lines · Andrei Borzenkov
Quoted 40 lines
18.07.2018 22:42, Goffredo Baroncelli пишет:
Quoted 39 lines · Duncan
Quoted 38 lines
Goffredo Baroncelli posted on Wed, 18 Jul 2018 07:59:52 +0200 as excerpted:
Quoted 27 lines · Duncan
Quoted 13 lines
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted:
Quoted 4 lines · waxhead
Striping and mirroring/pairing are orthogonal properties; mirror and parity are mutually exclusive.
I can't agree. I don't know whether you meant that in the global sense,
or purely in the btrfs context (which I suspect), but either way I can't agree.Quoted 5 lines
In the pure btrfs context, while striping and mirroring/pairing are orthogonal today, Hugo's whole point was that btrfs is theoretically flexible enough to allow both together and the feature may at some point be added, so it makes sense to have a layout notation format flexible enough to allow it as well.
When I say orthogonal, It means that these can be combined: i.e. you can have - striping (RAID0)
- parity (?)
- striping + parity (e.g. RAID5/6)
- mirroring (RAID1)
- mirroring + striping (RAID10)Quoted 3 lines
However you can't have mirroring+parity; this means that a notation where both 'C' ( = number of copy) and 'P' ( = number of parities) is too verbose.
Yes, you can have mirroring+parity, conceptually it's simply raid5/6 on top of mirroring or mirroring on top of raid5/6, much as raid10 is conceptually just raid0 on top of raid1, and raid01 is conceptually raid1 on top of raid0.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
And what about raid 615156156 (raid 6 on top of raid 1 on top of raid 5 on top of....) ???
Seriously, of course you can combine a lot of different profile; however the only ones that make sense are the ones above.
RAID50 (striping across RAID5) is common.
Yeah someone else report that. But other than reducing the number of disk per raid5 (increasing the ration number of disks/number of parity disks), which other advantages has ?
It allows distributing IO across virtually unlimited number of disks while confining failure domain to manageable size.
Quoted 2 lines
Limiting the number of disk per raid, in BTRFS would be quite simple to implement in the "chunk allocator"
You mean that currently RAID5 stripe size is equal to number of disks? Well, I suppose nobody is using btrfs with disk pools of two or three digits size.
Quoted 1 line · Andrei Borzenkov
20.07.2018 20:16, Goffredo Baroncelli пишет:
[snip]
Quoted 6 lines
Quoted 2 lines
Limiting the number of disk per raid, in BTRFS would be quite simple to implement in the "chunk allocator"
You mean that currently RAID5 stripe size is equal to number of disks? Well, I suppose nobody is using btrfs with disk pools of two or three digits size.
But they are (even if not very many of them) -- we've seen at least
one person with something like 40 or 50 devices in the array. They'd definitely got into /dev/sdac territory. I don't recall what RAID level they were using. I think it was either RAID-1 or -10.
That's the largest I can recall seeing mention of, though.
Hugo.
Signature · 4 lines
Hugo Mills | Have found Lost City of Atlantis. High Priest is hugo@... carfax.org.uk | winning at quoits. http://carfax.org.uk/ | PGP: E2AB1DE4 | Terry Pratchett
Quoted 10 lines · Hugo Mills
Quoted 1 line · Andrei Borzenkov
20.07.2018 20:16, Goffredo Baroncelli пишет:
[snip]
Quoted 6 lines
Quoted 2 lines
Limiting the number of disk per raid, in BTRFS would be quite simple to implement in the "chunk allocator"
You mean that currently RAID5 stripe size is equal to number of disks? Well, I suppose nobody is using btrfs with disk pools of two or three digits size.
Quoted 6 lines
But they are (even if not very many of them) -- we've seen at least
one person with something like 40 or 50 devices in the array. They'd definitely got into/dev/sdacterritory. I don't recall what RAID level they were using. I think it was either RAID-1 or -10.That's the largest I can recall seeing mention of, though.
I've talked to at least two people using it on 100+ disks in a SAN situation. In both cases however, BTRFS itself was only seeing about 20 devices and running in raid0 mode on them, with each of those being a RAID6 volume configured on the SAN node holding the disks for it. From what I understood when talking to them, they actually got rather good performance in this setup, though maintenance was a bit of a pain.
Quoted 23 lines · waxhead
Quoted 9 lines · David Sterba
An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
As just a humble BTRFS user I agree and really think it is about time to move far away from the RAID terminology. However adding some more descriptive profile names (or at least some aliases) would be much better for the commoners (such as myself). ...snip... > Which would make the above table look like so:
Old format / My Format / My suggested alias
SINGLE / R0.S0.P0 / SINGLE
DUP / R1.S1.P0 / DUP (or even MIRRORLOCAL1)
RAID0 / R0.Sm.P0 / STRIPE
RAID1 / R1.S0.P0 / MIRROR1
RAID1c3 / R2.S0.P0 / MIRROR2
RAID1c4 / R3.S0.P0 / MIRROR3
RAID10 / R1.Sm.P0 / STRIPE.MIRROR1
RAID5 / R1.Sm.P1 / STRIPE.PARITY1
RAID6 / R1.Sm.P2 / STRIPE.PARITY2
Quoted 1 line
And i think this is much more readable, but others may disagree. And as a side note... from a (hobby) coders perspective this is probably simpler to parse as well. ...snap...
...and before someone else points this out that my suggestion has an ugly flaw , I got a bit copy / paste happy and messed up the RAID 5 and 6 like profiles. The below table are corrected and hopefully it make the point why using the word 'replicas' is easier to understand than 'copies' even if I messed it up :)
Old format / My Format / My suggested alias
SINGLE / R0.S0.P0 / SINGLE DUP / R1.S1.P0 / DUP (or even MIRRORLOCAL1) RAID0 / R0.Sm.P0 / STRIPE RAID1 / R1.S0.P0 / MIRROR1 RAID1c3 / R2.S0.P0 / MIRROR2 RAID1c4 / R3.S0.P0 / MIRROR3 RAID10 / R1.Sm.P0 / STRIPE.MIRROR1 RAID5 / R0.Sm.P1 / STRIPE.PARITY1 RAID6 / R0.Sm.P2 / STRIPE.PARITY2
On Fri, Jul 13, 2018 at 08:46:28PM +0200, David Sterba wrote:
[snip]
Quoted 8 lines
An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
I'd suggest using lower-case letter for the c, s, p, rather than
upper, as it makes it much easier to read. The upper-case version tends to make the letters and numbers merge into each other. With lower-case c, s, p, the taller digits (or M) stand out:
1c
1cMs2p
2c3s8p (OK, just kidding about this one)
Hugo.
Signature · 4 lines
Hugo Mills | The English language has the mot juste for every hugo@... carfax.org.uk | occasion. http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 13 lines · David Sterba
Hi,
I have some goodies that go into the RAID56 problem, although not implementing all the remaining features, it can be useful independently.
This time my hackweek project
https://hackweek.suse.com/17/projects/do-something-about-btrfs-and-raid56
aimed to implement the fix for the write hole problem but I spent more time with analysis and design of the solution and don't have a working prototype for that yet.
Quoted 5 lines
This patchset brings a feature that will be used by the raid56 log, the log has to be on the same redundancy level and thus we need a 3-copy replication for raid6. As it was easy to extend to higher replication, I've added a 4-copy replication, that would allow triple copy raid (that does not have a standardized name).
So this special level will be used for RAID56 for now?
Or it will also be possible for metadata usage just like current RAID1?
If the latter, the metadata scrub problem will need to be considered more.
For more copies RAID1, it's will have higher possibility one or two devices missing, and then being scrubbed.
For metadata scrub, inlined csum can't ensure it's the latest one.
So for such RAID1 scrub, we need to read out all copies and compare their generation to find out the correct copy.
At least from the changeset, it doesn't look like it's addressed yet.
And this also reminds me that current scrub is not as flex as balance, I really like we could filter block groups to scrub just like balance, and do scrub in a block group basis, other than devid basis. That's to say, for a block group scrub, we don't really care which device we're scrubbing, we just need to ensure all device in this block is storing correct data.
Thanks,
Qu
Quoted 48 lines
The number of copies is fixed, so it's not N-copy for an arbitrary N. This would complicate the implementation too much, though I'd be willing to add a 5-copy replication for a small bribe.
The new raid profiles and covered by an incompatibility bit, called
extended_raid, the (idealistic) plan is to stuff as many new raid-related features as possible. The patch 4/4 mentions the 3- 4- copy raid1, configurable stripe length, write hole log and triple parity. If the plan turns out to be too ambitious, the ready and implemented features will be split and merged.An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:
https://marc.info/?l=linux-btrfs&m=136286324417767
Switching to this naming would be a good addition to the extended raid.
Regarding the missing raid56 features, I'll continue working on them as time permits in the following weeks/months, as I'm not aware of anybody working on that actively enough so to speak.
Anyway, git branches with the patches:
kernel: git://github.com/kdave/btrfs-devel dev/extended-raid-ncopies progs: git://github.com/kdave/btrfs-progs dev/extended-raid-ncopiesDavid Sterba (4):
btrfs: refactor block group replication factor calculation to a helper
btrfs: add support for 3-copy replication (raid1c3)
btrfs: add support for 4-copy replication (raid1c4)
btrfs: add incompatibility bit for extended raid featuresfs/btrfs/ctree.h | 1 + fs/btrfs/extent-tree.c | 45 +++++++----------- fs/btrfs/relocation.c | 1 + fs/btrfs/scrub.c | 4 +- fs/btrfs/super.c | 17 +++---- fs/btrfs/sysfs.c | 2 + fs/btrfs/volumes.c | 84 ++++++++++++++++++++++++++++++--- fs/btrfs/volumes.h | 6 +++ include/uapi/linux/btrfs.h | 12 ++++- include/uapi/linux/btrfs_tree.h | 6 +++ 10 files changed, 134 insertions(+), 44 deletions(-)
Quoted 35 lines · Qu Wenruo
Quoted 13 lines · David Sterba
Hi,
I have some goodies that go into the RAID56 problem, although not implementing all the remaining features, it can be useful independently.
This time my hackweek project
https://hackweek.suse.com/17/projects/do-something-about-btrfs-and-raid56
aimed to implement the fix for the write hole problem but I spent more time with analysis and design of the solution and don't have a working prototype for that yet.
Quoted 5 lines
This patchset brings a feature that will be used by the raid56 log, the log has to be on the same redundancy level and thus we need a 3-copy replication for raid6. As it was easy to extend to higher replication, I've added a 4-copy replication, that would allow triple copy raid (that does not have a standardized name).
So this special level will be used for RAID56 for now?
Or it will also be possible for metadata usage just like current RAID1?If the latter, the metadata scrub problem will need to be considered more.
For more copies RAID1, it's will have higher possibility one or two devices missing, and then being scrubbed.
For metadata scrub, inlined csum can't ensure it's the latest one.So for such RAID1 scrub, we need to read out all copies and compare their generation to find out the correct copy.
At least from the changeset, it doesn't look like it's addressed yet.
Quoted 7 lines
And this also reminds me that current scrub is not as flex as balance, I really like we could filter block groups to scrub just like balance, and do scrub in a block group basis, other than devid basis. That's to say, for a block group scrub, we don't really care which device we're scrubbing, we just need to ensure all device in this block is storing correct data.
This would actually be rather useful for non-parity cases too. Being able to scrub only metadata when the data chunks are using a profile that provides no rebuild support would be great for performance.
On the same note, it would be _really_ nice to be able to scrub a subset of the volume's directory tree, even if it were only per-subvolume.
Quoted 20 lines · Austin S. Hemmelgarn
Quoted 13 lines
So this special level will be used for RAID56 for now?
Or it will also be possible for metadata usage just like current RAID1?If the latter, the metadata scrub problem will need to be considered more.
For more copies RAID1, it's will have higher possibility one or two devices missing, and then being scrubbed.
For metadata scrub, inlined csum can't ensure it's the latest one.So for such RAID1 scrub, we need to read out all copies and compare their generation to find out the correct copy.
At least from the changeset, it doesn't look like it's addressed yet.Quoted 7 lines
And this also reminds me that current scrub is not as flex as balance, I really like we could filter block groups to scrub just like balance, and do scrub in a block group basis, other than devid basis. That's to say, for a block group scrub, we don't really care which device we're scrubbing, we just need to ensure all device in this block is storing correct data.
Quoted 6 lines
This would actually be rather useful for non-parity cases too. Being able to scrub only metadata when the data chunks are using a profile that provides no rebuild support would be great for performance.
On the same note, it would be _really_ nice to be able to scrub a subset of the volume's directory tree, even if it were only per-subvolume.
https://github.com/kdave/drafts/blob/master/btrfs/scrub-subvolume.txt
https://github.com/kdave/drafts/blob/master/btrfs/scrub-custom.txt
The idea is to build in-memory tree of block ranges that span the given subvolume or files and run scrub only there. The selective scrub on the block groups of a given type would be a special case of the above.
Quoted 14 lines · Qu Wenruo
Quoted 13 lines · David Sterba
Hi,
I have some goodies that go into the RAID56 problem, although not implementing all the remaining features, it can be useful independently.
This time my hackweek project
https://hackweek.suse.com/17/projects/do-something-about-btrfs-and-raid56
aimed to implement the fix for the write hole problem but I spent more time with analysis and design of the solution and don't have a working prototype for that yet.
Quoted 8 lines
Quoted 5 lines
This patchset brings a feature that will be used by the raid56 log, the log has to be on the same redundancy level and thus we need a 3-copy replication for raid6. As it was easy to extend to higher replication, I've added a 4-copy replication, that would allow triple copy raid (that does not have a standardized name).
So this special level will be used for RAID56 for now?
Or it will also be possible for metadata usage just like current RAID1?
It's a new profile usable in the same way as is raid1, ie. for the data or metadata. The patch that adds support to btrfs-progs has an mkfs example.
The raid56 will use that to store the log, essentially data forcibly stored on the n-copy raid1 chunk and used only for logging.
Quoted 9 lines
If the latter, the metadata scrub problem will need to be considered more.
For more copies RAID1, it's will have higher possibility one or two devices missing, and then being scrubbed.
For metadata scrub, inlined csum can't ensure it's the latest one.So for such RAID1 scrub, we need to read out all copies and compare their generation to find out the correct copy.
At least from the changeset, it doesn't look like it's addressed yet.
Nothing like this is implemented in the patches, but I don't understand how this differs from the current raid1 and one missing device. Sure we can't have 2 missing devices so the existing copy is automatically considered correct and up to date.
There are more corner case recovery scenario when there could be 3 copies slightly out of date due to device loss and scrub attempt, so yes this would need to be addressed.
Quoted 6 lines
And this also reminds me that current scrub is not as flex as balance, I really like we could filter block groups to scrub just like balance, and do scrub in a block group basis, other than devid basis. That's to say, for a block group scrub, we don't really care which device we're scrubbing, we just need to ensure all device in this block is storing correct data.
Right, a subset of the balance filters would be nice.