btrfs: basic tier support
cover btrfs: basic tier support
Storage tier is a big feature, this is just a basic support.
1) Add tier score to device
We use a single score value to define the tier level of a device. Different score means different tier, and bigger score is faster.
DAX device(dax=1)
SSD device(rotational=0)
HDD device(rotational=1)
TODO/FIXME: FIXME: detect bus(DIMM, NVMe, SCSI, SATA, Virtio, ...) TODO/FIXME: user-assigned property(refactoring the coming 'read_preferred' property?) to
set to the max score for some not-well-supported case.
In most case, only 1 or 2 tiers are used at the same time, so we group them into top tier and other tier(s).
2) tiering data and metadata
This based the patch 'btrfs: add ssd_metadata mode' from Goffredo Baroncelli <[email protected]>
We define a mount option to tiering data/metadata to slower/faster device(s) When there is only 1 tier, tiering is auto disabled.
mount option: tier[={off|auto|data_tier_X/metadata_tier_Y}] default is 'tier[=auto]'. 'tier' is same as 'tier=auto', 'tier=OF/TF' the policies to use the device(s):
Top-tier-Only(TO) : metadata only use top-tier device.
Top-tier-Firstly(TF) : metadata use top-tier device firstly.
Other-tier-First(OF) : data use other-tier device firstly.
Other-tier-Only(OO) : data only use other-tier device.data_tier_X is the policy for data, support OF, OO. metadata_tier_Y is the policy for metadata and system, support TF.
3) tier-aware mirror path select
This feature help the read performance, so it is enabled even if tier=off.
4) tier-aware free space cacl
Detect some case of free space 0 because of tier policy of data. Full support is yet TODO/FIXME.
5) TODO/FIXME: per-subvol tiering policy and then per-subvol profile(RAID)
per-subvol tiering policy and then per-subvol data profile(RAID) is needed for the full tier support.
data policy support TO, TF too, in addition to OF, OO.
But now as a workaround, we can keep them as 2 separated btrfs file system with disk partition and 'btrfs filesystem resize'.
wangyugui (4):
btrfs: add tier score to device
btrfs: tiering data and metadata
btrfs: tier-aware mirror path select
btrfs: tier-aware free space cacl
fs/btrfs/ctree.h | 17 +++++++ fs/btrfs/super.c | 90 ++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.c | 119 +++++++++++++++++++++++++++++++++++++++++++-- fs/btrfs/volumes.h | 5 ++ 4 files changed, 228 insertions(+), 3 deletions(-)
Signature · 1 line
2.29.1
PATCH 1/4 btrfs: add tier score to device
We use a single score value to define the tier level of a device. Different score means different tier, and bigger is faster.
DAX device(dax=1)
SSD device(rotational=0)
HDD device(rotational=1)
TODO/FIXME: FIXME: detect bus(DIMM, NVMe, SCSI, SATA, Virtio, ...) TODO/FIXME: user-assigned property(refactoring the coming 'read_preferred' property?) to
set to the max score for some not-well-supported case.
In most case, only 1 or 2 tiers are used at the same time, so we group them into top tier and other tier(s).
--- fs/btrfs/volumes.c | 18 ++++++++++++++++++ fs/btrfs/volumes.h | 5 +++++ 2 files changed, 23 insertions(+)
fs/btrfs/volumes.c +18 -0
@@ -608,6 +608,22 @@ static int btrfs_free_stale_devices(const char *path,return ret; } +/* +* Get the tier score to the device, bigger score is faster. +* FIXME: detect bus(DIMM, NVMe, SCSI, SATA, Virtio, ...) +* FIXME: detect media inside(SLC/MLC of SSD, SMR/PMR of HDD, ...) +* FIXME: user-assigned property to set to max score for some complex case. +*/ +static void dev_get_tier_score (struct btrfs_device *device,struct request_queue *q) +{ +if (blk_queue_dax (q)) + device->tier_score =50 ; +else if (blk_queue_nonrot (q)) + device->tier_score =10 ; +else + device->tier_score =0 ; +} +/* * This is only used on mount, and we are protected from competing things * messing with our fs_devices by the uuid_mutex, thus we do not need the
@@ -660,6 +676,7 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices, } q =bdev_get_queue (bdev); +dev_get_tier_score (device,q);if (!blk_queue_nonrot (q)) fs_devices->rotating = true;
@@ -2590,6 +2607,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_pathatomic64_add (device->total_bytes, &fs_info->free_chunk_space); +dev_get_tier_score (device,q);if (!blk_queue_nonrot (q)) fs_devices->rotating = true;
fs/btrfs/volumes.h +5 -0
@@ -138,6 +138,11 @@ struct btrfs_device {struct completion kobj_unregister;/* For sysfs/FSID/devinfo/devid/ */ struct kobject devid_kobj; + +/* Storage tier score, bigger score is faster. +* In most case, only 1 or 2 tiers are used at the same time, so we group them +* into top tier and other tier(s). */ +u8 tier_score; };/*
Signature · 1 line
2.29.1
PATCH 2/4 btrfs: tiering data and metadata
In reply to wangyuguiThis based the patch 'btrfs: add ssd_metadata mode' from Goffredo Baroncelli <[email protected]>
In most case, only 1 or 2 tiers are used at the same time, so we group them into top tier and other tier(s).
We define a mount option to tiering data/metadata to slower/faster device(s) When there is only 1 tier, tiering is auto disabled.
mount option: tier[={off|auto|data_tier_X/metadata_tier_Y}] default is 'tier[=auto]'. 'tier' is same as 'tier=auto', 'tier=OF/TF' the policies to use the device(s):
Top-tier-Only(TO) : metadata only use top-tier device.
Top-tier-Firstly(TF) : metadata use top-tier device firstly.
Other-tier-First(OF) : data use other-tier device firstly.
Other-tier-Only(OO) : data only use other-tier device.data_tier_X is the policy for data, support OF, OO. metadata_tier_Y is the policy for metadata and system, support TF.
--- fs/btrfs/ctree.h | 17 ++++++++++ fs/btrfs/super.c | 72 +++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.c | 80 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 167 insertions(+), 2 deletions(-)
fs/btrfs/ctree.h +17 -0
@@ -574,6 +574,20 @@ enum btrfs_exclusive_operation { BTRFS_EXCLOP_SWAP_ACTIVATE, }; +/* +* tier policy for btrfs data/metadata +* FIXME: per-subvol tier policy for full tier support. +* FIXME: per-subvol profile(RAID) is needed for full tier support too. +*/ +enum btrfs_tier_policy +{ +NOT_TIERING , +TOP_TIER_ONLY ,/* TO */ +TOP_TIER_FIRSTLY ,/* TF */ +OTHER_TIER_FIRSTLY ,/* OF */ +OTHER_TIER_ONLY ,/* OO */ +}; +struct btrfs_fs_info {u8 chunk_tree_uuid[BTRFS_UUID_SIZE ];unsigned long flags;
@@ -831,6 +845,9 @@ struct btrfs_fs_info {u64 avail_metadata_alloc_bits;u64 avail_system_alloc_bits; +enum btrfs_tier_policy data_tier_policy; +enum btrfs_tier_policy metadata_tier_policy; +/* restriper state */ spinlock_t balance_lock;struct mutex balance_mutex;
fs/btrfs/super.c +72 -0
114 lines not on this page · raw message
fs/btrfs/volumes.c +78 -2
105 lines not on this page · raw message
Signature · 1 line
2.29.1
PATCH 3/4 btrfs: tier-aware mirror path select
In reply to wangyuguiThis feature help the read performance, so it is enabled even if tier=off.
--- fs/btrfs/volumes.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
fs/btrfs/volumes.c +20 -1
@@ -5568,6 +5568,11 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,int tolerance;struct btrfs_device *srcdev; +/* tier-aware even if tier=off */ +int top_tier_num_stripes =0 ; +int top_tier_stripe_idxs[4 ];/* RAID1C4 */ +u8 top_tier_score =0 ; +ASSERT ((map->type & (BTRFS_BLOCK_GROUP_RAID1_MASK |BTRFS_BLOCK_GROUP_RAID10 )));
@@ -5576,7 +5581,21 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info, else num_stripes = map->num_stripes; - preferred_mirror = first + current->pid % num_stripes; +for (i =0 ; i < num_stripes; ++i) + { +if (map->stripes[i].dev->tier_score > top_tier_score) + { + top_tier_score = map->stripes[i].dev->tier_score; + top_tier_stripe_idxs[0 ] = i; + top_tier_num_stripes =1 ; + } +else if (map->stripes[i].dev->tier_score == top_tier_score) + { + top_tier_stripe_idxs[top_tier_num_stripes] = i; + top_tier_num_stripes++; + } + } + preferred_mirror = first + top_tier_stripe_idxs[current->pid % top_tier_num_stripes];if (dev_replace_is_ongoing && fs_info->dev_replace.cont_reading_from_srcdev_mode ==
Signature · 1 line
2.29.1
PATCH 4/4 btrfs: tier-aware free space cacl
In reply to wangyuguiDetect some case of free space 0 because of tier policy of data. Full support is yet TODO/FIXME.
--- fs/btrfs/super.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
fs/btrfs/super.c +18 -0
@@ -2105,6 +2105,9 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info,int num_stripes =1 ;int i =0 , nr_devices;const struct btrfs_raid_attr *rattr; +/* tier-aware */ +int nr_top_tier =0 ; +u8 top_tier_score =0 ;/* * We aren't under the device list lock, so this is racy-ish, but good
@@ -2176,12 +2179,27 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info, devices_info[i].dev = device; devices_info[i].max_avail = avail_space; +if (devices_info[i].dev->tier_score > top_tier_score) { + top_tier_score = devices_info[i].dev->tier_score; + nr_top_tier =1 ; + } +else if (devices_info[i].dev->tier_score == top_tier_score) { + nr_top_tier++; + } + i++; }rcu_read_unlock (); nr_devices = i; +if (fs_info->data_tier_policy ==OTHER_TIER_ONLY && + nr_top_tier != nr_devices && nr_devices - nr_top_tier < rattr->devs_min) { +/* FIXME: full support of tier-aware. */ + *free_bytes =0 ; +return 0 ; + } +btrfs_descending_sort_devices (devices_info, nr_devices); i = nr_devices -1 ;
Signature · 1 line
2.29.1