Skip to contentSkip to conversation

btrfs: basic tier support

PATCH New wangyugui 1 participant 5 messages Oct 29, 2020 Btrfs file system

lore
Share & tools
lore

wangyugui

author
#0 lore raw

cover btrfs: basic tier support

Storage tier is a big feature, this is just a basic support.

1) Add tier score to device
We use a single score value to define the tier level of a device. Different score means different tier, and bigger score is faster.
DAX device(dax=1)
SSD device(rotational=0)
HDD device(rotational=1)
TODO/FIXME: FIXME: detect bus(DIMM, NVMe, SCSI, SATA, Virtio, ...) TODO/FIXME: user-assigned property(refactoring the coming 'read_preferred' property?) to
set to the max score for some not-well-supported case.

In most case, only 1 or 2 tiers are used at the same time, so we group them into top tier and other tier(s).

2) tiering data and metadata
This based the patch 'btrfs: add ssd_metadata mode' from Goffredo Baroncelli <[email protected]>

We define a mount option to tiering data/metadata to slower/faster device(s) When there is only 1 tier, tiering is auto disabled.

mount option: tier[={off|auto|data_tier_X/metadata_tier_Y}] default is 'tier[=auto]'. 'tier' is same as 'tier=auto', 'tier=OF/TF' the policies to use the device(s):

    Top-tier-Only(TO)       : metadata only use top-tier device.
    Top-tier-Firstly(TF)    : metadata use top-tier device firstly.
    Other-tier-First(OF)    : data use other-tier device firstly.
    Other-tier-Only(OO)     : data only use other-tier device.

data_tier_X is the policy for data, support OF, OO. metadata_tier_Y is the policy for metadata and system, support TF.

3) tier-aware mirror path select
This feature help the read performance, so it is enabled even if tier=off.

4) tier-aware free space cacl
Detect some case of free space 0 because of tier policy of data. Full support is yet TODO/FIXME.

5) TODO/FIXME: per-subvol tiering policy and then per-subvol profile(RAID)
per-subvol tiering policy and then per-subvol data profile(RAID) is needed for the full tier support.
data policy support TO, TF too, in addition to OF, OO.
But now as a workaround, we can keep them as 2 separated btrfs file system with disk partition and 'btrfs filesystem resize'.

wangyugui (4):
btrfs: add tier score to device
btrfs: tiering data and metadata
btrfs: tier-aware mirror path select
btrfs: tier-aware free space cacl

 fs/btrfs/ctree.h   |  17 +++++++
 fs/btrfs/super.c   |  90 ++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.c | 119 +++++++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/volumes.h |   5 ++
 4 files changed, 228 insertions(+), 3 deletions(-)
Signature · 1 line
2.29.1
author
#1 lore raw

PATCH 1/4 btrfs: add tier score to device

We use a single score value to define the tier level of a device. Different score means different tier, and bigger is faster.
DAX device(dax=1)
SSD device(rotational=0)
HDD device(rotational=1)
TODO/FIXME: FIXME: detect bus(DIMM, NVMe, SCSI, SATA, Virtio, ...) TODO/FIXME: user-assigned property(refactoring the coming 'read_preferred' property?) to
set to the max score for some not-well-supported case.

In most case, only 1 or 2 tiers are used at the same time, so we group them into top tier and other tier(s).

---
 fs/btrfs/volumes.c | 18 ++++++++++++++++++
 fs/btrfs/volumes.h |  5 +++++
 2 files changed, 23 insertions(+)
fs/btrfs/volumes.c +18 -0
index 58b9c41..efffcbc 100644
@@ -608,6 +608,22 @@ static int btrfs_free_stale_devices(const char *path,
 	return ret;
 }
 
+/*
+ * Get the tier score to the device, bigger score is faster.
+ * FIXME: detect bus(DIMM, NVMe, SCSI, SATA, Virtio, ...)
+ * FIXME: detect media inside(SLC/MLC of SSD, SMR/PMR of HDD, ...)
+ * FIXME: user-assigned property to set to max score for some complex case.
+ */
+static void dev_get_tier_score(struct btrfs_device *device, struct request_queue *q)
+{
+	if (blk_queue_dax(q))
+		device->tier_score = 50;
+	else if (blk_queue_nonrot(q))
+		device->tier_score = 10;
+	else
+		device->tier_score = 0;
+}
+
 /*
  * This is only used on mount, and we are protected from competing things
  * messing with our fs_devices by the uuid_mutex, thus we do not need the
@@ -660,6 +676,7 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
 	}
 
 	q = bdev_get_queue(bdev);
+	dev_get_tier_score(device,q);
 	if (!blk_queue_nonrot(q))
 		fs_devices->rotating = true;
 
@@ -2590,6 +2607,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 
 	atomic64_add(device->total_bytes, &fs_info->free_chunk_space);
 
+	dev_get_tier_score(device,q);
 	if (!blk_queue_nonrot(q))
 		fs_devices->rotating = true;
 
fs/btrfs/volumes.h +5 -0
index bf27ac0..cf426ec 100644
@@ -138,6 +138,11 @@ struct btrfs_device {
 	struct completion kobj_unregister;
 	/* For sysfs/FSID/devinfo/devid/ */
 	struct kobject devid_kobj;
+
+	/* Storage tier score, bigger score is faster.
+	 * In most case, only 1 or 2 tiers are used at the same time, so we group them
+	 * into top tier and other tier(s). */
+	u8 tier_score;
 };
 
 /*
Signature · 1 line
2.29.1
author
#2 lore raw

PATCH 2/4 btrfs: tiering data and metadata

In reply to wangyugui

This based the patch 'btrfs: add ssd_metadata mode' from Goffredo Baroncelli <[email protected]>

In most case, only 1 or 2 tiers are used at the same time, so we group them into top tier and other tier(s).

We define a mount option to tiering data/metadata to slower/faster device(s) When there is only 1 tier, tiering is auto disabled.

mount option: tier[={off|auto|data_tier_X/metadata_tier_Y}] default is 'tier[=auto]'. 'tier' is same as 'tier=auto', 'tier=OF/TF' the policies to use the device(s):

    Top-tier-Only(TO)       : metadata only use top-tier device.
    Top-tier-Firstly(TF)    : metadata use top-tier device firstly.
    Other-tier-First(OF)    : data use other-tier device firstly.
    Other-tier-Only(OO)     : data only use other-tier device.

data_tier_X is the policy for data, support OF, OO. metadata_tier_Y is the policy for metadata and system, support TF.

---
 fs/btrfs/ctree.h   | 17 ++++++++++
 fs/btrfs/super.c   | 72 +++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.c | 80 ++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 167 insertions(+), 2 deletions(-)
fs/btrfs/ctree.h +17 -0
index aac3d6f..812d231 100644
@@ -574,6 +574,20 @@ enum btrfs_exclusive_operation {
 	BTRFS_EXCLOP_SWAP_ACTIVATE,
 };
 
+/*
+ * tier policy for btrfs data/metadata
+ * FIXME: per-subvol tier policy for full tier support.
+ * FIXME: per-subvol profile(RAID) is needed for full tier support too.
+ */
+enum btrfs_tier_policy
+{
+	NOT_TIERING,
+	TOP_TIER_ONLY,		/* TO */
+	TOP_TIER_FIRSTLY,	/* TF */
+	OTHER_TIER_FIRSTLY, /* OF */
+	OTHER_TIER_ONLY,	/* OO */
+};
+
 struct btrfs_fs_info {
 	u8 chunk_tree_uuid[BTRFS_UUID_SIZE];
 	unsigned long flags;
@@ -831,6 +845,9 @@ struct btrfs_fs_info {
 	u64 avail_metadata_alloc_bits;
 	u64 avail_system_alloc_bits;
 
+	enum btrfs_tier_policy data_tier_policy;
+	enum btrfs_tier_policy metadata_tier_policy;
+
 	/* restriper state */
 	spinlock_t balance_lock;
 	struct mutex balance_mutex;
fs/btrfs/super.c +72 -0

114 lines not on this page · raw message

fs/btrfs/volumes.c +78 -2

105 lines not on this page · raw message

Signature · 1 line
2.29.1
author
#3 lore raw

PATCH 3/4 btrfs: tier-aware mirror path select

In reply to wangyugui

This feature help the read performance, so it is enabled even if tier=off.

---
 fs/btrfs/volumes.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
fs/btrfs/volumes.c +20 -1
index 2a422ac..b65f916 100644
@@ -5568,6 +5568,11 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 	int tolerance;
 	struct btrfs_device *srcdev;
 
+	/* tier-aware even if tier=off */
+	int top_tier_num_stripes = 0;
+	int top_tier_stripe_idxs[4]; /* RAID1C4 */
+	u8 top_tier_score = 0;
+
 	ASSERT((map->type &
 		 (BTRFS_BLOCK_GROUP_RAID1_MASK | BTRFS_BLOCK_GROUP_RAID10)));
 
@@ -5576,7 +5581,21 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 	else
 		num_stripes = map->num_stripes;
 
-	preferred_mirror = first + current->pid % num_stripes;
+	for (i = 0; i < num_stripes; ++i)
+	{
+		if (map->stripes[i].dev->tier_score > top_tier_score)
+		{
+			top_tier_score = map->stripes[i].dev->tier_score;
+			top_tier_stripe_idxs[0] = i;
+			top_tier_num_stripes = 1;
+		}
+		else if (map->stripes[i].dev->tier_score == top_tier_score)
+		{
+			top_tier_stripe_idxs[top_tier_num_stripes] = i;
+			top_tier_num_stripes++;
+		}
+	}
+	preferred_mirror = first + top_tier_stripe_idxs[current->pid % top_tier_num_stripes];
 
 	if (dev_replace_is_ongoing &&
 	    fs_info->dev_replace.cont_reading_from_srcdev_mode ==
Signature · 1 line
2.29.1
author
#4 lore raw

PATCH 4/4 btrfs: tier-aware free space cacl

In reply to wangyugui

Detect some case of free space 0 because of tier policy of data. Full support is yet TODO/FIXME.

---
 fs/btrfs/super.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
fs/btrfs/super.c +18 -0
index c8dfa89..feb1ae3 100644
@@ -2105,6 +2105,9 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info,
 	int num_stripes = 1;
 	int i = 0, nr_devices;
 	const struct btrfs_raid_attr *rattr;
+	/* tier-aware */
+	int nr_top_tier = 0;
+	u8 top_tier_score = 0;
 
 	/*
 	 * We aren't under the device list lock, so this is racy-ish, but good
@@ -2176,12 +2179,27 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info,
 		devices_info[i].dev = device;
 		devices_info[i].max_avail = avail_space;
 
+		if (devices_info[i].dev->tier_score > top_tier_score) {
+			top_tier_score = devices_info[i].dev->tier_score;
+			nr_top_tier = 1;
+		}
+		else if (devices_info[i].dev->tier_score == top_tier_score) {
+			nr_top_tier++;
+		}
+
 		i++;
 	}
 	rcu_read_unlock();
 
 	nr_devices = i;
 
+	if (fs_info->data_tier_policy == OTHER_TIER_ONLY &&
+		nr_top_tier != nr_devices && nr_devices - nr_top_tier < rattr->devs_min) {
+		/* FIXME: full support of tier-aware. */
+		*free_bytes = 0;
+		return 0;
+	}
+
 	btrfs_descending_sort_devices(devices_info, nr_devices);
 
 	i = nr_devices - 1;
Signature · 1 line
2.29.1