Skip to contentSkip to conversation

BTRFS hot relocation support

RFC In review [email protected] 8 participants 27 messages May 6, 2013 – May 19, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover BTRFS hot relocation support

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc -f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1 transid 16 /dev/vdb
[ 140.283650] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.550759] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2 transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB
Btrfs v0.20-rc1-254-gb0136aa-dirty
root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 6 /dev/vdb
[ 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature is currently 109
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~# 

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

 fs/btrfs/Makefile            |   3 +-
 fs/btrfs/ctree.h             |  26 +-
 fs/btrfs/extent-tree.c       | 107 +++++-
 fs/btrfs/extent_io.c         |  31 +-
 fs/btrfs/extent_io.h         |   4 +
 fs/btrfs/file.c              |  36 +-
 fs/btrfs/hot_relocate.c      | 802 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/hot_relocate.h      |  48 +++
 fs/btrfs/inode-map.c         |  13 +-
 fs/btrfs/inode.c             |  92 ++++-
 fs/btrfs/ioctl.c             |  23 +-
 fs/btrfs/relocation.c        |  14 +-
 fs/btrfs/super.c             |  30 +-
 fs/btrfs/volumes.c           |  28 +-
 fs/hot_tracking.c            |   1 +
 include/linux/btrfs.h        |   4 +
 include/linux/hot_tracking.h |   1 +
 kernel/sysctl.c              |  22 ++
 18 files changed, 1234 insertions(+), 51 deletions(-)
 create mode 100644 fs/btrfs/hot_relocate.c
 create mode 100644 fs/btrfs/hot_relocate.h
Signature · 1 line
1.7.11.7
#1 lore raw

RFC 1/5 vfs: add one list_head field

From: Zhi Yong Wu <[email protected]>

Add one list_head field 'reloc_list' to accommodate
hot relocation support.

---
 fs/hot_tracking.c            | 1 +
 include/linux/hot_tracking.h | 1 +
 2 files changed, 2 insertions(+)
fs/hot_tracking.c +1 -0
index 3b0002c..7071ac8 100644
@@ -41,6 +41,7 @@ static void hot_comm_item_init(struct hot_comm_item *ci, int type)
 	clear_bit(HOT_IN_LIST, &ci->delete_flag);
 	clear_bit(HOT_DELETING, &ci->delete_flag);
 	INIT_LIST_HEAD(&ci->track_list);
+	INIT_LIST_HEAD(&ci->reloc_list);
 	memset(&ci->hot_freq_data, 0, sizeof(struct hot_freq_data));
 	ci->hot_freq_data.avg_delta_reads = (u64) -1;
 	ci->hot_freq_data.avg_delta_writes = (u64) -1;
include/linux/hot_tracking.h +1 -0
index 2272975..49f901c 100644
@@ -68,6 +68,7 @@ struct hot_comm_item {
 	struct rb_node rb_node;			/* rbtree index */
 	unsigned long delete_flag;
 	struct list_head track_list;		/* link to *_map[] */
+	struct list_head reloc_list;		/* used in hot relocation*/
 };
 
 /* An item representing an inode and its access frequency */
Signature · 1 line
1.7.11.7
#2 lore raw

RFC 2/5 btrfs: add one new block group

In reply to [email protected]

From: Zhi Yong Wu <[email protected]>

Introduce one new block group BTRFS_BLOCK_GROUP_DATA_SSD,
which is used to differentiate if the block space is reserved and allocated from one HDD disk or SSD disk.

---
 fs/btrfs/Makefile       |   3 +-
 fs/btrfs/ctree.h        |  24 ++++++++++-
 fs/btrfs/extent-tree.c  | 107 +++++++++++++++++++++++++++++++++++++++++++-----
 fs/btrfs/extent_io.c    |  31 ++++++++++++--
 fs/btrfs/extent_io.h    |   4 ++
 fs/btrfs/file.c         |  36 +++++++++++++---
 fs/btrfs/hot_relocate.c |  78 +++++++++++++++++++++++++++++++++++
 fs/btrfs/hot_relocate.h |  31 ++++++++++++++
 fs/btrfs/inode-map.c    |  13 +++++-
 fs/btrfs/inode.c        |  92 +++++++++++++++++++++++++++++++++--------
 fs/btrfs/ioctl.c        |  23 +++++++++--
 fs/btrfs/relocation.c   |  14 ++++++-
 fs/btrfs/super.c        |   3 +-
 fs/btrfs/volumes.c      |  28 ++++++++++++-
 14 files changed, 439 insertions(+), 48 deletions(-)
 create mode 100644 fs/btrfs/hot_relocate.c
 create mode 100644 fs/btrfs/hot_relocate.h
fs/btrfs/Makefile +2 -1
index 3932224..94f1ea5 100644
@@ -8,7 +8,8 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
 	   export.o tree-log.o free-space-cache.o zlib.o lzo.o \
 	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
-	   reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o
+	   reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
+	   hot_relocate.o
 
 btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
 btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
fs/btrfs/ctree.h +23 -1
index 701dec5..f4c4419 100644
@@ -961,6 +961,16 @@ struct btrfs_dev_replace_item {
 #define BTRFS_BLOCK_GROUP_RAID10	(1ULL << 6)
 #define BTRFS_BLOCK_GROUP_RAID5    (1 << 7)
 #define BTRFS_BLOCK_GROUP_RAID6    (1 << 8)
+/*
+ * New block groups for use with hot data relocation feature. When hot data
+ * relocation is on, *_SSD block groups are forced to nonrotating drives and
+ * the plain DATA and METADATA block groups are forced to rotating drives.
+ *
+ * This should be further optimized, i.e. force metadata to SSD or relocate
+ * inode metadata to SSD when any of its subfile ranges are relocated to SSD
+ * so that reads and writes aren't delayed by HDD seeks.
+ */
+#define BTRFS_BLOCK_GROUP_DATA_SSD	(1ULL << 9)
 #define BTRFS_BLOCK_GROUP_RESERVED	BTRFS_AVAIL_ALLOC_BIT_SINGLE
 
 enum btrfs_raid_types {
@@ -976,7 +986,8 @@ enum btrfs_raid_types {
 
 #define BTRFS_BLOCK_GROUP_TYPE_MASK	(BTRFS_BLOCK_GROUP_DATA |    \
 					 BTRFS_BLOCK_GROUP_SYSTEM |  \
-					 BTRFS_BLOCK_GROUP_METADATA)
+					 BTRFS_BLOCK_GROUP_METADATA | \
+					 BTRFS_BLOCK_GROUP_DATA_SSD)
 
 #define BTRFS_BLOCK_GROUP_PROFILE_MASK	(BTRFS_BLOCK_GROUP_RAID0 |   \
 					 BTRFS_BLOCK_GROUP_RAID1 |   \
@@ -1508,6 +1519,7 @@ struct btrfs_fs_info {
 	struct list_head space_info;
 
 	struct btrfs_space_info *data_sinfo;
+	struct btrfs_space_info *hot_data_sinfo;
 
 	struct reloc_control *reloc_ctl;
 
@@ -1532,6 +1544,7 @@ struct btrfs_fs_info {
 	u64 avail_data_alloc_bits;
 	u64 avail_metadata_alloc_bits;
 	u64 avail_system_alloc_bits;
+	u64 avail_data_ssd_alloc_bits;
 
 	/* restriper state */
 	spinlock_t balance_lock;
@@ -1544,6 +1557,7 @@ struct btrfs_fs_info {
 
 	unsigned data_chunk_allocations;
 	unsigned metadata_ratio;
+	unsigned data_ssd_chunk_allocations;
 
 	void *bdev_holder;
 
@@ -1901,6 +1915,7 @@ struct btrfs_ioctl_defrag_range_args {
 #define BTRFS_MOUNT_CHECK_INTEGRITY_INCLUDING_EXTENT_DATA (1 << 21)
 #define BTRFS_MOUNT_PANIC_ON_FATAL_ERROR	(1 << 22)
 #define BTRFS_MOUNT_HOT_TRACK		(1 << 23)
+#define BTRFS_MOUNT_HOT_MOVE		(1 << 24)
 
 #define btrfs_clear_opt(o, opt)		((o) &= ~BTRFS_MOUNT_##opt)
 #define btrfs_set_opt(o, opt)		((o) |= BTRFS_MOUNT_##opt)
@@ -1922,6 +1937,7 @@ struct btrfs_ioctl_defrag_range_args {
 #define BTRFS_INODE_NOATIME		(1 << 9)
 #define BTRFS_INODE_DIRSYNC		(1 << 10)
 #define BTRFS_INODE_COMPRESS		(1 << 11)
+#define BTRFS_INODE_HOT			(1 << 12)
 
 #define BTRFS_INODE_ROOT_ITEM_INIT	(1 << 31)
 
@@ -3014,6 +3030,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_root *root,
 int btrfs_cross_ref_exist(struct btrfs_trans_handle *trans,
 			  struct btrfs_root *root,
 			  u64 objectid, u64 offset, u64 bytenr);
+struct btrfs_block_group_cache *btrfs_lookup_first_block_group(
+				struct btrfs_fs_info *info, u64 bytenr);
 struct btrfs_block_group_cache *btrfs_lookup_block_group(
 						 struct btrfs_fs_info *info,
 						 u64 bytenr);
@@ -3070,6 +3088,8 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 			 struct btrfs_root *root,
 			 u64 bytenr, u64 num_bytes, u64 parent,
 			 u64 root_objectid, u64 owner, u64 offset, int for_cow);
+struct btrfs_block_group_cache *next_block_group(struct btrfs_root *root,
+			 struct btrfs_block_group_cache *cache);
 
 int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans,
 				    struct btrfs_root *root);
@@ -3102,6 +3122,7 @@ enum btrfs_reserve_flush_enum {
 
 int btrfs_check_data_free_space(struct inode *inode, u64 bytes);
 void btrfs_free_reserved_data_space(struct inode *inode, u64 bytes);
+void btrfs_free_reserved_ssd_data_space(struct inode *inode, u64 bytes);
 void btrfs_trans_release_metadata(struct btrfs_trans_handle *trans,
 				struct btrfs_root *root);
 int btrfs_orphan_reserve_metadata(struct btrfs_trans_handle *trans,
@@ -3118,6 +3139,7 @@ int btrfs_delalloc_reserve_metadata(struct inode *inode, u64 num_bytes);
 void btrfs_delalloc_release_metadata(struct inode *inode, u64 num_bytes);
 int btrfs_delalloc_reserve_space(struct inode *inode, u64 num_bytes);
 void btrfs_delalloc_release_space(struct inode *inode, u64 num_bytes);
+void btrfs_delalloc_release_ssd_space(struct inode *inode, u64 num_bytes);
 void btrfs_init_block_rsv(struct btrfs_block_rsv *rsv, unsigned short type);
 struct btrfs_block_rsv *btrfs_alloc_block_rsv(struct btrfs_root *root,
 					      unsigned short type);
fs/btrfs/extent-tree.c +96 -11

241 lines not on this page · raw message

fs/btrfs/extent_io.c +28 -3
index cdee391..608b7a8 100644
@@ -1400,9 +1400,11 @@ static noinline u64 find_delalloc_range(struct extent_io_tree *tree,
 {
 	struct rb_node *node;
 	struct extent_state *state;
+	struct btrfs_root *root;
 	u64 cur_start = *start;
 	u64 found = 0;
 	u64 total_bytes = 0;
+	int flag = EXTENT_DELALLOC;
 
 	spin_lock(&tree->lock);
 
@@ -1417,13 +1419,27 @@ static noinline u64 find_delalloc_range(struct extent_io_tree *tree,
 		goto out;
 	}
 
+	root = BTRFS_I(tree->mapping->host)->root;
 	while (1) {
 		state = rb_entry(node, struct extent_state, rb_node);
 		if (found && (state->start != cur_start ||
 			      (state->state & EXTENT_BOUNDARY))) {
 			goto out;
 		}
-		if (!(state->state & EXTENT_DELALLOC)) {
+		if (btrfs_test_opt(root, HOT_MOVE)) {
+			if (!(state->state & EXTENT_DELALLOC) ||
+				(!(state->state & EXTENT_HOT) &&
+				!(state->state & EXTENT_COLD))) {
+				if (!found)
+					*end = state->end;
+				goto out;
+			} else {
+				if (!found)
+					flag = (state->state & EXTENT_HOT) ?
+						EXTENT_HOT : EXTENT_COLD;
+			}
+		}
+		if (!(state->state & flag)) {
 			if (!found)
 				*end = state->end;
 			goto out;
@@ -1610,7 +1626,13 @@ again:
 	lock_extent_bits(tree, delalloc_start, delalloc_end, 0, &cached_state);
 
 	/* then test to make sure it is all still delalloc */
-	ret = test_range_bit(tree, delalloc_start, delalloc_end,
+	if (btrfs_test_opt(BTRFS_I(inode)->root, HOT_MOVE)) {
+		ret = test_range_bit(tree, delalloc_start, delalloc_end,
+			     EXTENT_DELALLOC | EXTENT_HOT, 1, cached_state);
+		ret |= test_range_bit(tree, delalloc_start, delalloc_end,
+			     EXTENT_DELALLOC | EXTENT_COLD, 1, cached_state);
+	} else
+		ret = test_range_bit(tree, delalloc_start, delalloc_end,
 			     EXTENT_DELALLOC, 1, cached_state);
 	if (!ret) {
 		unlock_extent_cached(tree, delalloc_start, delalloc_end,
@@ -1644,7 +1666,10 @@ int extent_clear_unlock_delalloc(struct inode *inode,
 		clear_bits |= EXTENT_LOCKED;
 	if (op & EXTENT_CLEAR_DIRTY)
 		clear_bits |= EXTENT_DIRTY;
-
+	if (op & EXTENT_CLEAR_HOT)
+		clear_bits |= EXTENT_HOT;
+	if (op & EXTENT_CLEAR_COLD)
+		clear_bits |= EXTENT_COLD;
 	if (op & EXTENT_CLEAR_DELALLOC)
 		clear_bits |= EXTENT_DELALLOC;
 
fs/btrfs/extent_io.h +4 -0
index 258c921..35e155f 100644
@@ -19,6 +19,8 @@
 #define EXTENT_FIRST_DELALLOC (1 << 12)
 #define EXTENT_NEED_WAIT (1 << 13)
 #define EXTENT_DAMAGED (1 << 14)
+#define EXTENT_HOT (1 << 15)
+#define EXTENT_COLD (1 << 16)
 #define EXTENT_IOBITS (EXTENT_LOCKED | EXTENT_WRITEBACK)
 #define EXTENT_CTLBITS (EXTENT_DO_ACCOUNTING | EXTENT_FIRST_DELALLOC)
 
@@ -51,6 +53,8 @@
 #define EXTENT_END_WRITEBACK	 0x20
 #define EXTENT_SET_PRIVATE2	 0x40
 #define EXTENT_CLEAR_ACCOUNTING  0x80
+#define EXTENT_CLEAR_HOT	 0x100
+#define EXTENT_CLEAR_COLD	 0x200
 
 /*
  * page->private values.  Every page that is controlled by the extent
fs/btrfs/file.c +31 -5
index ade03e6..941b50e 100644
@@ -40,6 +40,7 @@
 #include "locking.h"
 #include "compat.h"
 #include "volumes.h"
+#include "hot_relocate.h"
 
 static struct kmem_cache *btrfs_inode_defrag_cachep;
 /*
@@ -513,6 +514,10 @@ int btrfs_dirty_pages(struct btrfs_root *root, struct inode *inode,
 	num_bytes = ALIGN(write_bytes + pos - start_pos, root->sectorsize);
 
 	end_of_last_block = start_pos + num_bytes - 1;
+
+	if (btrfs_test_opt(root, HOT_MOVE))
+		hot_set_extent(inode, start_pos, end_of_last_block, cached, 1);
+
 	err = btrfs_set_extent_delalloc(inode, start_pos, end_of_last_block,
 					cached);
 	if (err)
@@ -1372,7 +1377,12 @@ static noinline ssize_t __btrfs_buffered_write(struct file *file,
 				    pos, first_index, write_bytes,
 				    force_page_uptodate);
 		if (ret) {
-			btrfs_delalloc_release_space(inode,
+			if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+				BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+				btrfs_delalloc_release_ssd_space(inode,
+					num_pages << PAGE_CACHE_SHIFT);
+			} else
+				btrfs_delalloc_release_space(inode,
 					num_pages << PAGE_CACHE_SHIFT);
 			break;
 		}
@@ -1410,7 +1420,12 @@ static noinline ssize_t __btrfs_buffered_write(struct file *file,
 				BTRFS_I(inode)->outstanding_extents++;
 				spin_unlock(&BTRFS_I(inode)->lock);
 			}
-			btrfs_delalloc_release_space(inode,
+			if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT)
+				btrfs_delalloc_release_ssd_space(inode,
+					(num_pages - dirty_pages) <<
+					PAGE_CACHE_SHIFT);
+			else
+				btrfs_delalloc_release_space(inode,
 					(num_pages - dirty_pages) <<
 					PAGE_CACHE_SHIFT);
 		}
@@ -1420,8 +1435,13 @@ static noinline ssize_t __btrfs_buffered_write(struct file *file,
 						dirty_pages, pos, copied,
 						NULL);
 			if (ret) {
-				btrfs_delalloc_release_space(inode,
-					dirty_pages << PAGE_CACHE_SHIFT);
+				if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+					BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+					btrfs_delalloc_release_ssd_space(inode,
+						dirty_pages << PAGE_CACHE_SHIFT);
+				} else
+					btrfs_delalloc_release_space(inode,
+						dirty_pages << PAGE_CACHE_SHIFT);
 				btrfs_drop_pages(pages, num_pages);
 				break;
 			}
@@ -2282,7 +2302,13 @@ out:
 		btrfs_qgroup_free(root, alloc_end - alloc_start);
 out_reserve_fail:
 	/* Let go of our reservation. */
-	btrfs_free_reserved_data_space(inode, alloc_end - alloc_start);
+	if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+		BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+		btrfs_free_reserved_ssd_data_space(inode,
+					alloc_end - alloc_start);
+	} else
+		btrfs_free_reserved_data_space(inode,
+					alloc_end - alloc_start);
 	return ret;
 }
 
fs/btrfs/hot_relocate.c new file +78 -0

78 lines not on this page · raw message

fs/btrfs/hot_relocate.h new file +31 -0
new file mode 100644
index 0000000..b8427ba
@@ -0,0 +1,31 @@
+/*
+ * fs/btrfs/hot_relocate.h
+ *
+ * Copyright (C) 2013 IBM Corp. All rights reserved.
+ * Written by Zhi Yong Wu <[email protected]>
+ *	      Ben Chociej <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ */
+
+#ifndef __HOT_RELOCATE__
+#define __HOT_RELOCATE__
+
+#include <linux/hot_tracking.h>
+#include "ctree.h"
+#include "btrfs_inode.h"
+#include "volumes.h"
+
+enum {
+	TYPE_ROT,       /* rot -> rotating */
+	TYPE_NONROT,    /* nonrot -> nonrotating */
+	MAX_RELOC_TYPES
+};
+
+void hot_set_extent(struct inode *inode, u64 start, u64 end,
+		struct extent_state **cached_state, int flag);
+int hot_get_chunk_type(struct inode *inode, u64 start, u64 end);
+
+#endif /* __HOT_RELOCATE__ */
fs/btrfs/inode-map.c +11 -2
index d26f67a..a720135 100644
@@ -497,10 +497,19 @@ again:
 	ret = btrfs_prealloc_file_range_trans(inode, trans, 0, 0, prealloc,
 					      prealloc, prealloc, &alloc_hint);
 	if (ret) {
-		btrfs_delalloc_release_space(inode, prealloc);
+		if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+			BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+			btrfs_delalloc_release_ssd_space(inode, prealloc);
+		} else
+			btrfs_delalloc_release_space(inode, prealloc);
 		goto out_put;
 	}
-	btrfs_free_reserved_data_space(inode, prealloc);
+
+	if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+		BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+		btrfs_free_reserved_ssd_data_space(inode, prealloc);
+	} else
+		btrfs_free_reserved_data_space(inode, prealloc);
 
 	ret = btrfs_write_out_ino_cache(root, trans, path);
 out_put:
fs/btrfs/inode.c +76 -16

197 lines not on this page · raw message

fs/btrfs/ioctl.c +19 -4
index 2c02310..b9925fd 100644
@@ -56,6 +56,7 @@
 #include "rcu-string.h"
 #include "send.h"
 #include "dev-replace.h"
+#include "hot_relocate.h"
 
 /* Mask out flags that are inappropriate for the given type of inode. */
 static inline __u32 btrfs_mask_flags(umode_t mode, __u32 flags)
@@ -1098,10 +1099,17 @@ again:
 		spin_lock(&BTRFS_I(inode)->lock);
 		BTRFS_I(inode)->outstanding_extents++;
 		spin_unlock(&BTRFS_I(inode)->lock);
-		btrfs_delalloc_release_space(inode,
+		if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+			btrfs_delalloc_release_ssd_space(inode,
+				     (page_cnt - i_done) << PAGE_CACHE_SHIFT);
+		} else
+			btrfs_delalloc_release_space(inode,
 				     (page_cnt - i_done) << PAGE_CACHE_SHIFT);
 	}
 
+	if (btrfs_test_opt(BTRFS_I(inode)->root, HOT_MOVE))
+		hot_set_extent(inode, page_start,
+				page_end - 1, &cached_state, 1);
 
 	set_extent_defrag(&BTRFS_I(inode)->io_tree, page_start, page_end - 1,
 			  &cached_state, GFP_NOFS);
@@ -1124,7 +1132,13 @@ out:
 		unlock_page(pages[i]);
 		page_cache_release(pages[i]);
 	}
-	btrfs_delalloc_release_space(inode, page_cnt << PAGE_CACHE_SHIFT);
+	if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+		BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+		btrfs_delalloc_release_ssd_space(inode,
+				page_cnt << PAGE_CACHE_SHIFT);
+	} else
+		btrfs_delalloc_release_space(inode,
+				page_cnt << PAGE_CACHE_SHIFT);
 	return ret;
 
 }
@@ -3014,8 +3028,9 @@ long btrfs_ioctl_space_info(struct btrfs_root *root, void __user *arg)
 	u64 types[] = {BTRFS_BLOCK_GROUP_DATA,
 		       BTRFS_BLOCK_GROUP_SYSTEM,
 		       BTRFS_BLOCK_GROUP_METADATA,
-		       BTRFS_BLOCK_GROUP_DATA | BTRFS_BLOCK_GROUP_METADATA};
-	int num_types = 4;
+		       BTRFS_BLOCK_GROUP_DATA | BTRFS_BLOCK_GROUP_METADATA,
+		       BTRFS_BLOCK_GROUP_DATA_SSD};
+	int num_types = 5;
 	int alloc_size;
 	int ret = 0;
 	u64 slot_count = 0;
fs/btrfs/relocation.c +12 -2
index b67171e..5d44488 100644
@@ -31,6 +31,7 @@
 #include "async-thread.h"
 #include "free-space-cache.h"
 #include "inode-map.h"
+#include "hot_relocate.h"
 
 /*
  * backref_node, mapping_node and tree_block start with this
@@ -2935,8 +2936,14 @@ int prealloc_file_extent_cluster(struct inode *inode,
 			break;
 		nr++;
 	}
-	btrfs_free_reserved_data_space(inode, cluster->end +
-				       1 - cluster->start);
+
+	if (BTRFS_I(inode)->flags & BTRFS_INODE_HOT) {
+		BTRFS_I(inode)->flags &= ~BTRFS_INODE_HOT;
+		btrfs_free_reserved_ssd_data_space(inode,
+				cluster->end + 1 - cluster->start);
+	} else
+		btrfs_free_reserved_data_space(inode,
+				cluster->end + 1 - cluster->start);
 out:
 	mutex_unlock(&inode->i_mutex);
 	return ret;
@@ -3065,6 +3072,9 @@ static int relocate_file_extent_cluster(struct inode *inode,
 			nr++;
 		}
 
+		if (btrfs_test_opt(BTRFS_I(inode)->root, HOT_MOVE))
+			hot_set_extent(inode, page_start, page_end, NULL, 1);
+
 		btrfs_set_extent_delalloc(inode, page_start, page_end, NULL);
 		set_page_dirty(page);
 
fs/btrfs/super.c +2 -1
index b1bab1c..bdd8850 100644
@@ -1527,7 +1527,8 @@ static int btrfs_statfs(struct dentry *dentry, struct kstatfs *buf)
 	mutex_lock(&fs_info->chunk_mutex);
 	rcu_read_lock();
 	list_for_each_entry_rcu(found, head, list) {
-		if (found->flags & BTRFS_BLOCK_GROUP_DATA) {
+		if ((found->flags & BTRFS_BLOCK_GROUP_DATA) ||
+			(found->flags & BTRFS_BLOCK_GROUP_DATA_SSD)) {
 			total_free_data += found->disk_total - found->disk_used;
 			total_free_data -=
 				btrfs_account_ro_block_groups_free_space(found);
fs/btrfs/volumes.c +26 -2
index 2854c82..d516557 100644
@@ -1450,6 +1450,8 @@ int btrfs_rm_device(struct btrfs_root *root, char *device_path)
 		all_avail = root->fs_info->avail_data_alloc_bits |
 			    root->fs_info->avail_system_alloc_bits |
 			    root->fs_info->avail_metadata_alloc_bits;
+		if (btrfs_test_opt(root, HOT_MOVE))
+			all_avail |= root->fs_info->avail_data_ssd_alloc_bits;
 	} while (read_seqretry(&root->fs_info->profiles_lock, seq));
 
 	num_devices = root->fs_info->fs_devices->num_devices;
@@ -3736,7 +3738,8 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 	devs_increment = btrfs_raid_array[index].devs_increment;
 	ncopies = btrfs_raid_array[index].ncopies;
 
-	if (type & BTRFS_BLOCK_GROUP_DATA) {
+	if (type & BTRFS_BLOCK_GROUP_DATA ||
+		type & BTRFS_BLOCK_GROUP_DATA_SSD) {
 		max_stripe_size = 1024 * 1024 * 1024;
 		max_chunk_size = 10 * max_stripe_size;
 	} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
@@ -3775,9 +3778,30 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 		struct btrfs_device *device;
 		u64 max_avail;
 		u64 dev_offset;
+		int dev_rot;
+		int skip = 0;
 
 		device = list_entry(cur, struct btrfs_device, dev_alloc_list);
 
+		/*
+		 * If HOT_MOVE is set, the chunk type being allocated
+		 * determines which disks the data may be allocated on.
+		 * This can cause problems if, for example, the data alloc
+		 * profile is RAID0 and there are only two devices, 1 SSD +
+		 * 1 HDD. All allocations to BTRFS_BLOCK_GROUP_DATA_SSD
+		 * in this config will return -ENOSPC as the allocation code
+		 * can't find allowable space for the second stripe.
+		 */
+		dev_rot = !blk_queue_nonrot(bdev_get_queue(device->bdev));
+		if (btrfs_test_opt(extent_root, HOT_MOVE)) {
+			int ret1 = type & (BTRFS_BLOCK_GROUP_DATA |
+				BTRFS_BLOCK_GROUP_METADATA |
+				BTRFS_BLOCK_GROUP_SYSTEM) && !dev_rot;
+			int ret2 = type & BTRFS_BLOCK_GROUP_DATA_SSD && dev_rot;
+			if (ret1 || ret2)
+				skip = 1;
+		}
+
 		cur = cur->next;
 
 		if (!device->writeable) {
@@ -3786,7 +3810,7 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 			continue;
 		}
 
-		if (!device->in_fs_metadata ||
+		if (skip || !device->in_fs_metadata ||
 		    device->is_tgtdev_for_dev_replace)
 			continue;
 
Signature · 1 line
1.7.11.7
#3 lore raw

RFC 3/5 btrfs: add one hot relocation kthread

In reply to [email protected]

From: Zhi Yong Wu <[email protected]>

Add one private kthread for hot relocation. It will check
if there're some extents which is hotter than the threshold and queue them at first, if no, it will return and wait for its next turn; otherwise, it will check if SSD ratio is beyond beyond its usage threshold, if no, it will directly relocate those hot extents from HDD disk to SSD disk; otherwise it will find the extents with low temperature and queue them, then relocate those extents with low temperature and queue them, and finally relocate the hot extents from from HDD disk to SSD disk.

---
 fs/btrfs/ctree.h        |   2 +
 fs/btrfs/hot_relocate.c | 720 +++++++++++++++++++++++++++++++++++++++++++++++-
 fs/btrfs/hot_relocate.h |  21 ++
 fs/btrfs/super.c        |   1 +
 4 files changed, 742 insertions(+), 2 deletions(-)
fs/btrfs/ctree.h +2 -0
index f4c4419..77d9b1c 100644
@@ -1619,6 +1619,8 @@ struct btrfs_fs_info {
 	struct btrfs_dev_replace dev_replace;
 
 	atomic_t mutually_exclusive_operation_running;
+
+	void *hot_reloc;
 };
 
 /*
fs/btrfs/hot_relocate.c +718 -2

739 lines not on this page · raw message

fs/btrfs/hot_relocate.h +21 -0
index b8427ba..077d9b3 100644
@@ -24,8 +24,29 @@ enum {
 	MAX_RELOC_TYPES
 };
 
+#define HOT_RELOC_INTERVAL  120
+#define HOT_RELOC_THRESHOLD 150
+#define HOT_RELOC_MAX_ITEMS 250
+
+#define HEAT_MAX_VALUE    (MAP_SIZE - 1)
+#define HIGH_WATER_LEVEL  75 /* when to raise the threshold */
+#define LOW_WATER_LEVEL   50 /* when to lower the threshold */
+#define THRESH_UP_SPEED   10 /* how much to raise it by */
+#define THRESH_DOWN_SPEED 1  /* how much to lower it by */
+#define THRESH_MAX_VALUE  100
+
+struct hot_reloc {
+	struct btrfs_fs_info *fs_info;
+	struct list_head hot_relocq[MAX_RELOC_TYPES];
+	int thresh;
+	struct task_struct *hot_reloc_kthread;
+	struct mutex hot_reloc_mutex;
+};
+
 void hot_set_extent(struct inode *inode, u64 start, u64 end,
 		struct extent_state **cached_state, int flag);
 int hot_get_chunk_type(struct inode *inode, u64 start, u64 end);
+int hot_relocate_init(struct btrfs_fs_info *fs_info);
+void hot_relocate_exit(struct btrfs_fs_info *fs_info);
 
 #endif /* __HOT_RELOCATE__ */
fs/btrfs/super.c +1 -0
index bdd8850..4cbd0de 100644
@@ -57,6 +57,7 @@
 #include "compression.h"
 #include "rcu-string.h"
 #include "dev-replace.h"
+#include "hot_relocate.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/btrfs.h>
Signature · 1 line
1.7.11.7
#4 lore raw

RFC 4/5 procfs: add three proc interfaces

In reply to [email protected]

From: Zhi Yong Wu <[email protected]>

Add three proc interfaces hot-reloc-interval, hot-reloc-threshold,
and hot-reloc-max-items under the dir /proc/sys/fs/ in order to turn HOT_RELOC_INTERVAL, HOT_RELOC_THRESHOLD, and HOT_RELOC_MAX_ITEMS into be tunable.

---
 fs/btrfs/hot_relocate.c | 26 +++++++++++++++++---------
 fs/btrfs/hot_relocate.h |  4 ----
 include/linux/btrfs.h   |  4 ++++
 kernel/sysctl.c         | 22 ++++++++++++++++++++++
 4 files changed, 43 insertions(+), 13 deletions(-)
fs/btrfs/hot_relocate.c +17 -9
index 683e154..aa8c9f0 100644
@@ -25,7 +25,7 @@
  * The relocation code below operates on the heat map lists to identify
  * hot or cold data logical file ranges that are candidates for relocation.
  * The triggering mechanism for relocation is controlled by a global heat
- * threshold integer value (HOT_RELOC_THRESHOLD). Ranges are
+ * threshold integer value (sysctl_hot_reloc_threshold). Ranges are
  * queued for relocation by the periodically executing relocate kthread,
  * which updates the global heat threshold and responds to space pressure
  * on the SSDs.
@@ -52,6 +52,15 @@
  * (assuming, critically, the HOT_MOVE option is set at mount time).
  */
 
+int sysctl_hot_reloc_threshold = 150;
+EXPORT_SYMBOL_GPL(sysctl_hot_reloc_threshold);
+
+int sysctl_hot_reloc_interval __read_mostly = 120;
+EXPORT_SYMBOL_GPL(sysctl_hot_reloc_interval);
+
+int sysctl_hot_reloc_max_items __read_mostly = 250;
+EXPORT_SYMBOL_GPL(sysctl_hot_reloc_max_items);
+
 static void hot_set_extent_bits(struct extent_io_tree *tree, u64 start,
 		u64 end, struct extent_state **cached_state,
 		gfp_t mask, int storage_type, int flag)
@@ -165,7 +174,7 @@ static int hot_calc_ssd_ratio(struct hot_reloc *hot_reloc)
 static int hot_update_threshold(struct hot_reloc *hot_reloc,
 				int update)
 {
-	int thresh = hot_reloc->thresh;
+	int thresh = sysctl_hot_reloc_threshold;
 	int ratio = hot_calc_ssd_ratio(hot_reloc);
 
 	/* Sometimes update global threshold, others not */
@@ -189,7 +198,7 @@ static int hot_update_threshold(struct hot_reloc *hot_reloc,
 			thresh = 0;
 	}
 
-	hot_reloc->thresh = thresh;
+	sysctl_hot_reloc_threshold = thresh;
 	return ratio;
 }
 
@@ -280,7 +289,7 @@ static int hot_queue_extent(struct hot_reloc *hot_reloc,
 		hot_comm_item_put(ci);
 		spin_unlock(&he->i_lock);
 
-		if (*counter >= HOT_RELOC_MAX_ITEMS)
+		if (*counter >= sysctl_hot_reloc_max_items)
 			break;
 
 		if (kthread_should_stop()) {
@@ -361,7 +370,7 @@ again:
 		while (1) {
 			lock_extent(tree, page_start, page_end);
 			ordered = btrfs_lookup_ordered_extent(inode,
-							page_start);
+							      page_start);
 			unlock_extent(tree, page_start, page_end);
 			if (!ordered)
 				break;
@@ -642,7 +651,7 @@ void hot_do_relocate(struct hot_reloc *hot_reloc)
 
 	run++;
 	ratio = hot_update_threshold(hot_reloc, !(run % 15));
-	thresh = hot_reloc->thresh;
+	thresh = sysctl_hot_reloc_threshold;
 
 	INIT_LIST_HEAD(&hot_reloc->hot_relocq[TYPE_NONROT]);
 
@@ -652,7 +661,7 @@ void hot_do_relocate(struct hot_reloc *hot_reloc)
 	if (count_to_hot == 0)
 		return;
 
-	count_to_cold = HOT_RELOC_MAX_ITEMS;
+	count_to_cold = sysctl_hot_reloc_max_items;
 
 	/* Don't move cold data to HDD unless there's space pressure */
 	if (ratio < HIGH_WATER_LEVEL)
@@ -734,7 +743,7 @@ static int hot_relocate_kthread(void *arg)
 	unsigned long delay;
 
 	do {
-		delay = HZ * HOT_RELOC_INTERVAL;
+		delay = HZ * sysctl_hot_reloc_interval;
 		if (mutex_trylock(&hot_reloc->hot_reloc_mutex)) {
 			hot_do_relocate(hot_reloc);
 			mutex_unlock(&hot_reloc->hot_reloc_mutex);
@@ -766,7 +775,6 @@ int hot_relocate_init(struct btrfs_fs_info *fs_info)
 
 	fs_info->hot_reloc = hot_reloc;
 	hot_reloc->fs_info = fs_info;
-	hot_reloc->thresh = HOT_RELOC_THRESHOLD;
 	for (i = 0; i < MAX_RELOC_TYPES; i++)
 		INIT_LIST_HEAD(&hot_reloc->hot_relocq[i]);
 	mutex_init(&hot_reloc->hot_reloc_mutex);
fs/btrfs/hot_relocate.h +0 -4
index 077d9b3..ca30944 100644
@@ -24,9 +24,6 @@ enum {
 	MAX_RELOC_TYPES
 };
 
-#define HOT_RELOC_INTERVAL  120
-#define HOT_RELOC_THRESHOLD 150
-#define HOT_RELOC_MAX_ITEMS 250
 
 #define HEAT_MAX_VALUE    (MAP_SIZE - 1)
 #define HIGH_WATER_LEVEL  75 /* when to raise the threshold */
@@ -38,7 +35,6 @@ enum {
 struct hot_reloc {
 	struct btrfs_fs_info *fs_info;
 	struct list_head hot_relocq[MAX_RELOC_TYPES];
-	int thresh;
 	struct task_struct *hot_reloc_kthread;
 	struct mutex hot_reloc_mutex;
 };
include/linux/btrfs.h +4 -0
index 22d7991..7179819 100644
@@ -3,4 +3,8 @@
 
 #include <uapi/linux/btrfs.h>
 
+extern int sysctl_hot_reloc_threshold;
+extern int sysctl_hot_reloc_interval;
+extern int sysctl_hot_reloc_max_items;
+
 #endif /* _LINUX_BTRFS_H */
kernel/sysctl.c +22 -0
index 11e4a3d..c1db57e 100644
@@ -62,6 +62,7 @@
 #include <linux/capability.h>
 #include <linux/binfmts.h>
 #include <linux/sched/sysctl.h>
+#include <linux/btrfs.h>
 
 #include <asm/uaccess.h>
 #include <asm/processor.h>
@@ -1617,6 +1618,27 @@ static struct ctl_table fs_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec,
 	},
+	{
+		.procname	= "hot-reloc-threshold",
+		.data		= &sysctl_hot_reloc_threshold,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
+		.procname	= "hot-reloc-interval",
+		.data		= &sysctl_hot_reloc_interval,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
+		.procname	= "hot-reloc-max-items",
+		.data		= &sysctl_hot_reloc_max_items,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
 	{ }
 };
 
Signature · 1 line
1.7.11.7
#5 lore raw

RFC 5/5 btrfs: add hot relocation support

In reply to [email protected]

From: Zhi Yong Wu <[email protected]>

Add one new mount option '-o hot_move' for hot
relocation support. When hot relocation is enabled, hot tracking will be enabled automatically.
Its usage looks like:
mount -o hot_move
mount -o nouser,hot_move
mount -o nouser,hot_move,loop
mount -o hot_move,nouser

---
 fs/btrfs/super.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
fs/btrfs/super.c +23 -3
index 4cbd0de..b342f6f 100644
@@ -311,8 +311,13 @@ static void btrfs_put_super(struct super_block *sb)
 	 * process...  Whom would you report that to?
 	 */
 
+	/* Hot data relocation */
+	if (btrfs_test_opt(btrfs_sb(sb)->tree_root, HOT_MOVE))
+		hot_relocate_exit(btrfs_sb(sb));
+
 	/* Hot data tracking */
-	if (btrfs_test_opt(btrfs_sb(sb)->tree_root, HOT_TRACK))
+	if (btrfs_test_opt(btrfs_sb(sb)->tree_root, HOT_MOVE)
+		|| btrfs_test_opt(btrfs_sb(sb)->tree_root, HOT_TRACK))
 		hot_track_exit(sb);
 }
 
@@ -327,7 +332,7 @@ enum {
 	Opt_no_space_cache, Opt_recovery, Opt_skip_balance,
 	Opt_check_integrity, Opt_check_integrity_including_extent_data,
 	Opt_check_integrity_print_mask, Opt_fatal_errors, Opt_hot_track,
-	Opt_err,
+	Opt_hot_move, Opt_err,
 };
 
 static match_table_t tokens = {
@@ -368,6 +373,7 @@ static match_table_t tokens = {
 	{Opt_check_integrity_print_mask, "check_int_print_mask=%d"},
 	{Opt_fatal_errors, "fatal_errors=%s"},
 	{Opt_hot_track, "hot_track"},
+	{Opt_hot_move, "hot_move"},
 	{Opt_err, NULL},
 };
 
@@ -636,6 +642,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options)
 		case Opt_hot_track:
 			btrfs_set_opt(info->mount_opt, HOT_TRACK);
 			break;
+		case Opt_hot_move:
+			btrfs_set_opt(info->mount_opt, HOT_MOVE);
+			break;
 		case Opt_err:
 			printk(KERN_INFO "btrfs: unrecognized mount option "
 			       "'%s'\n", p);
@@ -863,17 +872,26 @@ static int btrfs_fill_super(struct super_block *sb,
 		goto fail_close;
 	}
 
-	if (btrfs_test_opt(fs_info->tree_root, HOT_TRACK)) {
+	if (btrfs_test_opt(fs_info->tree_root, HOT_MOVE)
+		|| btrfs_test_opt(fs_info->tree_root, HOT_TRACK)) {
 		err = hot_track_init(sb);
 		if (err)
 			goto fail_hot;
 	}
 
+	if (btrfs_test_opt(fs_info->tree_root, HOT_MOVE)) {
+		err = hot_relocate_init(fs_info);
+		if (err)
+			goto fail_reloc;
+	}
+
 	save_mount_options(sb, data);
 	cleancache_init_fs(sb);
 	sb->s_flags |= MS_ACTIVE;
 	return 0;
 
+fail_reloc:
+	hot_track_exit(sb);
 fail_hot:
 	dput(sb->s_root);
 	sb->s_root = NULL;
@@ -974,6 +992,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
 		seq_puts(seq, ",fatal_errors=panic");
 	if (btrfs_test_opt(root, HOT_TRACK))
 		seq_puts(seq, ",hot_track");
+	if (btrfs_test_opt(root, HOT_MOVE))
+		seq_puts(seq, ",hot_move");
 	return 0;
 }
 
Signature · 1 line
1.7.11.7
#6 lore raw
In reply to [email protected]
Quoted 8 lines · [email protected] <[email protected]> schrieb:

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

How will it compare to bcache? I'm currently thinking about buying an SSD but bcache requires some efforts in migrating the storage to use. And after all those hassles I am even not sure if it would work easily with a dracut generated initramfs.

Bcache seems to be quite clever with its approach. This one looks completely different and more targetted to relocate data which is used often instead of trying to reduce head movement. I'm quite happy with the throuput of my 3x HDD btrfs pool (according to bootchart up to 600 MB/s during boot). A single SSD would be slower since head movement seems not to be the issue during boot. Will this patch relocate such data? Or does it try to relocate only data which requires random head movement?

Thanks,
Kai

#7 lore raw
Quoted 11 lines · Kai Krakow
Quoted 8 lines · [email protected] <[email protected]> schrieb:

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

Quoted 4 lines

How will it compare to bcache? I'm currently thinking about buying an SSD but bcache requires some efforts in migrating the storage to use. And after all those hassles I am even not sure if it would work easily with a dracut generated initramfs.

On the side note: dm-cache, which is already in-kernel, do not need to
reformat backing storage.

Signature · 2 lines
Tomasz Torcz                Only gods can safely risk perfection,
xmpp: [email protected]     it's a dangerous thing for a man.  -- Alia
#8 lore raw
Quoted 7 lines · Tomasz Torcz
Quoted 4 lines

How will it compare to bcache? I'm currently thinking about buying an SSD but bcache requires some efforts in migrating the storage to use. And after all those hassles I am even not sure if it would work easily with a dracut generated initramfs.

On the side note: dm-cache, which is already in-kernel, do not need to
reformat backing storage.

Oh thanks, good pointer. I haven't looked so much into dm-cache yet because it seemed to have come "out of nowhere" while I was tracking news around bcache development for a while now.

Regards,
Kai

#9 lore raw
In reply to [email protected]
Quoted 7 lines
Quoted 4 lines

How will it compare to bcache? I'm currently thinking about buying an SSD but bcache requires some efforts in migrating the storage to use. And after all those hassles I am even not sure if it would work easily with a dracut generated initramfs.

On the side note: dm-cache, which is already in-kernel, do not need to
reformat backing storage.

On the other hand dm-cache is somewhat complex to assemble, and letting the system automount the unsynchronised backing device is a recipe for data loss.

It will need lvm integration to become really convenient to use.

Anyway, here's a shameless plug for a tool that converts to bcache
in-place: https://github.com/g2p/blocks#bcache-conversion

#10 lore raw
Quoted 11 lines · Gabriel de Perthuis
Quoted 7 lines
Quoted 4 lines

How will it compare to bcache? I'm currently thinking about buying an SSD but bcache requires some efforts in migrating the storage to use. And after all those hassles I am even not sure if it would work easily with a dracut generated initramfs.

On the side note: dm-cache, which is already in-kernel, do not need to
reformat backing storage.

On the other hand dm-cache is somewhat complex to assemble, and letting the system automount the unsynchronised backing device is a recipe for data loss.

Yes, that was my first impression, too, after reading of how it works. How safe is bcache on that matter?

Quoted 2 lines

Anyway, here's a shameless plug for a tool that converts to bcache
in-place: https://github.com/g2p/blocks#bcache-conversion

Did I say: I love your shameless plugs? ;-)

I've read the docs for this tool with interest. Still I do not feel very comfortable with converting my storage for some unknown outcome. Sure, I can take backups (and by any means: I will). But it takes time: backup, try, restore, try again, maybe restore... I don't want to find out that it was all useless because it's just not ready to boot a multi-device btrfs through dracut. So you see, the point is: Will that work? I didn't see any docs answering my questions.

Of course, if it would work I'd happily contribute documentation to your project.

Regards,
Kai

#11 lore raw
Quoted 10 lines · Kai Krakow
Quoted 6 lines · Gabriel de Perthuis
Quoted 2 lines

On the side note: dm-cache, which is already in-kernel, do not need to
reformat backing storage.

On the other hand dm-cache is somewhat complex to assemble, and letting the system automount the unsynchronised backing device is a recipe for data loss.

Yes, that was my first impression, too, after reading of how it works. How safe is bcache on that matter?

The bcache superblock is there just to prevent the naked backing device from becoming available. So it's safe in that respect. LVM has something similar with hidden volumes.

Quoted 5 lines
Quoted 2 lines

Anyway, here's a shameless plug for a tool that converts to bcache
in-place: https://github.com/g2p/blocks#bcache-conversion

Did I say: I love your shameless plugs? ;-)

Quoted 7 lines

I've read the docs for this tool with interest. Still I do not feel very comfortable with converting my storage for some unknown outcome. Sure, I can take backups (and by any means: I will). But it takes time: backup, try, restore, try again, maybe restore... I don't want to find out that it was all useless because it's just not ready to boot a multi-device btrfs through dracut. So you see, the point is: Will that work? I didn't see any docs answering my questions.

Try it with a throwaway filesystem inside a VM. The bcache list will appreciate the feedback on Dracut, even if you don't make the switch for real.

Quoted 2 lines

Of course, if it would work I'd happily contribute documentation to your project.

That would be very welcome.

#12 lore raw
In reply to [email protected]

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

Quoted 116 lines · On Mon, May 6, 2013 at 4:53 PM, <[email protected]> wrote:

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc -f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1 transid 16 /dev/vdb
[ 140.283650] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.550759] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2 transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB
Btrfs v0.20-rc1-254-gb0136aa-dirty
root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 6 /dev/vdb
[ 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature is currently 109
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~#

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

 fs/btrfs/Makefile            |   3 +-
 fs/btrfs/ctree.h             |  26 +-
 fs/btrfs/extent-tree.c       | 107 +++++-
 fs/btrfs/extent_io.c         |  31 +-
 fs/btrfs/extent_io.h         |   4 +
 fs/btrfs/file.c              |  36 +-
 fs/btrfs/hot_relocate.c      | 802 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/hot_relocate.h      |  48 +++
 fs/btrfs/inode-map.c         |  13 +-
 fs/btrfs/inode.c             |  92 ++++-
 fs/btrfs/ioctl.c             |  23 +-
 fs/btrfs/relocation.c        |  14 +-
 fs/btrfs/super.c             |  30 +-
 fs/btrfs/volumes.c           |  28 +-
 fs/hot_tracking.c            |   1 +
 include/linux/btrfs.h        |   4 +
 include/linux/hot_tracking.h |   1 +
 kernel/sysctl.c              |  22 ++
 18 files changed, 1234 insertions(+), 51 deletions(-)
 create mode 100644 fs/btrfs/hot_relocate.c
 create mode 100644 fs/btrfs/hot_relocate.h

--
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Regards,

Zhi Yong Wu
#13 lore raw
Quoted 6 lines · Zhi Yong Wu

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

Which one is better?

Please do some measurements. Select typical file system use cases, and publish and compare the measurement results of the two approaches.

Quoted 109 lines · On Mon, May 6, 2013 at 4:53 PM, <[email protected]> wrote:
Quoted 108 lines

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc -f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1 transid 16 /dev/vdb
[ 140.283650] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.550759] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2 transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB
Btrfs v0.20-rc1-254-gb0136aa-dirty
root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 6 /dev/vdb
[ 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature is currently 109
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~#

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

  fs/btrfs/Makefile            |   3 +-
  fs/btrfs/ctree.h             |  26 +-
  fs/btrfs/extent-tree.c       | 107 +++++-
  fs/btrfs/extent_io.c         |  31 +-
  fs/btrfs/extent_io.h         |   4 +
  fs/btrfs/file.c              |  36 +-
  fs/btrfs/hot_relocate.c      | 802 +++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/hot_relocate.h      |  48 +++
  fs/btrfs/inode-map.c         |  13 +-
  fs/btrfs/inode.c             |  92 ++++-
  fs/btrfs/ioctl.c             |  23 +-
  fs/btrfs/relocation.c        |  14 +-
  fs/btrfs/super.c             |  30 +-
  fs/btrfs/volumes.c           |  28 +-
  fs/hot_tracking.c            |   1 +
  include/linux/btrfs.h        |   4 +
  include/linux/hot_tracking.h |   1 +
  kernel/sysctl.c              |  22 ++
  18 files changed, 1234 insertions(+), 51 deletions(-)

create mode 100644 fs/btrfs/hot_relocate.c
create mode 100644 fs/btrfs/hot_relocate.h

#14 lore raw

btrfs maintainer's opinion is very important, i guess.

Quoted 139 lines · Stefan Behrens
Quoted 7 lines · Zhi Yong Wu

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

Which one is better?

Please do some measurements. Select typical file system use cases, and publish and compare the measurement results of the two approaches.

Quoted 120 lines · On Mon, May 6, 2013 at 4:53 PM, <[email protected]> wrote:
Quoted 119 lines

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc
-f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1
transid 16 /dev/vdb
[ 140.283650] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2
transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
transid 3 /dev/vdb
[ 140.550759] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2
transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2

transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB Btrfs v0.20-rc1-254-gb0136aa-dirty

root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
transid 6 /dev/vdb
[ 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature
is currently 109
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~#

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

  fs/btrfs/Makefile            |   3 +-
  fs/btrfs/ctree.h             |  26 +-
  fs/btrfs/extent-tree.c       | 107 +++++-
  fs/btrfs/extent_io.c         |  31 +-
  fs/btrfs/extent_io.h         |   4 +
  fs/btrfs/file.c              |  36 +-
  fs/btrfs/hot_relocate.c      | 802
+++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/hot_relocate.h      |  48 +++
  fs/btrfs/inode-map.c         |  13 +-
  fs/btrfs/inode.c             |  92 ++++-
  fs/btrfs/ioctl.c             |  23 +-
  fs/btrfs/relocation.c        |  14 +-
  fs/btrfs/super.c             |  30 +-
  fs/btrfs/volumes.c           |  28 +-
  fs/hot_tracking.c            |   1 +
  include/linux/btrfs.h        |   4 +
  include/linux/hot_tracking.h |   1 +
  kernel/sysctl.c              |  22 ++
  18 files changed, 1234 insertions(+), 51 deletions(-)

create mode 100644 fs/btrfs/hot_relocate.c
create mode 100644 fs/btrfs/hot_relocate.h

Signature · 3 lines
Regards,

Zhi Yong Wu
#15 lore raw
Quoted 1 line · Zhi Yong Wu

btrfs maintainer's opinion is very important, i guess.

My opinion is not important and I shall shut up?

Quoted 16 lines
Quoted 14 lines · Stefan Behrens
Quoted 7 lines · Zhi Yong Wu

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

Which one is better?

Please do some measurements. Select typical file system use cases, and publish and compare the measurement results of the two approaches.

#16 lore raw

No, no, you misunderstood what i mean.:) You know, it will perhaps take a lot of time to do some benchmark. We only hope to get some suggestion about if it should be worth to continue working on. Very welcome to your and other guys' opinions.

Quoted 30 lines · Stefan Behrens
Quoted 2 lines · Zhi Yong Wu

btrfs maintainer's opinion is very important, i guess.

My opinion is not important and I shall shut up?

Quoted 19 lines
Quoted 17 lines · Stefan Behrens
Quoted 8 lines · Zhi Yong Wu

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

Which one is better?

Please do some measurements. Select typical file system use cases, and publish and compare the measurement results of the two approaches.

Signature · 3 lines
Regards,

Zhi Yong Wu
#17 lore raw
In reply to Stefan Behrens
Quoted 8 lines · Stefan Behrens
Quoted 6 lines · Zhi Yong Wu

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

Quoted 5 lines

Which one is better?

Please do some measurements. Select typical file system use cases, and publish and compare the measurement results of the two approaches.

Hi Stefan,

AFAIU, the key issue is that the hot relocation feature should be implemented in file system or in block device because file system knows which data is hot, and the application could use fadvise/ioctl/... interfaces to give a hint to file system to keep some data in fast device. But IIUC dm-cache/bcache only can do is like: "hey, this data should be hot just because it is touched twice." In some cases, touched twice not always means that it should be kept in fast device.

Regards,
- Zheng

#18 lore raw
In reply to Zhi Yong Wu

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 1 line · Zhi Yong Wu

i want to know if btrfs hot relocation support is still meanful

It is to me. The problem with bcache is that it is a cache. ie if you have a 256GB SSD and a 500GB HDD then you'll have total storage of 500GB.
Hot relocation was described as having 750GB available in the same scenario.

I'd also expect btrfs level support to be more friendly such as being able to mount -o degraded if one of the devices is missing.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlGLSIkACgkQmOOfHg372QSHQACeLwbhnVsH7+/6ZSIaGAcMUyBe gPwAoNAKBAFB65XZQbLyxyCJPHODR+9z
=6OXt
-----END PGP SIGNATURE-----

#19 lore raw
In reply to Zhi Yong Wu

Am Donnerstag, 9. Mai 2013, 07:13:56 schrieb Zhi Yong Wu:

Quoted 1 line

HI, all

Hi!

Quoted 4 lines

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

I really look forward to VFS hot data tracking with BTRFS (and other filesystem) support.

ZFS and BTRFS have shown that RAID support within the filesystem can make a lot of sense. I think hot relocation is another area which can be done more accurate within the filesystem. I think there are several features only possible by going this route:

1) Mark files as hot via ioctl. MySQL can mark InnoDB journal files for example.

2) Proper BTRFS RAID support (as noted elsewhere in this thread)

3) Easier setup. With BTRFS flexibility I would expect that a SSD as hot data cache can be added and removed on the fly during filesystem is mounted. Only seems supported at mkfs-time as I read the patch docs, but from my basic technical understanding of BTRFS it can be extented to be done on the fly with a mounted FS as well.

Block based caching can make sense for other use cases like raw device with Oracle DB on it or maybe a swap device. Or for filesystems not (yet) supporting VFS hot data tracking.

Thanks,

Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
#20 lore raw
Quoted 5 lines · Martin Steigerwald

Am Donnerstag, 9. Mai 2013, 07:13:56 schrieb Zhi Yong Wu:

Quoted 1 line

HI, all

Hi!

Quoted 7 lines
Quoted 4 lines

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

I really look forward to VFS hot data tracking with BTRFS (and other filesystem) support.

I also really want to see this will happen.

Quoted 7 lines

ZFS and BTRFS have shown that RAID support within the filesystem can make a lot of sense. I think hot relocation is another area which can be done more accurate within the filesystem. I think there are several features only possible by going this route:

1) Mark files as hot via ioctl. MySQL can mark InnoDB journal files for example.

It has been implemented to get its hot status of one specified file via ioctl. It is not hard to enable its set function. Before it is done, i more hope that the core of VFS hot tracking can get merged by kernel upstream. You know, it is harder if too new functions have been done.
We can put it in to-do list at first, what do you think?

Quoted 8 lines

2) Proper BTRFS RAID support (as noted elsewhere in this thread)

3) Easier setup. With BTRFS flexibility I would expect that a SSD as hot data cache can be added and removed on the fly during filesystem is mounted. Only seems supported at mkfs-time as I read the patch docs, but from my basic technical understanding of BTRFS it can be extented to be done on the fly with a mounted FS as well.

Yes, it is supported only at mkfs time. It shouldn't be hard to enable adding or removing a nonrotating disk as hot cache on the fly. You know, before this is done, i would like to make sure that the current design for BTRFS hot relocation is this RFC patchset is going in the direction and is appropriate. If it is going in wrong direction, we will be doing a lot of meanless work. E.g. How about the current idea on introducing one new block group for nonrotating disks. When working on this work, i were trying to change the least current btrfs code as far as possible. You know, if we change too many current btrfs code, it will introduce regression bug more easily, and so the patchset will be also harder to get accepted by btrfs upstream.

What do you think?

Quoted 10 lines

Block based caching can make sense for other use cases like raw device with Oracle DB on it or maybe a swap device. Or for filesystems not (yet) supporting VFS hot data tracking.

Thanks,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7

--
Regards,

Zhi Yong Wu

#21 lore raw

Am Sonntag, 19. Mai 2013, 21:43:14 schrieb Zhi Yong Wu:

Quoted 1 line

On Sun, May 19, 2013 at 6:41 PM, Martin Steigerwald <[email protected]>

Quoted 1 line · wrote:
Quoted 1 line

Am Donnerstag, 9. Mai 2013, 07:13:56 schrieb Zhi Yong Wu:

[…]

Quoted 5 lines
Quoted 5 lines

ZFS and BTRFS have shown that RAID support within the filesystem can make a lot of sense. I think hot relocation is another area which can be done more accurate within the filesystem. I think there are several features only possible by going this route:

Quoted 8 lines
Quoted 2 lines

1) Mark files as hot via ioctl. MySQL can mark InnoDB journal files for example.

It has been implemented to get its hot status of one specified file via ioctl. It is not hard to enable its set function. Before it is done, i more hope that the core of VFS hot tracking can get merged by kernel upstream. You know, it is harder if too new functions have been done.
We can put it in to-do list at first, what do you think?

Yes, of course. I didn´t want to imply that you have to implement it. I didn´t contribute to BTRFS code yet, so I do not see myself in a position to request anything from you – and even if I did, what you code is still your decision. Many thanks for your work on this!

I just wanted to note that IMHO its possible with the VFS approach while its at least difficult to do with the block layer based approaches.

Quoted 20 lines
Quoted 7 lines

2) Proper BTRFS RAID support (as noted elsewhere in this thread)

3) Easier setup. With BTRFS flexibility I would expect that a SSD as hot data cache can be added and removed on the fly during filesystem is mounted. Only seems supported at mkfs-time as I read the patch docs, but from my basic technical understanding of BTRFS it can be extented to be done on the fly with a mounted FS as well.

Yes, it is supported only at mkfs time. It shouldn't be hard to enable adding or removing a nonrotating disk as hot cache on the fly. You know, before this is done, i would like to make sure that the current design for BTRFS hot relocation is this RFC patchset is going in the direction and is appropriate. If it is going in wrong direction, we will be doing a lot of meanless work. E.g. How about the current idea on introducing one new block group for nonrotating disks. When working on this work, i were trying to change the least current btrfs code as far as possible. You know, if we change too many current btrfs code, it will introduce regression bug more easily, and so the patchset will be also harder to get accepted by btrfs upstream.

Quoted 1 line

What do you think?

Yes, see above. Keep it simple and extend later I´d say :)

Thanks,

Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
#22 lore raw
In reply to Martin Steigerwald
Quoted 13 lines · Martin Steigerwald

Am Donnerstag, 9. Mai 2013, 07:13:56 schrieb Zhi Yong Wu:

Quoted 1 line

HI, all

Hi!

Quoted 4 lines

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

I really look forward to VFS hot data tracking with BTRFS (and other filesystem) support.

Quoted 8 lines

ZFS and BTRFS have shown that RAID support within the filesystem can make a lot of sense. I think hot relocation is another area which can be done more accurate within the filesystem. I think there are several features only possible by going this route:

1) Mark files as hot via ioctl. MySQL can mark InnoDB journal files for example.

2) Proper BTRFS RAID support (as noted elsewhere in this thread)

After the current design draft for BTRFS hot relocation in this RFC get basic agreement as btrfs community, maintainer or main developers, i will dive into this further.
By the way, it is in my to-do list, thanks.

Quoted 8 lines

3) Easier setup. With BTRFS flexibility I would expect that a SSD as hot data cache can be added and removed on the fly during filesystem is mounted. Only seems supported at mkfs-time as I read the patch docs, but from my basic technical understanding of BTRFS it can be extented to be done on the fly with a mounted FS as well.

Quoted 8 lines

Block based caching can make sense for other use cases like raw device with Oracle DB on it or maybe a swap device. Or for filesystems not (yet) supporting VFS hot data tracking.

Thanks,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7

--
Regards,

Zhi Yong Wu

#23 lore raw
In reply to [email protected]
Quoted 6 lines · Zhi Yong Wu

HI, all

I saw that bcache will be merged into kernel upstream soon, so i
want to know if btrfs hot relocation support is still meanful, if no, i will not continue to work on it. can anyone let me know this? thanks.

bcache performance would be poor if Btrfs-raid1 is used; the hits will be spread across mirrors are random, bcache will have to cache twice as much and convergence could be slow.

Hot tracking also has much more precise info about which files are hot, whereas bcache can't know anything about files that are in the page cache and risks evicting some of them.

It all comes down to benchmarks, but these two situations could be easy wins for hot-reloc.

#24 lore raw
In reply to [email protected]

HI,

What do you think if its design approach goes correctly? Do you
have any comments or better design idea for BTRFS hot relocation support? any comments are appreciated, thanks.

Quoted 116 lines · On Mon, May 6, 2013 at 4:53 PM, <[email protected]> wrote:

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc -f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1 transid 16 /dev/vdb
[ 140.283650] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.550759] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2 transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB
Btrfs v0.20-rc1-254-gb0136aa-dirty
root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 6 /dev/vdb
[ 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature is currently 109
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~#

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

 fs/btrfs/Makefile            |   3 +-
 fs/btrfs/ctree.h             |  26 +-
 fs/btrfs/extent-tree.c       | 107 +++++-
 fs/btrfs/extent_io.c         |  31 +-
 fs/btrfs/extent_io.h         |   4 +
 fs/btrfs/file.c              |  36 +-
 fs/btrfs/hot_relocate.c      | 802 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/hot_relocate.h      |  48 +++
 fs/btrfs/inode-map.c         |  13 +-
 fs/btrfs/inode.c             |  92 ++++-
 fs/btrfs/ioctl.c             |  23 +-
 fs/btrfs/relocation.c        |  14 +-
 fs/btrfs/super.c             |  30 +-
 fs/btrfs/volumes.c           |  28 +-
 fs/hot_tracking.c            |   1 +
 include/linux/btrfs.h        |   4 +
 include/linux/hot_tracking.h |   1 +
 kernel/sysctl.c              |  22 ++
 18 files changed, 1234 insertions(+), 51 deletions(-)
 create mode 100644 fs/btrfs/hot_relocate.c
 create mode 100644 fs/btrfs/hot_relocate.h

--
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Regards,

Zhi Yong Wu
#25 lore raw

Hi!

I think such a solution as part of the filesystem could do much better than something outside of it (like bcache). But I'm not sure: What makes data hot? I think the most benefit is detecting random read access and mark only those data as hot, also writes should go to the SSD first and then should be spooled to the harddisks in background. Bcache does a lot regarding this.

Since this is within the filesystem, users could even mark files as being always "hot" with some attribute or ioctl. This could be used by a boot- readahead and preload implementation to automatically make files hot used during booting or for preloading when I start an application.

On the other side hot relocation should be able to reduce writes to the SSD as good as possible, for example: Do not defragment files during autodefrag, it makes no sense. Also write data in bursts of erase block size etc.

And also important: What if the SSD dies due to wearing? Will it gracefully fall back to harddisk? What does "relocation" mean? Files (hot data) should only be cached in copy to SSD, and not moved there. It should be possible for btrfs to just drop a failing SSD from the filesystem without data loss because otherwise one should use two SSDs in raid-1 mode to get a safe cache storage.

Altogether I think that a spinning media btrfs raid can outperform a single SSD so hot relocation should probably be used to reduce head movements because this is where SSD really excels. So everything that involves heavy head movement should go to SSD first, then written back to harddisk. And I think there's a lot potential to optimize because a COW filesystem like btrfs naturally has a lot of head movement.

What do you think?

BTW: I have not tried the one or the other yet because I'm still deciding which way to go. Your patches are more welcome because I do not need to migrate my storage to bcache-provided block devices. OTOH the bcache implementation looks a lot more mature (with regard to performance and safety) at this point because it provides many of the above mentioned features - most importantly gracefully handling failing SSDs.

Regarding btrfs raid outperforms SSD: During boot my spinning media 3 device btrfs raid reads boot files with up to 600 MB/s (from LZ compressed fs), boot takes about 7 seconds until the display manager starts (which takes another 30 seconds but that's another story), and the system is pretty crowded with services I actually wouldn't need if I optimized for boot performance. But I think systemd's read-ahead implementation has a lot influence on this fast booting: It defragments and relocates boot files on btrfs during boot so the harddisks can sequentially read all this stuff. I think it also compresses boot files if compression is enabled because booting is IO bound, not CPU bound. Benchmarks showed that my btrfs raid could technically read up to 450 MB/s, so I think the 600 MB/s counts for decompressed data. A single SSD could not do that. For that same reason I created a small script to defragment and compress files used by the preload daemon. Without benchmarking it, this felt like another small performance boost. So I'm eager what could be next with some sort of SSD cache because the only problem left seems to be heavy head movement which slows down the system.

Quoted 133 lines · Zhi Yong Wu

HI,

What do you think if its design approach goes correctly? Do you
have any comments or better design idea for BTRFS hot relocation support? any comments are appreciated, thanks.

Quoted 122 lines · On Mon, May 6, 2013 at 4:53 PM, <[email protected]> wrote:

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc
-f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1
[ transid 16 /dev/vdb 140.283650] device fsid
[ c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
[ transid 3 /dev/vdb 140.550759] device fsid
[ 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2
[ transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2

[ transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB Btrfs v0.20-rc1-254-gb0136aa-dirty

root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
[ transid 6 /dev/vdb 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature
is currently 109 root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~#

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

 fs/btrfs/Makefile            |   3 +-
 fs/btrfs/ctree.h             |  26 +-
 fs/btrfs/extent-tree.c       | 107 +++++-
 fs/btrfs/extent_io.c         |  31 +-
 fs/btrfs/extent_io.h         |   4 +
 fs/btrfs/file.c              |  36 +-
 fs/btrfs/hot_relocate.c      | 802
 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/hot_relocate.h      |  48 +++
 fs/btrfs/inode-map.c         |  13 +-
 fs/btrfs/inode.c             |  92 ++++-
 fs/btrfs/ioctl.c             |  23 +-
 fs/btrfs/relocation.c        |  14 +-
 fs/btrfs/super.c             |  30 +-
 fs/btrfs/volumes.c           |  28 +-
 fs/hot_tracking.c            |   1 +
 include/linux/btrfs.h        |   4 +
 include/linux/hot_tracking.h |   1 +
 kernel/sysctl.c              |  22 ++
 18 files changed, 1234 insertions(+), 51 deletions(-)
 create mode 100644 fs/btrfs/hot_relocate.c
 create mode 100644 fs/btrfs/hot_relocate.h

--
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#26 lore raw

HI,

You should check the patchset about VFS hot tracking
https://lwn.net/Articles/550495/

Quoted 199 lines · Kai Krakow

Hi!

I think such a solution as part of the filesystem could do much better than something outside of it (like bcache). But I'm not sure: What makes data hot? I think the most benefit is detecting random read access and mark only those data as hot, also writes should go to the SSD first and then should be spooled to the harddisks in background. Bcache does a lot regarding this.

Since this is within the filesystem, users could even mark files as being always "hot" with some attribute or ioctl. This could be used by a boot- readahead and preload implementation to automatically make files hot used during booting or for preloading when I start an application.

On the other side hot relocation should be able to reduce writes to the SSD as good as possible, for example: Do not defragment files during autodefrag, it makes no sense. Also write data in bursts of erase block size etc.

And also important: What if the SSD dies due to wearing? Will it gracefully fall back to harddisk? What does "relocation" mean? Files (hot data) should only be cached in copy to SSD, and not moved there. It should be possible for btrfs to just drop a failing SSD from the filesystem without data loss because otherwise one should use two SSDs in raid-1 mode to get a safe cache storage.

Altogether I think that a spinning media btrfs raid can outperform a single SSD so hot relocation should probably be used to reduce head movements because this is where SSD really excels. So everything that involves heavy head movement should go to SSD first, then written back to harddisk. And I think there's a lot potential to optimize because a COW filesystem like btrfs naturally has a lot of head movement.

What do you think?

BTW: I have not tried the one or the other yet because I'm still deciding which way to go. Your patches are more welcome because I do not need to migrate my storage to bcache-provided block devices. OTOH the bcache implementation looks a lot more mature (with regard to performance and safety) at this point because it provides many of the above mentioned features - most importantly gracefully handling failing SSDs.

Regarding btrfs raid outperforms SSD: During boot my spinning media 3 device btrfs raid reads boot files with up to 600 MB/s (from LZ compressed fs), boot takes about 7 seconds until the display manager starts (which takes another 30 seconds but that's another story), and the system is pretty crowded with services I actually wouldn't need if I optimized for boot performance. But I think systemd's read-ahead implementation has a lot influence on this fast booting: It defragments and relocates boot files on btrfs during boot so the harddisks can sequentially read all this stuff. I think it also compresses boot files if compression is enabled because booting is IO bound, not CPU bound. Benchmarks showed that my btrfs raid could technically read up to 450 MB/s, so I think the 600 MB/s counts for decompressed data. A single SSD could not do that. For that same reason I created a small script to defragment and compress files used by the preload daemon. Without benchmarking it, this felt like another small performance boost. So I'm eager what could be next with some sort of SSD cache because the only problem left seems to be heavy head movement which slows down the system.

Quoted 133 lines · Zhi Yong Wu

HI,

What do you think if its design approach goes correctly? Do you
have any comments or better design idea for BTRFS hot relocation support? any comments are appreciated, thanks.

Quoted 122 lines · On Mon, May 6, 2013 at 4:53 PM, <[email protected]> wrote:

From: Zhi Yong Wu <[email protected]>

The patchset as RFC is sent out mainly to see if it goes in the
correct development direction.

The patchset is trying to introduce hot relocation support
for BTRFS. In hybrid storage environment, when the data in HDD disk get hot, it can be relocated to SSD disk by BTRFS hot relocation support automatically; also, if SSD disk ratio exceed its upper threshold, the data which get cold can be looked up and relocated to HDD disk to make more space in SSD disk at first, and then the data which get hot will be relocated to SSD disk automatically.

BTRFS hot relocation mainly reserve block space from SSD disk
at first, load the hot data to page cache from HDD, allocate block space from SSD disk, and finally write the data to SSD disk.

If you'd like to play with it, pls pull the patchset from
my git on github:
https://github.com/wuzhy/kernel.git hot_reloc

For how to use, please refer too the example below:

root@debian-i386:~# echo 0 > /sys/block/vdc/queue/rotational
^^^ Above command will hack /dev/vdc to be one SSD disk
root@debian-i386:~# echo 999999 > /proc/sys/fs/hot-age-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-update-interval
root@debian-i386:~# echo 10 > /proc/sys/fs/hot-reloc-interval
root@debian-i386:~# mkfs.btrfs -d single -m single -h /dev/vdb /dev/vdc
-f

WARNING! - Btrfs v0.20-rc1-254-gb0136aa-dirty IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using

[ 140.279011] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 1
[ transid 16 /dev/vdb 140.283650] device fsid
[ c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2 transid 16 /dev/vdc
[ 140.517089] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
[ transid 3 /dev/vdb 140.550759] device fsid
[ 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1 transid 3 /dev/vdb
[ 140.552473] device fsid c563a6dc-f192-41a9-9fe1-5a3aa01f5e4c devid 2
[ transid 16 /dev/vdc
adding device /dev/vdc id 2
[ 140.636215] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 2

[ transid 3 /dev/vdc
fs created label (null) on /dev/vdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 14.65GB Btrfs v0.20-rc1-254-gb0136aa-dirty

root@debian-i386:~# mount -o hot_move /dev/vdb /data2
[ 144.855471] device fsid 197d47a7-b9cd-46a8-9360-eb087b119424 devid 1
[ transid 6 /dev/vdb 144.870444] btrfs: disk space caching is enabled
[ 144.904214] VFS: Turning on hot data tracking
root@debian-i386:~# dd if=/dev/zero of=/data2/test1 bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 23.4948 s, 91.4 MB/s
root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.0G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=2.00GB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.19MB
Data_SSD: total=8.00MB, used=0.00
root@debian-i386:~# echo 108 > /proc/sys/fs/hot-reloc-threshold
^^^ Above command will start HOT RLEOCATE, because The data temperature
is currently 109 root@debian-i386:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 16G 13G 2.2G 86% /
tmpfs 4.8G 0 4.8G 0% /lib/init/rw
udev 10M 176K 9.9M 2% /dev
tmpfs 4.8G 0 4.8G 0% /dev/shm
/dev/vdb 15G 2.1G 13G 14% /data2
root@debian-i386:~# btrfs fi df /data2
Data: total=3.01GB, used=6.25MB
System: total=4.00MB, used=4.00KB
Metadata: total=8.00MB, used=2.26MB
Data_SSD: total=2.01GB, used=2.00GB
root@debian-i386:~#

Zhi Yong Wu (5):
vfs: add one list_head field
btrfs: add one new block group
btrfs: add one hot relocation kthread
procfs: add three proc interfaces
btrfs: add hot relocation support

 fs/btrfs/Makefile            |   3 +-
 fs/btrfs/ctree.h             |  26 +-
 fs/btrfs/extent-tree.c       | 107 +++++-
 fs/btrfs/extent_io.c         |  31 +-
 fs/btrfs/extent_io.h         |   4 +
 fs/btrfs/file.c              |  36 +-
 fs/btrfs/hot_relocate.c      | 802
 +++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/hot_relocate.h      |  48 +++
 fs/btrfs/inode-map.c         |  13 +-
 fs/btrfs/inode.c             |  92 ++++-
 fs/btrfs/ioctl.c             |  23 +-
 fs/btrfs/relocation.c        |  14 +-
 fs/btrfs/super.c             |  30 +-
 fs/btrfs/volumes.c           |  28 +-
 fs/hot_tracking.c            |   1 +
 include/linux/btrfs.h        |   4 +
 include/linux/hot_tracking.h |   1 +
 kernel/sysctl.c              |  22 ++
 18 files changed, 1234 insertions(+), 51 deletions(-)
 create mode 100644 fs/btrfs/hot_relocate.c
 create mode 100644 fs/btrfs/hot_relocate.h

--
1.7.11.7

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Regards,

Zhi Yong Wu