Balance filters: stripes, enhanced limit and usage

A few more enhancements, I'd like to see all changes to the balance filters merged into one major release. Thanks.

Changelog (v3):
* I've noticed that we can enhance the 'usage' filter the same way, so do it to be consistent with the rest
* the flags for all the new filters have been renamed and look more consistent as well:
BTRFS_BALANCE_ARGS_LIMIT_RANGE
BTRFS_BALANCE_ARGS_STRIPES_RANGE
BTRFS_BALANCE_ARGS_USAGE_RANGE
* Not: the flags mask is not yet updated, this will follow, also the progs need to be updated accordingly

Changelog (v2):
* the limit_min logic was wrong and did not work, all other filters are locally applied to the processed chunk, but for the limit_min we need to count all filters first and then decide if we want to rebalance or not

Update to balance filters, intended for 4.4:

* new 'stripes=<range>' - process only stripes that cross given number of devices, specified by a range

* updated 'limit=<range>' - previously a single number was accepted, it's a range so now we can specify a minimum number of chunks to process

There will be more documentation about the use in the btrfs-progs patches, the kernel side just applies the ranges. The update to 'limit' is backward compatible, reuses the previous struct member.

Need the v2 btrfs-progs patchset.

Note to merging: there's a missing patch that actuall adds both filter flags to the mask, however this patch[1] will supposedly be merged to 4.3-rc6. Current integration is based on 4.3-rc5 so I can't send a branch containing that without pulling unrelated patches. I'll send the patch independently so you can decide how to apply it once your first 4.4 pull request gets merged.

[1] btrfs: check unsupported filters in balance arguments

    (8eb934591f8bf584969454a658f629cd06e59f3a)

----------------------------------------------------------------
The following changes since commit 049e6dde7e57f0054fdc49102e7ef4830c698b46:

Linux 4.3-rc4 (2015-10-04 16:57:17 +0100)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git dev/balance-filters

for you to fetch changes up to c0b33bea07c4e2ae6d4d9ed56b968e7c09b933c4:

btrfs: extend balance filter usage to take minimum and maximum (2015-10-20 18:58:49 +0200)

----------------------------------------------------------------
David Sterba (2):
btrfs: extend balance filter limit to take minimum and maximum
btrfs: extend balance filter usage to take minimum and maximum

Gabríel Arthúr Pétursson (1):
btrfs: add balance filter for stripes

 fs/btrfs/ctree.h           |  37 +++++++++++++---
 fs/btrfs/volumes.c         | 102 ++++++++++++++++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.h         |   3 ++
 include/uapi/linux/btrfs.h |  31 ++++++++++++--
 4 files changed, 164 insertions(+), 9 deletions(-)

The 'limit' filter is underdesigned, it should have been a range for [min,max], with some relaxed semantics when one of the bounds is missing. Besides that, using a full u64 for a single value is a waste of bytes.

Let's fix both by extending the use of the u64 bytes for the [min,max] range. This can be done in a backward compatible way, the range will be interpreted only if the appropriate flag is set (BTRFS_BALANCE_ARGS_LIMIT_RANGE).

---
 fs/btrfs/ctree.h           | 14 ++++++++++++--
 fs/btrfs/volumes.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h         |  1 +
 include/uapi/linux/btrfs.h | 13 ++++++++++++-
 4 files changed, 67 insertions(+), 3 deletions(-)

index 938efe33be80..852c738d6954 100644
@@ -846,8 +846,18 @@ struct btrfs_disk_balance_args {
 	/* BTRFS_BALANCE_ARGS_* */
 	__le64 flags;
 
-	/* BTRFS_BALANCE_ARGS_LIMIT value */
-	__le64 limit;
+	/*
+	 * BTRFS_BALANCE_ARGS_LIMIT with value 'limit'
+	 * BTRFS_BALANCE_ARGS_LIMIT_RANGE - the extend version can use minimum
+	 * and maximum
+	 */
+	union {
+		__le64 limit;
+		struct {
+			__le32 limit_min;
+			__le32 limit_max;
+		};
+	};
 
 	__le64 unused[7];
 } __attribute__ ((__packed__));

index 2ca784a14e84..c34578ee49f9 100644
@@ -375,6 +375,7 @@ struct map_lookup {
 #define BTRFS_BALANCE_ARGS_DRANGE	(1ULL << 3)
 #define BTRFS_BALANCE_ARGS_VRANGE	(1ULL << 4)
 #define BTRFS_BALANCE_ARGS_LIMIT	(1ULL << 5)
+#define BTRFS_BALANCE_ARGS_LIMIT_RANGE	(1ULL << 6)
 
 /*
  * Profile changing flags.  When SOFT is set we won't relocate chunk if

index b6dec05c7196..11f13108b78b 100644
@@ -217,7 +217,18 @@ struct btrfs_balance_args {
 
 	__u64 flags;
 
-	__u64 limit;		/* limit number of processed chunks */
+	/*
+	 * BTRFS_BALANCE_ARGS_LIMIT with value 'limit'
+	 * BTRFS_BALANCE_ARGS_LIMIT_RANGE - the extend version can use minimum
+	 * and maximum
+	 */
+	union {
+		__u64 limit;		/* limit number of processed chunks */
+		struct {
+			__u32 limit_min;
+			__u32 limit_max;
+		};
+	};
 	__u64 unused[7];
 } __attribute__ ((__packed__));
 

2.6.2

From: Gabríel Arthúr Pétursson <[email protected]>

Balance block groups which have the given number of stripes, defined by a range min..max. This is useful to selectively rebalance only chunks that do not span enough devices, applies to RAID0/10/5/6.

---
 fs/btrfs/ctree.h           |  9 ++++++++-
 fs/btrfs/volumes.c         | 19 +++++++++++++++++++
 fs/btrfs/volumes.h         |  1 +
 include/uapi/linux/btrfs.h | 10 +++++++++-
 4 files changed, 37 insertions(+), 2 deletions(-)

index 852c738d6954..76055a8317e9 100644
@@ -859,7 +859,14 @@ struct btrfs_disk_balance_args {
 		};
 	};
 
-	__le64 unused[7];
+	/*
+	 * Process chunks that cross stripes_min..stripes_max devices,
+	 * BTRFS_BALANCE_ARGS_STRIPES_RANGE
+	 */
+	__le32 stripes_min;
+	__le32 stripes_max;
+
+	__le64 unused[6];
 } __attribute__ ((__packed__));
 
 /*

index b90a79c2bcf9..8d88a8737c79 100644
@@ -3170,6 +3170,19 @@ static int chunk_vrange_filter(struct extent_buffer *leaf,
 	return 1;
 }
 
+static int chunk_stripes_range_filter(struct extent_buffer *leaf,
+			       struct btrfs_chunk *chunk,
+			       struct btrfs_balance_args *bargs)
+{
+	int num_stripes = btrfs_chunk_num_stripes(leaf, chunk);
+
+	if (bargs->stripes_min <= num_stripes
+			&& num_stripes <= bargs->stripes_max)
+		return 0;
+
+	return 1;
+}
+
 static int chunk_soft_convert_filter(u64 chunk_type,
 				     struct btrfs_balance_args *bargs)
 {

@@ -3236,6 +3249,12 @@ static int should_balance_chunk(struct btrfs_root *root,
 		return 0;
 	}
 
+	/* stripes filter */
+	if ((bargs->flags & BTRFS_BALANCE_ARGS_STRIPES_RANGE) &&
+	    chunk_stripes_range_filter(leaf, chunk, bargs)) {
+		return 0;
+	}
+
 	/* soft profile changing mode */
 	if ((bargs->flags & BTRFS_BALANCE_ARGS_SOFT) &&
 	    chunk_soft_convert_filter(chunk_type, bargs)) {

index c34578ee49f9..1d823ad3b914 100644
@@ -376,6 +376,7 @@ struct map_lookup {
 #define BTRFS_BALANCE_ARGS_VRANGE	(1ULL << 4)
 #define BTRFS_BALANCE_ARGS_LIMIT	(1ULL << 5)
 #define BTRFS_BALANCE_ARGS_LIMIT_RANGE	(1ULL << 6)
+#define BTRFS_BALANCE_ARGS_STRIPES_RANGE (1ULL << 7)
 
 /*
  * Profile changing flags.  When SOFT is set we won't relocate chunk if

index 11f13108b78b..17a94704b183 100644
@@ -229,7 +229,15 @@ struct btrfs_balance_args {
 			__u32 limit_max;
 		};
 	};
-	__u64 unused[7];
+
+	/*
+	 * Process chunks that cross stripes_min..stripes_max devices,
+	 * BTRFS_BALANCE_ARGS_STRIPES_RANGE
+	 */
+	__le32 stripes_min;
+	__le32 stripes_max;
+
+	__u64 unused[6];
 } __attribute__ ((__packed__));
 
 /* report balance progress to userspace */

2.6.2

This sounds like an awesome improvement to re-balance a filesystem with newly added devices which don't have any data chunks yet. So in theory for RAID6 I would simple need to execute

btrfs balance start -dstripes=0..$devicecount-1 /filesystem

to have btrfs balance those chunks which are not spread across all devices yet?

Regards

Christian

Exactly.

Similar to the 'limit' filter, we can enhance the 'usage' filter to accept a range. The change is backward compatible, the range is applied only in connection with the BTRFS_BALANCE_ARGS_USAGE_RANGE flag.

We don't have a usecase yet, the current syntax has been sufficient. The enhancement should provide parity with other range-like filters.

---
 fs/btrfs/ctree.h           | 14 ++++++++++++--
 fs/btrfs/volumes.c         | 41 ++++++++++++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.h         |  1 +
 include/uapi/linux/btrfs.h |  8 +++++++-
 4 files changed, 60 insertions(+), 4 deletions(-)

index 76055a8317e9..31e5a3ed0a5c 100644
@@ -823,8 +823,18 @@ struct btrfs_disk_balance_args {
 	 */
 	__le64 profiles;
 
-	/* usage filter */
-	__le64 usage;
+	/*
+	 * usage filter
+	 * BTRFS_BALANCE_ARGS_USAGE with a single value means '0..N'
+	 * BTRFS_BALANCE_ARGS_USAGE_RANGE - range syntax, min..max
+	 */
+	union {
+		__le64 usage;
+		struct {
+			__le32 usage_min;
+			__le32 usage_max;
+		};
+	};
 
 	/* devid filter */
 	__le64 devid;

index 1d823ad3b914..d2be0872ce2f 100644
@@ -377,6 +377,7 @@ struct map_lookup {
 #define BTRFS_BALANCE_ARGS_LIMIT	(1ULL << 5)
 #define BTRFS_BALANCE_ARGS_LIMIT_RANGE	(1ULL << 6)
 #define BTRFS_BALANCE_ARGS_STRIPES_RANGE (1ULL << 7)
+#define BTRFS_BALANCE_ARGS_USAGE_RANGE	(1ULL << 8)
 
 /*
  * Profile changing flags.  When SOFT is set we won't relocate chunk if

index 17a94704b183..dea893199257 100644
@@ -206,7 +206,13 @@ struct btrfs_ioctl_feature_flags {
  */
 struct btrfs_balance_args {
 	__u64 profiles;
-	__u64 usage;
+	union {
+		__le64 usage;
+		struct {
+			__le32 usage_min;
+			__le32 usage_max;
+		};
+	};
 	__u64 devid;
 	__u64 pstart;
 	__u64 pend;

2.6.2