Skip to contentSkip to conversation

progs integration branch moved to master (new default leafsize)

PATCH Chris Mason 4 participants 10 messages Nov 8, 2013 – Nov 23, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi everyone,

This patch is now the tip of the master branch for btrfs-progs, which has been updated to include most of the backlogged progs patches. Please take a look and give it a shake. This was based on Dave's integration tree (many thanks Dave!) minus the patches for online dedup. I've pulled in the coverity fixes and a few others from the list as well.

The patch below switches our default mkfs leafsize up to 16K. This should be a better choice in almost every workload, but now is your chance to complain if it causes trouble.

--------------------------------
16KB is faster and leads to less metadata fragmentation in almost all workloads. It does slightly increase lock contention on the root nodes in some workloads, but that is best dealt with by adding more subvolumes (for now).

This uses 16KB or the page size, whichever is bigger. If you're doing a mixed block group mkfs, it uses the sectorsize instead.

Since the kernel refuses to mount a mixed block group FS where the metadata leaf size doesn't match the data sectorsize, this also adds a similar check during mkfs.

---
 mkfs.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
mkfs.c +18 -1
index bf8a831..cd0af9e 100644
@@ -46,6 +46,8 @@
 
 static u64 index_cnt = 2;
 
+#define DEFAULT_MKFS_LEAF_SIZE 16384
+
 struct directory_name_entry {
 	char *dir_name;
 	char *path;
@@ -1222,7 +1224,7 @@ int main(int ac, char **av)
 	u64 alloc_start = 0;
 	u64 metadata_profile = 0;
 	u64 data_profile = 0;
-	u32 leafsize = sysconf(_SC_PAGESIZE);
+	u32 leafsize = max_t(u32, sysconf(_SC_PAGESIZE), DEFAULT_MKFS_LEAF_SIZE);
 	u32 sectorsize = 4096;
 	u32 nodesize = leafsize;
 	u32 stripesize = 4096;
@@ -1232,6 +1234,7 @@ int main(int ac, char **av)
 	int ret;
 	int i;
 	int mixed = 0;
+	int leaf_forced = 0;
 	int data_profile_opt = 0;
 	int metadata_profile_opt = 0;
 	int discard = 1;
@@ -1269,6 +1272,7 @@ int main(int ac, char **av)
 			case 'n':
 				nodesize = parse_size(optarg);
 				leafsize = parse_size(optarg);
+				leaf_forced = 1;
 				break;
 			case 'L':
 				label = parse_label(optarg);
@@ -1386,8 +1390,21 @@ int main(int ac, char **av)
 				BTRFS_BLOCK_GROUP_RAID0 : 0; /* raid0 or single */
 		}
 	} else {
+		u32 best_leafsize = max_t(u32, sysconf(_SC_PAGESIZE), sectorsize);
 		metadata_profile = 0;
 		data_profile = 0;
+
+		if (!leaf_forced) {
+			leafsize = best_leafsize;
+			nodesize = best_leafsize;
+			if (check_leaf_or_node_size(leafsize, sectorsize))
+				exit(1);
+		}
+		if (leafsize != sectorsize) {
+			fprintf(stderr, "Error: mixed metadata/data block groups "
+				"require metadata blocksizes equal to the sectorsize\n");
+			exit(1);
+		}
 	}
 
 	ret = test_num_disk_vs_raid(metadata_profile, data_profile,
Signature · 1 line
1.8.2
#1 lore raw
Quoted 9 lines · Chris Mason

Hi everyone,

This patch is now the tip of the master branch for btrfs-progs, which has been updated to include most of the backlogged progs patches. Please take a look and give it a shake. This was based on Dave's integration tree (many thanks Dave!) minus the patches for online dedup. I've pulled in the coverity fixes and a few others from the list as well.

Quoted 3 lines

The patch below switches our default mkfs leafsize up to 16K. This should be a better choice in almost every workload, but now is your chance to complain if it causes trouble.

Thanks for that and nicely timely!

Compiling on Gentoo (3.11.5-gentoo, sys-fs/btrfs-progs-9999) gives:

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

* Please do not file a Gentoo bug and instead report the above QA
* issues directly to the upstream developers of this software.
* Homepage: https://btrfs.wiki.kernel.org

Quoted 5 lines

--------------------------------
16KB is faster and leads to less metadata fragmentation in almost all workloads. It does slightly increase lock contention on the root nodes in some workloads, but that is best dealt with by adding more subvolumes (for now).

Interesting and I was wondering about that. Good update.

Also, hopefully that is a little more friendly for SSDs where often you see improved performance for 8kByte or 16kByte (aligned) writes...

Testing in progress,

Regards,
Martin

Quoted 72 lines

This uses 16KB or the page size, whichever is bigger. If you're doing a mixed block group mkfs, it uses the sectorsize instead.

Since the kernel refuses to mount a mixed block group FS where the metadata leaf size doesn't match the data sectorsize, this also adds a similar check during mkfs.

---
 mkfs.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
mkfs.c +18 -1
index bf8a831..cd0af9e 100644
@@ -46,6 +46,8 @@
 
 static u64 index_cnt = 2;
 
+#define DEFAULT_MKFS_LEAF_SIZE 16384
+
 struct directory_name_entry {
 	char *dir_name;
 	char *path;
@@ -1222,7 +1224,7 @@ int main(int ac, char **av)
 	u64 alloc_start = 0;
 	u64 metadata_profile = 0;
 	u64 data_profile = 0;
-	u32 leafsize = sysconf(_SC_PAGESIZE);
+	u32 leafsize = max_t(u32, sysconf(_SC_PAGESIZE), DEFAULT_MKFS_LEAF_SIZE);
 	u32 sectorsize = 4096;
 	u32 nodesize = leafsize;
 	u32 stripesize = 4096;
@@ -1232,6 +1234,7 @@ int main(int ac, char **av)
 	int ret;
 	int i;
 	int mixed = 0;
+	int leaf_forced = 0;
 	int data_profile_opt = 0;
 	int metadata_profile_opt = 0;
 	int discard = 1;
@@ -1269,6 +1272,7 @@ int main(int ac, char **av)
 			case 'n':
 				nodesize = parse_size(optarg);
 				leafsize = parse_size(optarg);
+				leaf_forced = 1;
 				break;
 			case 'L':
 				label = parse_label(optarg);
@@ -1386,8 +1390,21 @@ int main(int ac, char **av)
 				BTRFS_BLOCK_GROUP_RAID0 : 0; /* raid0 or single */
 		}
 	} else {
+		u32 best_leafsize = max_t(u32, sysconf(_SC_PAGESIZE), sectorsize);
 		metadata_profile = 0;
 		data_profile = 0;
+
+		if (!leaf_forced) {
+			leafsize = best_leafsize;
+			nodesize = best_leafsize;
+			if (check_leaf_or_node_size(leafsize, sectorsize))
+				exit(1);
+		}
+		if (leafsize != sectorsize) {
+			fprintf(stderr, "Error: mixed metadata/data block groups "
+				"require metadata blocksizes equal to the sectorsize\n");
+			exit(1);
+		}
 	}
 
 	ret = test_num_disk_vs_raid(metadata_profile, data_profile,
#2 lore raw

Quoting Martin (2013-11-08 18:53:06)

Quoted 19 lines · Chris Mason
Quoted 9 lines

Hi everyone,

This patch is now the tip of the master branch for btrfs-progs, which has been updated to include most of the backlogged progs patches. Please take a look and give it a shake. This was based on Dave's integration tree (many thanks Dave!) minus the patches for online dedup. I've pulled in the coverity fixes and a few others from the list as well.

Quoted 3 lines

The patch below switches our default mkfs leafsize up to 16K. This should be a better choice in almost every workload, but now is your chance to complain if it causes trouble.

Thanks for that and nicely timely!

Compiling on Gentoo (3.11.5-gentoo, sys-fs/btrfs-progs-9999) gives:

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

I'm not seeing these warnings with the current master branch, could you please rerun?

-chris

#3 lore raw
Quoted 29 lines · Chris Mason

Quoting Martin (2013-11-08 18:53:06)

Quoted 19 lines · Chris Mason
Quoted 9 lines

Hi everyone,

This patch is now the tip of the master branch for btrfs-progs, which has been updated to include most of the backlogged progs patches. Please take a look and give it a shake. This was based on Dave's integration tree (many thanks Dave!) minus the patches for online dedup. I've pulled in the coverity fixes and a few others from the list as well.

Quoted 3 lines

The patch below switches our default mkfs leafsize up to 16K. This should be a better choice in almost every workload, but now is your chance to complain if it causes trouble.

Thanks for that and nicely timely!

Compiling on Gentoo (3.11.5-gentoo, sys-fs/btrfs-progs-9999) gives:

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

Quoted 4 lines

I'm not seeing these warnings with the current master branch, could you please rerun?

From just now:

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

* Please do not file a Gentoo bug and instead report the above QA
* issues directly to the upstream developers of this software.
* Homepage: https://btrfs.wiki.kernel.org

Quoted 1 line
Quoted 1 line
Quoted 1 line

Installing (1 of 1) sys-fs/btrfs-progs-9999

... Which is exactly the same.

This is for Gentoo for:

gcc: x86_64-pc-linux-gnu-4.7.3

# gcc --version
gcc (Gentoo 4.7.3-r1 p1.3, pie-0.5.5) 4.7.3

Kernel: 3.11.9-gentoo

# btrfs version
Btrfs v0.20-rc1-597-g5aff090

And the "-9999" pulls the code in from:

Quoted 1 line

From git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs

9f0c53f..5aff090 master -> master
GIT update -->
repository:
git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git
updating from commit: 9f0c53f574b242b0d5988db2972c8aac77ef35a9
to commit: 5aff090a3951e7d787b32bb5c49adfec65091385

 cmds-filesystem.c | 79
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 mkfs.c            | 18 +++++++++---------
 2 files changed, 88 insertions(+), 9 deletions(-)

branch: master
storage directory:
"/usr/portage/distfiles/egit-src/btrfs-progs.git"
checkout type: bare repository
Cloning into '/var/tmp/portage/sys-fs/btrfs-progs-9999/work/btrfs-progs-9999'... done.
Checking connectivity... done
Branch branch-master set up to track remote branch master from origin. Switched to a new branch 'branch-master'

Hope that helps,

Regards,
Martin

#4 lore raw

Quoting Martin (2013-11-22 04:03:41)

Quoted 36 lines · Chris Mason
Quoted 29 lines

Quoting Martin (2013-11-08 18:53:06)

Quoted 19 lines · Chris Mason
Quoted 9 lines

Hi everyone,

This patch is now the tip of the master branch for btrfs-progs, which has been updated to include most of the backlogged progs patches. Please take a look and give it a shake. This was based on Dave's integration tree (many thanks Dave!) minus the patches for online dedup. I've pulled in the coverity fixes and a few others from the list as well.

Quoted 3 lines

The patch below switches our default mkfs leafsize up to 16K. This should be a better choice in almost every workload, but now is your chance to complain if it causes trouble.

Thanks for that and nicely timely!

Compiling on Gentoo (3.11.5-gentoo, sys-fs/btrfs-progs-9999) gives:

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

Quoted 2 lines

I'm not seeing these warnings with the current master branch, could you please rerun?

From just now:

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

Does gentoo modify the optimizations from the Makefile? We actually have many strict-aliasing warnings, but I didn't think they came up until -O2.

At any rate, I'm adding -fno-strict-aliasing just to be sure.

-chris

#5 lore raw
Quoted 11 lines · Chris Mason

Quoting Martin (2013-11-22 04:03:41)

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

Quoted 3 lines

Does gentoo modify the optimizations from the Makefile? We actually have many strict-aliasing warnings, but I didn't think they came up until -O2.

For that system, I have -Os set in the Gentoo "make.conf".

Quoted 1 line

At any rate, I'm adding -fno-strict-aliasing just to be sure.

Good to catch to avoid unexpectedness,

Regards,
Martin

#6 lore raw

Quoting Martin (2013-11-22 14:50:17)

Quoted 16 lines · Chris Mason
Quoted 11 lines

Quoting Martin (2013-11-22 04:03:41)

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

Quoted 3 lines

Does gentoo modify the optimizations from the Makefile? We actually have many strict-aliasing warnings, but I didn't think they came up until -O2.

Quoted 6 lines

For that system, I have -Os set in the Gentoo "make.conf".

Quoted 1 line

At any rate, I'm adding -fno-strict-aliasing just to be sure.

Good to catch to avoid unexpectedness,

Ok, please try with the current master to make sure the options are being picked up properly. If you're overriding the -fno-strict-aliasing, please don't ;)

-chris

#7 lore raw
Quoted 20 lines · Chris Mason

Quoting Martin (2013-11-22 14:50:17)

Quoted 19 lines · Chris Mason
Quoted 11 lines

Quoting Martin (2013-11-22 04:03:41)

Quoted 8 lines

* QA Notice: Package triggers severe warnings which indicate that it
* may exhibit random runtime failures.
* disk-io.c:91:5: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1930:5: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]
* volumes.c:1931:6: warning: dereferencing type-punned pointer will
break strict-aliasing rules [-Wstrict-aliasing]

Quoted 3 lines

Does gentoo modify the optimizations from the Makefile? We actually have many strict-aliasing warnings, but I didn't think they came up until -O2.

For that system, I have -Os set in the Gentoo "make.conf".

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

At any rate, I'm adding -fno-strict-aliasing just to be sure.

Good to catch to avoid unexpectedness,

Ok, please try with the current master to make sure the options are being picked up properly. If you're overriding the -fno-strict-aliasing, please don't ;)

No changes my side for that system and...

btrfs-progs now compiles with no warnings given. That looks like a "fixed".

# emerge -vD btrfs-progs

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild R *] sys-fs/btrfs-progs-9999 0 kB

Total: 1 package (1 reinstall), Size of downloads: 0 kB

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Verifying ebuild manifests

Emerging (1 of 1) sys-fs/btrfs-progs-9999
Unpacking source...

GIT update -->
repository:
git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git
at the commit: 8116550e16628794b76051b6b8ea503055c08d6f
branch: master
storage directory:
"/usr/portage/distfiles/egit-src/btrfs-progs.git"
checkout type: bare repository
Cloning into '/var/tmp/portage/sys-fs/btrfs-progs-9999/work/btrfs-progs-9999'... done.
Checking connectivity... done
Branch branch-master set up to track remote branch master from origin. Switched to a new branch 'branch-master'

... And then a clean compile. No warnings.

Thanks,
Martin

#8 lore raw
In reply to Chris Mason

Chris Mason posted on Fri, 22 Nov 2013 08:40:38 -0500 as excerpted:

Quoted 5 lines

Does gentoo modify the optimizations from the Makefile? We actually have many strict-aliasing warnings, but I didn't think they came up until -O2.

At any rate, I'm adding -fno-strict-aliasing just to be sure.

FWIW, Gentoo lets users set their own CFLAGS/CXXFLAGS/LDFLAGS, etc.

Gentoo recommends -O2 at the global system level and I believe sets that as the default before user modification, as well. -Os is considered a viable option, but users setting -O3 or -Ofast or whatever are often considered ricers and can be told to try to reproduce bugs with -O2 set. And -O1 or lower is not recommended either, simply because it's not widely tested.

Individual package builds can and routinely do, however, override or filter specific flags where they've been demonstrated to be problematic, and in some cases, may add flags such as -fno-strict-aliasing where upstream code requires it.

But of course a (normally masked unless a user specifically unmasks it) -9999 version is gentoo's way of noting a live-from-upstream-vcs version, and that's the case here as well. btrfs-progs-9999 fetches and builds the live-git btrfs-progs direct from upstream git, and as such, it's subject to far less QA than ~arch (~x86, ~amd64, etc) aka testing packages, which in turn are less stable-tested than stable arch (x86, amd64, etc) packages. Basically, gentooers unmasking -9999 packages are expected to be able to deal with any breakage a live-vcs build may have and/or trigger.

And it's likely that -fno-strict-aliasing simply hasn't made it to the live build yet. But by the same token, I don't believe it'll override an upstream makefile's use of the flag, so now that you've added it...

... And indeed, after just rebuilding it here, with the commit (8116550) adding -fno-strict-aliasing as HEAD, I see no further type-punned pointer warnings. =:^)

I do however still see this:

cmds-restore.c: In function 'next_leaf':
cmds-restore.c:174:20: warning: array subscript is above array bounds [- Warray-bounds]

  slot = path->slots[level] + 1;
                    ^

(I am running a newer gcc than Martin, however:
gcc (Gentoo 4.8.1-r1 p1.2, pie-0.5.7) 4.8.1
)

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#9 lore raw
In reply to Chris Mason
Quoted 3 lines · Chris Mason

The patch below switches our default mkfs leafsize up to 16K. This should be a better choice in almost every workload, but now is your chance to complain if it causes trouble.

We should also turn the extended refs on by default now, it's been around since 3.7, I'm not expecting backward compatibility issues in general.

david