Skip to contentSkip to conversation

fs: improve codegen in link_path_walk()

PATCH Applied Mateusz Guzik 3 participants 3 messages Apr 12, 2025 – Apr 14, 2025 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs: improve codegen in link_path_walk()

Looking at the asm produced by gcc 13.3 for x86-64:
1. may_lookup() usage was not optimized for succeeding, despite the routine being inlined and rightfully starting with likely(!err)
2. the compiler assumed the path will have an indefinite amount of slashes to skip, after which the result will be an empty name

As such:
1. predict may_lookup() succeeding
2. check for one slash, no explicit predicts. do roll forward with skipping more slashes while predicting there is only one
3. predict the path to find was not a mere slash

This also has a side effect of shrinking the file:
add/remove: 1/1 grow/shrink: 0/3 up/down: 934/-1012 (-78)

Function                                     old     new   delta
link_path_walk                                 -     934    +934
path_parentat                                138     112     -26

path_openat 4864 4823 -41
path_lookupat 418 374 -44
link_path_walk.part.constprop 901 - -901
Total: Before=46639, After=46561, chg -0.17%

---

I'm looking at skipping perm checks with an "everybody can MAY_EXEC and there are no acls" bit for opflags. This crapper is a side effect of straighetning out the code before I get there.

 fs/namei.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
fs/namei.c +7 -4
index 360a86ca1f02..40a636bbfa0c 100644
@@ -2424,9 +2424,12 @@ static int link_path_walk(const char *name, struct nameidata *nd)
 	nd->flags |= LOOKUP_PARENT;
 	if (IS_ERR(name))
 		return PTR_ERR(name);
-	while (*name=='/')
-		name++;
-	if (!*name) {
+	if (*name == '/') {
+		do {
+			name++;
+		} while (unlikely(*name == '/'));
+	}
+	if (unlikely(!*name)) {
 		nd->dir_mode = 0; // short-circuit the 'hardening' idiocy
 		return 0;
 	}
@@ -2439,7 +2442,7 @@ static int link_path_walk(const char *name, struct nameidata *nd)
 
 		idmap = mnt_idmap(nd->path.mnt);
 		err = may_lookup(idmap, nd);
-		if (err)
+		if (unlikely(err))
 			return err;
 
 		nd->last.name = name;
Signature · 1 line
2.43.0
maintainer Applied
#1 lore raw
Quoted 6 lines · Mateusz Guzik

Looking at the asm produced by gcc 13.3 for x86-64:
1. may_lookup() usage was not optimized for succeeding, despite the routine being inlined and rightfully starting with likely(!err)
2. the compiler assumed the path will have an indefinite amount of slashes to skip, after which the result will be an empty name

Quoted 7 lines

As such:
1. predict may_lookup() succeeding
2. check for one slash, no explicit predicts. do roll forward with skipping more slashes while predicting there is only one
3. predict the path to find was not a mere slash

[...]

Applied to the vfs-6.16.misc branch of the vfs/vfs.git tree. Patches in the vfs-6.16.misc branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs-6.16.misc

[1/1] fs: improve codegen in link_path_walk()

      https://git.kernel.org/vfs/vfs/c/80cf41f567f5
reviewer Reviewed-by on v1 1/1
#2 lore raw
In reply to Mateusz Guzik
Quoted 17 lines · Mateusz Guzik

Looking at the asm produced by gcc 13.3 for x86-64:
1. may_lookup() usage was not optimized for succeeding, despite the routine being inlined and rightfully starting with likely(!err)
2. the compiler assumed the path will have an indefinite amount of slashes to skip, after which the result will be an empty name

As such:
1. predict may_lookup() succeeding
2. check for one slash, no explicit predicts. do roll forward with skipping more slashes while predicting there is only one
3. predict the path to find was not a mere slash

This also has a side effect of shrinking the file:
add/remove: 1/1 grow/shrink: 0/3 up/down: 934/-1012 (-78)

Function                                     old     new   delta
link_path_walk                                 -     934    +934
path_parentat                                138     112     -26
Quoted 6 lines

path_openat 4864 4823 -41
path_lookupat 418 374 -44
link_path_walk.part.constprop 901 - -901
Total: Before=46639, After=46561, chg -0.17%

Looks sensible. Feel free to add:

Honza

Quoted 41 lines

---

I'm looking at skipping perm checks with an "everybody can MAY_EXEC and there are no acls" bit for opflags. This crapper is a side effect of straighetning out the code before I get there.

 fs/namei.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
fs/namei.c +7 -4
index 360a86ca1f02..40a636bbfa0c 100644
@@ -2424,9 +2424,12 @@ static int link_path_walk(const char *name, struct nameidata *nd)
 	nd->flags |= LOOKUP_PARENT;
 	if (IS_ERR(name))
 		return PTR_ERR(name);
-	while (*name=='/')
-		name++;
-	if (!*name) {
+	if (*name == '/') {
+		do {
+			name++;
+		} while (unlikely(*name == '/'));
+	}
+	if (unlikely(!*name)) {
 		nd->dir_mode = 0; // short-circuit the 'hardening' idiocy
 		return 0;
 	}
@@ -2439,7 +2442,7 @@ static int link_path_walk(const char *name, struct nameidata *nd)
 
 		idmap = mnt_idmap(nd->path.mnt);
 		err = may_lookup(idmap, nd);
-		if (err)
+		if (unlikely(err))
 			return err;
 
 		nd->last.name = name;
Signature · 1 line
2.43.0
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR