Skip to contentSkip to conversation

kheaders: Follow symlinks to source files.

PATCH In review Yifan Hong 2 participants 5 messages Apr 20, 2023 – May 3, 2023

lore
Share & tools
lore

Yifan Hong

author
#0 lore raw

PATCH v2 1/1 kheaders: Follow symlinks to source files.

When the kernel is built inside a sandbox container, a forest of symlinks to the source files may be created in the container. In this case, the generated kheaders.tar.xz should follow these symlinks to access the source files, instead of packing the symlinks themselves.

---
 kernel/gen_kheaders.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/gen_kheaders.sh +1 -1
index 1ef9a87511f5..70a75a9a7535 100755
@@ -87,7 +87,7 @@ find $cpio_dir -type f -print0 |
 # pre-sorted, as --sort=name might not be available.
 find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
     tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
-    --owner=0 --group=0 --numeric-owner --no-recursion \
+    --owner=0 --group=0 --numeric-owner --no-recursion --dereference \
     -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
 
 echo $headers_md5 > kernel/kheaders.md5
Signature · 1 line
2.40.0.634.g4ca3ef3211-goog
#1 lore raw
Quoted 10 lines · Yifan Hong

When the kernel is built inside a sandbox container, a forest of symlinks to the source files may be created in the container. In this case, the generated kheaders.tar.xz should follow these symlinks to access the source files, instead of packing the symlinks themselves.

Please teach me how to reproduce your problem.

I built the kernel inside a container,
but I do not see any difference.

We have some symbolic links in include/,
but those reference relative paths.
How building the kernel in a container
makes a difference?

masahiro@bc5c2c958b53:/tmp/foo$ tar Jxf kheaders_data.tar.xz
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l
./include/dt-bindings/clock/qcom,dispcc-sm8350.h
./include/dt-bindings/clock/qcom,dispcc-sm8150.h
./include/dt-bindings/input/linux-event-codes.h
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l | xargs ls -l
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8150.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8350.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 36 Apr 23 11:47
./include/dt-bindings/input/linux-event-codes.h ->
../../uapi/linux/input-event-codes.h
Quoted 20 lines
---
 kernel/gen_kheaders.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/gen_kheaders.sh +1 -2
index 1ef9a87511f5..70a75a9a7535 100755
@@ -87,7 +87,7 @@ find $cpio_dir -type f -print0 |
 # pre-sorted, as --sort=name might not be available.
 find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
     tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
-    --owner=0 --group=0 --numeric-owner --no-recursion \
+    --owner=0 --group=0 --numeric-owner --no-recursion --dereference \
     -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
 
 echo $headers_md5 > kernel/kheaders.md5
--

2.40.0.634.g4ca3ef3211-goog

Signature · 2 lines
Best Regards
Masahiro Yamada
author
#2 lore raw
Quoted 10 lines · Masahiro Yamada
Quoted 8 lines · Yifan Hong

When the kernel is built inside a sandbox container, a forest of symlinks to the source files may be created in the container. In this case, the generated kheaders.tar.xz should follow these symlinks to access the source files, instead of packing the symlinks themselves.

Quoted 6 lines
Quoted 2 lines

Please teach me how to reproduce your problem.

Thanks for your reply. When I say "container", I mean a forest of symlinks that point to the original source files. Here are the steps I do to mimic the sandbox that the container program creates.

On an x86 machine, I have a linux tree checkout at /mnt/sdb/linux, and an empty directory at /mnt/sdb/linux2. Then I **created this fake sandbox** in /mnt/sdb/linux2 by doing:

```
$ cd /mnt/sdb/linux
$ find . -type d -exec mkdir -p /mnt/sdb/linux2/{} \;
$ find . -type f -exec ln -sf /mnt/sdb/linux/{} /mnt/sdb/linux2/{} \;
```

Then, I cross compile an arm64 kernel with tinyconfig. Here are the steps I take:

```
export ARCH=arm64
export SUBARCH=arm
export CROSS_COMPILE=/home/elsk/toolchains/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-
# downloaded from https://toolchains.bootlin.com/
make tinyconfig; make menuconfig # then select CONFIG_SYSFS and CONFIG_IKHEADERS
make -j64
```

This will show that kheaders_data.tar.xz contains the symlinks, e.g. in my case:

```
$ tar tvf ./kernel/kheaders_data.tar.xz | head
[...]
lrwxrwxrwx 0/0               0 2023-04-24 11:19
./arch/arm64/include/asm/acenv.h ->
/mnt/sdb/linux/./arch/arm64/include/asm/acenv.h
```

However, the issue for me is that these symlinks are no longer useful under sysfs when I am booting the kernel on the target arm64 machine, because it is a different machine (arm64) than the machine (x86) where I compiled this kernel.

Quoted 8 lines

I built the kernel inside a container,
but I do not see any difference.

Quoted 4 lines

We have some symbolic links in include/,
but those reference relative paths.
How building the kernel in a container
makes a difference?

Please see above; the container contains a forest of symlinks pointing to the original source tree. Then make is executed in that container directory.

Quoted 58 lines

masahiro@bc5c2c958b53:/tmp/foo$ tar Jxf kheaders_data.tar.xz
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l
./include/dt-bindings/clock/qcom,dispcc-sm8350.h
./include/dt-bindings/clock/qcom,dispcc-sm8150.h
./include/dt-bindings/input/linux-event-codes.h
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l | xargs ls -l
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8150.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8350.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 36 Apr 23 11:47
./include/dt-bindings/input/linux-event-codes.h ->
../../uapi/linux/input-event-codes.h
Quoted 18 lines
---
 kernel/gen_kheaders.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/gen_kheaders.sh +1 -2
index 1ef9a87511f5..70a75a9a7535 100755
@@ -87,7 +87,7 @@ find $cpio_dir -type f -print0 |
 # pre-sorted, as --sort=name might not be available.
 find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
     tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
-    --owner=0 --group=0 --numeric-owner --no-recursion \
+    --owner=0 --group=0 --numeric-owner --no-recursion --dereference \
     -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
 
 echo $headers_md5 > kernel/kheaders.md5
--
Quoted 2 lines

2.40.0.634.g4ca3ef3211-goog

--
Best Regards
Masahiro Yamada

--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

author
#3 lore raw
Quoted 65 lines · Yifan Hong
Quoted 10 lines · Masahiro Yamada
Quoted 8 lines · Yifan Hong

When the kernel is built inside a sandbox container, a forest of symlinks to the source files may be created in the container. In this case, the generated kheaders.tar.xz should follow these symlinks to access the source files, instead of packing the symlinks themselves.

Quoted 6 lines
Quoted 2 lines

Please teach me how to reproduce your problem.

Thanks for your reply. When I say "container", I mean a forest of symlinks that point to the original source files. Here are the steps I do to mimic the sandbox that the container program creates.

On an x86 machine, I have a linux tree checkout at /mnt/sdb/linux, and an empty directory at /mnt/sdb/linux2. Then I **created this fake sandbox** in /mnt/sdb/linux2 by doing:

```
$ cd /mnt/sdb/linux
$ find . -type d -exec mkdir -p /mnt/sdb/linux2/{} \;
$ find . -type f -exec ln -sf /mnt/sdb/linux/{} /mnt/sdb/linux2/{} \;
```

Then, I cross compile an arm64 kernel with tinyconfig. Here are the steps I take:

```
export ARCH=arm64
export SUBARCH=arm
export CROSS_COMPILE=/home/elsk/toolchains/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-
# downloaded from https://toolchains.bootlin.com/
make tinyconfig; make menuconfig # then select CONFIG_SYSFS and CONFIG_IKHEADERS
make -j64
```

This will show that kheaders_data.tar.xz contains the symlinks, e.g. in my case:

```
$ tar tvf ./kernel/kheaders_data.tar.xz | head
[...]
lrwxrwxrwx 0/0               0 2023-04-24 11:19
./arch/arm64/include/asm/acenv.h ->
/mnt/sdb/linux/./arch/arm64/include/asm/acenv.h
```

However, the issue for me is that these symlinks are no longer useful under sysfs when I am booting the kernel on the target arm64 machine, because it is a different machine (arm64) than the machine (x86) where I compiled this kernel.

Quoted 8 lines

I built the kernel inside a container,
but I do not see any difference.

Quoted 67 lines
Quoted 4 lines

We have some symbolic links in include/,
but those reference relative paths.
How building the kernel in a container
makes a difference?

Please see above; the container contains a forest of symlinks pointing to the original source tree. Then make is executed in that container directory.

Quoted 58 lines

masahiro@bc5c2c958b53:/tmp/foo$ tar Jxf kheaders_data.tar.xz
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l
./include/dt-bindings/clock/qcom,dispcc-sm8350.h
./include/dt-bindings/clock/qcom,dispcc-sm8150.h
./include/dt-bindings/input/linux-event-codes.h
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l | xargs ls -l
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8150.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8350.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 36 Apr 23 11:47
./include/dt-bindings/input/linux-event-codes.h ->
../../uapi/linux/input-event-codes.h
Quoted 18 lines
---
 kernel/gen_kheaders.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/gen_kheaders.sh +1 -2
index 1ef9a87511f5..70a75a9a7535 100755
@@ -87,7 +87,7 @@ find $cpio_dir -type f -print0 |
 # pre-sorted, as --sort=name might not be available.
 find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
     tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
-    --owner=0 --group=0 --numeric-owner --no-recursion \
+    --owner=0 --group=0 --numeric-owner --no-recursion --dereference \
     -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
 
 echo $headers_md5 > kernel/kheaders.md5
--
Quoted 2 lines

2.40.0.634.g4ca3ef3211-goog

--
Best Regards
Masahiro Yamada

--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

Gently nagging. Is there anything I need to fix or do to improve this patch so it may be accepted? Thank you!

#4 lore raw
Quoted 131 lines · Yifan Hong
Quoted 129 lines · Yifan Hong
Quoted 10 lines · Masahiro Yamada
Quoted 8 lines · Yifan Hong

When the kernel is built inside a sandbox container, a forest of symlinks to the source files may be created in the container. In this case, the generated kheaders.tar.xz should follow these symlinks to access the source files, instead of packing the symlinks themselves.

Quoted 6 lines
Quoted 2 lines

Please teach me how to reproduce your problem.

Thanks for your reply. When I say "container", I mean a forest of symlinks that point to the original source files. Here are the steps I do to mimic the sandbox that the container program creates.

On an x86 machine, I have a linux tree checkout at /mnt/sdb/linux, and an empty directory at /mnt/sdb/linux2. Then I **created this fake sandbox** in /mnt/sdb/linux2 by doing:

```
$ cd /mnt/sdb/linux
$ find . -type d -exec mkdir -p /mnt/sdb/linux2/{} \;
$ find . -type f -exec ln -sf /mnt/sdb/linux/{} /mnt/sdb/linux2/{} \;
```

Then, I cross compile an arm64 kernel with tinyconfig. Here are the steps I take:

```
export ARCH=arm64
export SUBARCH=arm
export CROSS_COMPILE=/home/elsk/toolchains/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-
# downloaded from https://toolchains.bootlin.com/
make tinyconfig; make menuconfig # then select CONFIG_SYSFS and CONFIG_IKHEADERS
make -j64
```

This will show that kheaders_data.tar.xz contains the symlinks, e.g. in my case:

```
$ tar tvf ./kernel/kheaders_data.tar.xz | head
[...]
lrwxrwxrwx 0/0               0 2023-04-24 11:19
./arch/arm64/include/asm/acenv.h ->
/mnt/sdb/linux/./arch/arm64/include/asm/acenv.h
```

However, the issue for me is that these symlinks are no longer useful under sysfs when I am booting the kernel on the target arm64 machine, because it is a different machine (arm64) than the machine (x86) where I compiled this kernel.

Quoted 8 lines

I built the kernel inside a container,
but I do not see any difference.

Quoted 4 lines

We have some symbolic links in include/,
but those reference relative paths.
How building the kernel in a container
makes a difference?

Please see above; the container contains a forest of symlinks pointing to the original source tree. Then make is executed in that container directory.

Quoted 55 lines

masahiro@bc5c2c958b53:/tmp/foo$ tar Jxf kheaders_data.tar.xz
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l
./include/dt-bindings/clock/qcom,dispcc-sm8350.h
./include/dt-bindings/clock/qcom,dispcc-sm8150.h
./include/dt-bindings/input/linux-event-codes.h
masahiro@bc5c2c958b53:/tmp/foo$ find . -type l | xargs ls -l
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8150.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 20 Apr 23 11:47
./include/dt-bindings/clock/qcom,dispcc-sm8350.h ->
qcom,dispcc-sm8250.h
lrwxrwxrwx 1 masahiro masahiro 36 Apr 23 11:47
./include/dt-bindings/input/linux-event-codes.h ->
../../uapi/linux/input-event-codes.h
Quoted 18 lines
---
 kernel/gen_kheaders.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/gen_kheaders.sh +1 -2
index 1ef9a87511f5..70a75a9a7535 100755
@@ -87,7 +87,7 @@ find $cpio_dir -type f -print0 |
 # pre-sorted, as --sort=name might not be available.
 find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
     tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
-    --owner=0 --group=0 --numeric-owner --no-recursion \
+    --owner=0 --group=0 --numeric-owner --no-recursion --dereference \
     -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
 
 echo $headers_md5 > kernel/kheaders.md5
--
Quoted 2 lines

2.40.0.634.g4ca3ef3211-goog

--
Best Regards
Masahiro Yamada

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

Gently nagging. Is there anything I need to fix or do to improve this patch so it may be accepted? Thank you!

I guess it is better to not apply this.

Presumably, the fix will not end with this single patch.

Signature · 2 lines
Best Regards
Masahiro Yamada