gen_initramfs_list.sh escaping problem or stale dependency file?

PATCH In review Florian Fainelli 3 participants 7 messages Sep 13, 2016 – Oct 27, 2016 Kbuild

lore

Share & tools

lore

Florian Fainelli

author

2016-09-13 00:12 UTC #0 lore raw

Hi,

I have a root filesystem embedding filenames that look like these:

/lib/data/<vid>:<pid>

these are essentially files that can be matched against an USB vendor/product id in an easy way.

Now, the fun part is that this is only a problem when doing the following (using OpenWrt/LEDE as a build system):

1:
- set CONFIG_INITRAMFS_SOURCE=""
- build kernel modules
- build my user-space tools
- build the kernel image
- reconfigure the kernel to now use an initramfs
- build the kernel w/ initramfs

and then back to step 1 with the kernel build, would I hit this error:

usr/Makefile:64: *** multiple target patterns. Stop.

which comes from usr/.initramfs_data.cpio.d containing these files without escaping:

deps_initramfs := ./scripts/gen_initramfs_list.sh \
/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb \
/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb/lib \
...

/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb/lib/network/wwan \
/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb/lib/network/wwan/19d2:0063 \

Which sorts of make sense here because the file name contains a ":" which is not escaped, so GNU Make tries to interpret it.

Now the part that does not quite make sense to me is why this file is even relevant here considering that the first thing we do is set CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.

Any clues what could be wrong here? I am happy to provide any build drops you may need to reproduce that.

Thanks!

Signature · 1 line

Florian

Michal Marek

2016-09-13 07:24 UTC #1 lore raw

Quoted 23 lines · Florian Fainelli

Hi,
I have a root filesystem embedding filenames that look like these:
/lib/data/<vid>:<pid>
these are essentially files that can be matched against an USB vendor/product id in an easy way.
Now, the fun part is that this is only a problem when doing the following (using OpenWrt/LEDE as a build system):
1:
- set CONFIG_INITRAMFS_SOURCE=""
- build kernel modules
- build my user-space tools
- build the kernel image
- reconfigure the kernel to now use an initramfs
- build the kernel w/ initramfs
and then back to step 1 with the kernel build, would I hit this error:
usr/Makefile:64: *** multiple target patterns. Stop.

[...]

Quoted 6 lines

Which sorts of make sense here because the file name contains a ":" which is not escaped, so GNU Make tries to interpret it.
Now the part that does not quite make sense to me is why this file is even relevant here considering that the first thing we do is set CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.

It is possible that we read usr/Makefile twice for some reason. But the real problem is the lack of escaping. Can you try the following (untested) patch?

scripts/gen_initramfs_list.sh +4 -1

index 17fa901418ae..5d3188e74101 100755
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/\([:%]\)/\\\1/g; s/$/ \\/'
 }
 
 # for each file print a line in following format

Thanks,
Michal

Florian Fainelli

author Tested-by

2016-09-13 17:28 UTC #2 lore raw

Quoted 28 lines · Michal Marek

Quoted 23 lines · Florian Fainelli
Hi,
I have a root filesystem embedding filenames that look like these:
/lib/data/<vid>:<pid>
these are essentially files that can be matched against an USB vendor/product id in an easy way.
Now, the fun part is that this is only a problem when doing the following (using OpenWrt/LEDE as a build system):
1:
- set CONFIG_INITRAMFS_SOURCE=""
- build kernel modules
- build my user-space tools
- build the kernel image
- reconfigure the kernel to now use an initramfs
- build the kernel w/ initramfs
and then back to step 1 with the kernel build, would I hit this error:
usr/Makefile:64: *** multiple target patterns. Stop.
[...]
Quoted 3 lines
Which sorts of make sense here because the file name contains a ":" which is not escaped, so GNU Make tries to interpret it.

Quoted 7 lines

Quoted 3 lines
Now the part that does not quite make sense to me is why this file is even relevant here considering that the first thing we do is set CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.
It is possible that we read usr/Makefile twice for some reason. But the real problem is the lack of escaping. Can you try the following (untested) patch?

This patch works for me:

Tested-by Florian Fainelli <[email protected]>

Kind of surprising that this has not showed up before, but maybe people don't really go through the same steps while re-configuring/re-building their kernels

Thanks Michal!

Quoted 22 lines

scripts/gen_initramfs_list.sh +4 -1

index 17fa901418ae..5d3188e74101 100755
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/\([:%]\)/\\\1/g; s/$/ \\/'
 }
 
 # for each file print a line in following format

Thanks,
Michal

Signature · 1 line

Florian

Florian Fainelli

author

2016-09-19 20:01 UTC #3 lore raw

In reply to Michal Marek

Quoted 28 lines · Michal Marek

Quoted 23 lines · Florian Fainelli
Hi,
I have a root filesystem embedding filenames that look like these:
/lib/data/<vid>:<pid>
these are essentially files that can be matched against an USB vendor/product id in an easy way.
Now, the fun part is that this is only a problem when doing the following (using OpenWrt/LEDE as a build system):
1:
- set CONFIG_INITRAMFS_SOURCE=""
- build kernel modules
- build my user-space tools
- build the kernel image
- reconfigure the kernel to now use an initramfs
- build the kernel w/ initramfs
and then back to step 1 with the kernel build, would I hit this error:
usr/Makefile:64: *** multiple target patterns. Stop.
[...]
Quoted 3 lines
Which sorts of make sense here because the file name contains a ":" which is not escaped, so GNU Make tries to interpret it.

Quoted 7 lines

Quoted 3 lines
Now the part that does not quite make sense to me is why this file is even relevant here considering that the first thing we do is set CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.
It is possible that we read usr/Makefile twice for some reason. But the real problem is the lack of escaping. Can you try the following (untested) patch?

Can you submit an official patch for this? Thanks a lot!

Quoted 22 lines

scripts/gen_initramfs_list.sh +4 -1

index 17fa901418ae..5d3188e74101 100755
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/\([:%]\)/\\\1/g; s/$/ \\/'
 }
 
 # for each file print a line in following format

Thanks,
Michal

Signature · 1 line

Florian

Michal Marek

2016-09-23 07:05 UTC #4 lore raw

Quoted 33 lines · Florian Fainelli

Quoted 32 lines · Michal Marek
Quoted 23 lines · Florian Fainelli
Hi,
I have a root filesystem embedding filenames that look like these:
/lib/data/<vid>:<pid>
these are essentially files that can be matched against an USB vendor/product id in an easy way.
Now, the fun part is that this is only a problem when doing the following (using OpenWrt/LEDE as a build system):
1:
- set CONFIG_INITRAMFS_SOURCE=""
- build kernel modules
- build my user-space tools
- build the kernel image
- reconfigure the kernel to now use an initramfs
- build the kernel w/ initramfs
and then back to step 1 with the kernel build, would I hit this error:
usr/Makefile:64: *** multiple target patterns. Stop.
[...]
Quoted 6 lines
Which sorts of make sense here because the file name contains a ":" which is not escaped, so GNU Make tries to interpret it.
Now the part that does not quite make sense to me is why this file is even relevant here considering that the first thing we do is set CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.

Quoted 5 lines

Quoted 3 lines
It is possible that we read usr/Makefile twice for some reason. But the real problem is the lack of escaping. Can you try the following (untested) patch?
Can you submit an official patch for this? Thanks a lot!

The % escape is wrong. I'm trying to fix it or drop this escape.

Michal

Michal Marek

2016-09-23 08:35 UTC #5 lore raw

PATCH 1/1 initramfs: Escape colons in depfile

Special characters are problematic in depfiles, but we can fix colons easily.

Reported-by Florian Fainelli <[email protected]>

Signed-off-by Michal Marek <[email protected]>

---
 scripts/gen_initramfs_list.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

scripts/gen_initramfs_list.sh +4 -1

index 17fa901418ae..0055b07b03b6 100755
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/:/\\:/g; s/$/ \\/'
 }
 
 # for each file print a line in following format

Signature · 1 line

2.6.6

John Crispin

2016-10-27 14:08 UTC #6 lore raw

Re: [LEDE-DEV] [PATCH] initramfs: Escape colons in depfile

Hi,

this seems to be a backport for the kernel tree. please resend it as a patch adding the actual patch that is being backported to the generic targets patch folder. this would need to be done for 3.18, 4.1 and 4.4

John

Quoted 26 lines · Michal Marek

Special characters are problematic in depfiles, but we can fix colons easily.
Reported-by Florian Fainelli <[email protected]>
Signed-off-by Michal Marek <[email protected]>
---
 scripts/gen_initramfs_list.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/gen_initramfs_list.sh +4 -1
index 17fa901418ae..0055b07b03b6 100755
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/:/\\:/g; s/$/ \\/'
 }
 
 # for each file print a line in following format