Skip to contentSkip to conversation

kbuild: fix package build error due to broken symlinks

PATCH In review Masahiro Yamada 2 participants 5 messages Mar 25, 2023 – Apr 2, 2023 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kbuild: fix package build error due to broken symlinks

'make deb-pkg' and 'make rpm-pkg' fail if a broken symlink exists in a dirty source tree. Handle symlinks properly, and also, keep the executable permission.

Fixes 05e96e96a315 ("kbuild: use git-archive for source package creation")

---

 scripts/package/gen-diff-patch | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
scripts/package/gen-diff-patch +27 -9
index f842ab50a780..23551de92e1b 100755
@@ -23,16 +23,34 @@ fi
 git -C ${srctree} status --porcelain --untracked-files=all |
 while read stat path
 do
-	if [ "${stat}" = '??' ]; then
-
-		if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
-			! head -n1 .tmp_diff | grep -q "Binary files"; then
-			{
-				echo "--- /dev/null"
-				echo "+++ linux/$path"
-				cat .tmp_diff | tail -n +3
-			} >> ${untracked_patch}
+	if [ "${stat}" != '??' ]; then
+		continue
+	fi
+
+	if [ -L "${path}" ]; then
+		{
+			echo "diff --git a/${path} b/${path}"
+			echo "new file mode 120000"
+			echo "--- /dev/null"
+			echo "+++ b/$path"
+			echo "@@ -0,0 +1 @@"
+			printf "+"; readlink ${path}
+			echo '\ No newline at end of file'
+		} >> ${untracked_patch}
+	elif ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
+	     ! head -n1 .tmp_diff | grep -q "Binary files"; then
+		if [ -x ${path} ]; then
+			mode=100755
+		else
+			mode=100644
 		fi
+		{
+			echo "diff --git a/${path} b/${path}"
+			echo "new file mode ${mode}"
+			echo "--- /dev/null"
+			echo "+++ b/$path"
+			cat .tmp_diff | tail -n +3
+		} >> ${untracked_patch}
 	fi
 done
 
Signature · 1 line
2.34.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 30 lines · Masahiro Yamada

'make deb-pkg' and 'make rpm-pkg' fail if a broken symlink exists in a dirty source tree. Handle symlinks properly, and also, keep the executable permission.

Fixes 05e96e96a315 ("kbuild: use git-archive for source package creation")

---

 scripts/package/gen-diff-patch | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
scripts/package/gen-diff-patch +2 -9
index f842ab50a780..23551de92e1b 100755
@@ -23,16 +23,34 @@ fi
 git -C ${srctree} status --porcelain --untracked-files=all |
 while read stat path
 do
-	if [ "${stat}" = '??' ]; then
-
-		if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
-			! head -n1 .tmp_diff | grep -q "Binary files"; then
-			{
-				echo "--- /dev/null"
-				echo "+++ linux/$path"
-				cat .tmp_diff | tail -n +3
-			} >> ${untracked_patch}
+	if [ "${stat}" != '??' ]; then
+		continue

scripts/package/gen-diff-patch fi · line 37

Quoted 10 lines
scripts/package/gen-diff-patch +10 -0
+	fi
+
+	if [ -L "${path}" ]; then
+		{
+			echo "diff --git a/${path} b/${path}"
+			echo "new file mode 120000"
+			echo "--- /dev/null"
+			echo "+++ b/$path"
+			echo "@@ -0,0 +1 @@"
+			printf "+"; readlink ${path}

Better quote "${path}"?

scripts/package/gen-diff-patch fi · line 39

Quoted 2 lines
scripts/package/gen-diff-patch +2 -0
+			echo '\ No newline at end of file'
+		} >> ${untracked_patch}

Here quoting should not be necessary as mkdebian and mkspec give only save filenames, but for consistency I'd quote ${untracked_patch} as well.

scripts/package/gen-diff-patch fi · line 42

Quoted 3 lines
scripts/package/gen-diff-patch +3 -0
+	elif ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
+	     ! head -n1 .tmp_diff | grep -q "Binary files"; then
+		if [ -x ${path} ]; then

Same here.

scripts/package/gen-diff-patch fi · line 53

Quoted 11 lines
scripts/package/gen-diff-patch +10 -0
+			mode=100755
+		else
+			mode=100644
 		fi
+		{
+			echo "diff --git a/${path} b/${path}"
+			echo "new file mode ${mode}"
+			echo "--- /dev/null"
+			echo "+++ b/$path"
+			cat .tmp_diff | tail -n +3
+		} >> ${untracked_patch}

And possibly here?

Quoted 5 lines
scripts/package/gen-diff-patch +0 -0
 	fi
 done
 
Signature · 1 line
2.34.1
#2 lore raw
Quoted 36 lines · Nicolas Schier
Quoted 34 lines · Masahiro Yamada

'make deb-pkg' and 'make rpm-pkg' fail if a broken symlink exists in a dirty source tree. Handle symlinks properly, and also, keep the executable permission.

Fixes 05e96e96a315 ("kbuild: use git-archive for source package creation")

---

 scripts/package/gen-diff-patch | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
scripts/package/gen-diff-patch +6 -9
index f842ab50a780..23551de92e1b 100755
@@ -23,16 +23,34 @@ fi
 git -C ${srctree} status --porcelain --untracked-files=all |
 while read stat path
 do
-     if [ "${stat}" = '??' ]; then
-
-             if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
-                     ! head -n1 .tmp_diff | grep -q "Binary files"; then
-                     {
-                             echo "--- /dev/null"
-                             echo "+++ linux/$path"
-                             cat .tmp_diff | tail -n +3
-                     } >> ${untracked_patch}
+     if [ "${stat}" != '??' ]; then
+             continue
+     fi
+
+     if [ -L "${path}" ]; then
+             {
Quoted 8 lines

scripts/package/gen-diff-patch fi · line 37

Quoted 6 lines
scripts/package/gen-diff-patch +6 -0
+                     echo "diff --git a/${path} b/${path}"
+                     echo "new file mode 120000"
+                     echo "--- /dev/null"
+                     echo "+++ b/$path"
+                     echo "@@ -0,0 +1 @@"
+                     printf "+"; readlink ${path}

Better quote "${path}"?

Thanks for the suggestion.

Quoting variables are correct in most cases. But, that is not enough to generate a valid patch when a file path contains spaces.

'git format-patch' produces a patch that
is accepted by GNU patch and also by dpkg-source.

I learned a trick from GIT source code.

If you are interested, what GIT does [1].

[1] https://github.com/git/git/commit/1a9eb3b9d50367bee8fe85022684d812816fe531

I will send v2 later, where I made some more efforts to fix several corner cases even if that is not perfect.

Signature · 2 lines
Best Regards
Masahiro Yamada
#3 lore raw
Quoted 58 lines · On Sat, Apr 01, 2023 at 11:55:44PM +0900 Masahiro Yamada wrote:
Quoted 36 lines · Nicolas Schier
Quoted 34 lines · Masahiro Yamada

'make deb-pkg' and 'make rpm-pkg' fail if a broken symlink exists in a dirty source tree. Handle symlinks properly, and also, keep the executable permission.

Fixes 05e96e96a315 ("kbuild: use git-archive for source package creation")

---

 scripts/package/gen-diff-patch | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
scripts/package/gen-diff-patch +6 -9
index f842ab50a780..23551de92e1b 100755
@@ -23,16 +23,34 @@ fi
 git -C ${srctree} status --porcelain --untracked-files=all |
 while read stat path
 do
-     if [ "${stat}" = '??' ]; then
-
-             if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
-                     ! head -n1 .tmp_diff | grep -q "Binary files"; then
-                     {
-                             echo "--- /dev/null"
-                             echo "+++ linux/$path"
-                             cat .tmp_diff | tail -n +3
-                     } >> ${untracked_patch}
+     if [ "${stat}" != '??' ]; then
+             continue
+     fi
+
+     if [ -L "${path}" ]; then
+             {
Quoted 8 lines

scripts/package/gen-diff-patch fi · line 37

Quoted 6 lines
scripts/package/gen-diff-patch +6 -0
+                     echo "diff --git a/${path} b/${path}"
+                     echo "new file mode 120000"
+                     echo "--- /dev/null"
+                     echo "+++ b/$path"
+                     echo "@@ -0,0 +1 @@"
+                     printf "+"; readlink ${path}

Better quote "${path}"?

Thanks for the suggestion.

Quoting variables are correct in most cases. But, that is not enough to generate a valid patch when a file path contains spaces.

'git format-patch' produces a patch that
is accepted by GNU patch and also by dpkg-source.

Quoted 7 lines

I learned a trick from GIT source code.

If you are interested, what GIT does [1].

[1] https://github.com/git/git/commit/1a9eb3b9d50367bee8fe85022684d812816fe531

thanks for the pointer, that is really interesting!

Kind regards,
Nicolas

Quoted 13 lines

I will send v2 later, where I made some more efforts to fix several corner cases even if that is not perfect.

Signature · 2 lines
Best Regards
Masahiro Yamada
#4 lore raw
Quoted 66 lines · Nicolas Schier
Quoted 64 lines · On Sat, Apr 01, 2023 at 11:55:44PM +0900 Masahiro Yamada wrote:
Quoted 36 lines · Nicolas Schier
Quoted 34 lines · Masahiro Yamada

'make deb-pkg' and 'make rpm-pkg' fail if a broken symlink exists in a dirty source tree. Handle symlinks properly, and also, keep the executable permission.

Fixes 05e96e96a315 ("kbuild: use git-archive for source package creation")

---

 scripts/package/gen-diff-patch | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
scripts/package/gen-diff-patch +6 -9
index f842ab50a780..23551de92e1b 100755
@@ -23,16 +23,34 @@ fi
 git -C ${srctree} status --porcelain --untracked-files=all |
 while read stat path
 do
-     if [ "${stat}" = '??' ]; then
-
-             if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
-                     ! head -n1 .tmp_diff | grep -q "Binary files"; then
-                     {
-                             echo "--- /dev/null"
-                             echo "+++ linux/$path"
-                             cat .tmp_diff | tail -n +3
-                     } >> ${untracked_patch}
+     if [ "${stat}" != '??' ]; then
+             continue
+     fi
+
+     if [ -L "${path}" ]; then
+             {
Quoted 8 lines

scripts/package/gen-diff-patch fi · line 37

Quoted 6 lines
scripts/package/gen-diff-patch +6 -0
+                     echo "diff --git a/${path} b/${path}"
+                     echo "new file mode 120000"
+                     echo "--- /dev/null"
+                     echo "+++ b/$path"
+                     echo "@@ -0,0 +1 @@"
+                     printf "+"; readlink ${path}

Better quote "${path}"?

Thanks for the suggestion.

Quoting variables are correct in most cases. But, that is not enough to generate a valid patch when a file path contains spaces.

'git format-patch' produces a patch that
is accepted by GNU patch and also by dpkg-source.

I learned a trick from GIT source code.

If you are interested, what GIT does [1].

Quoted 10 lines
Quoted 1 line

[1] https://github.com/git/git/commit/1a9eb3b9d50367bee8fe85022684d812816fe531

thanks for the pointer, that is really interesting!

Kind regards,
Nicolas

Quoted 2 lines

I will send v2 later, where I made some more efforts to fix several corner cases even if that is not perfect.

I wrote a patch (attached to this email) but I am still thinking if this is worth the complicated scripting...

Another approach would be to stop caring untracked files.

Signature · 2 lines
Best Regards
Masahiro Yamada

From 4b91ae3f482950209c356a98b79604394f16765c Mon Sep 17 00:00:00 2001
From: Masahiro Yamada <[email protected]>
Date: Sat, 1 Apr 2023 03:57:31 +0900
Subject: [PATCH] kbuild: make package builds more robust

Running 'make deb-pkg' in a dirty repository (that is, 'git diff' outputs something) has some problems.

[1] If a file path contains whitespaces, 'make deb-pkg' fails with an error message like follows:

diff: ./scripts/package/../../"a b": No such file or directory

    [snip]

dpkg-source: error: expected ^@@ at line 2 of diff ...

'git status --porcelain' double-quotes a file path that contains whitespaces. Remove the surrounding double-quotes, and add TAB to the '+++' line to make it a valid patch. This is the same as what GIT does.

See GIT commit 1a9eb3b9d503 ("git-diff/git-apply: make diff output a bit friendlier to GNU patch (part 2)") for reference.

I gave up other special characters such as TAB, new line. It would be possible to create a patch that GNU patch accepts by quoting the file paths (and 'git diff' does this), but dpkg-source fails anyway.

[2] The executable bit is lost.

[3] Symbolic links are not handled.

Rewrite the code to make it work more correctly.

---
 scripts/package/gen-diff-patch | 79 ++++++++++++++++++++++++++++------
 1 file changed, 66 insertions(+), 13 deletions(-)
scripts/package/gen-diff-patch +66 -13
index f842ab50a780..405b7d1925a5 100755
@@ -21,24 +21,77 @@ if [ ! -s "${diff_patch}" ]; then
 fi
 
 git -C ${srctree} status --porcelain --untracked-files=all |
-while read stat path
+while read -r stat path
 do
-	if [ "${stat}" = '??' ]; then
-
-		if ! diff -u /dev/null "${srctree}/${path}" > .tmp_diff &&
-			! head -n1 .tmp_diff | grep -q "Binary files"; then
-			{
-				echo "--- /dev/null"
-				echo "+++ linux/$path"
-				cat .tmp_diff | tail -n +3
-			} >> ${untracked_patch}
-		fi
+	if [ "${stat}" != '??' ]; then
+		continue
 	fi
-done
+
+	# path may be quoted if it contains whitespaces. Unquote it
+	path="${path#\"}"
+	path="${path%\"}"
+
+	case "${path}" in
+	*\\*)
+		# Give up escape sequences such as \n, \t.
+		# It is possible to create a patch by quoting file paths
+		# (and it is what GIT does) but dpkg-source fails with
+		# "... patches file with C-style encoded filename" error.
+		continue
+		;;
+	*\ *)
+		# GNU patch expects TAB on ---/+++ lines for paths with spaces.
+		tab="$(printf '\t')"
+		;;
+	*)
+		tab=
+		;;
+	esac
+
+	if [ ! -f "${path}" ]; then
+		continue
+	fi
+
+	if [ -L "${path}" ]; then
+		echo "diff --git a/${path} b/${path}"
+		echo "new file mode 120000"
+		echo "--- /dev/null"
+		echo "+++ b/${path}${tab}"
+		echo "@@ -0,0 +1 @@"
+		printf "+"; readlink "${path}"
+		echo '\ No newline at end of file'
+		continue
+	fi
+
+	if [ -x "${path}" ]; then
+		mode=100755
+	else
+		mode=100644
+	fi
+
+	if [ ! -s "${path}" ]; then
+		echo "diff --git a/${path} b/${path}"
+		echo "new file mode ${mode}"
+		continue
+	fi
+
+	if diff -u /dev/null "${srctree}/${path}" > .tmp_diff || [ "$?" != 1 ]; then
+		continue
+	fi
+
+	if [ "$(head -n6 .tmp_diff)" = Binary ] ; then
+		continue
+	fi
+
+	echo "diff --git a/${path} b/${path}"
+	echo "new file mode ${mode}"
+	echo "--- /dev/null"
+	echo "+++ b/${path}${tab}"
+	cat .tmp_diff | tail -n +3
+done > "${untracked_patch}"
 
 rm -f .tmp_diff
 
 if [ ! -s "${diff_patch}" ]; then
 	rm -f "${diff_patch}"
-	exit
 fi
Signature · 1 line
2.37.2