Skip to contentSkip to conversation

Restrict tar transform to regular files Without this

PATCH Superseded Timothy Pearson 2 participants 4 messages Apr 26, 2019 – Apr 27, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Restrict tar transform to regular files Without this

restriction, symlinks are rewritten and corrupted.

Example without the restriction:

Input tree (valid):
package-githash/file1
package-githash/link -> ./file1

Output tree (broken):
package-githash/file1
package-githash/link -> package-githash/file1

---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/git +1 -1
index 17ca04eb98..1acb795c81 100755
@@ -190,7 +190,7 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
 
 # Create GNU-format tarballs, since that's the format of the tarballs on
 # sources.buildroot.org and used in the *.hash files
-tar cf - --transform="s#^\./#${basename}/#" \
+tar cf - --transform="flags=r;s#^\./#${basename}/#" \
          --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
          -T "${output}.list.sorted" >"${output}.tar"
 gzip -6 -n <"${output}.tar" >"${output}"
#1 lore raw

PATCH 1/1 Restrict tar transform to regular files Without this

Timothy, All,

On 2019-04-26 11:49 -0500, Timothy Pearson spake thusly:

Quoted 1 line

restriction, symlinks are rewritten and corrupted.

Something is amiss in your commit log. It shoul;d be:

titile on a single line

Some text on
multiple lines.

Quoted 18 lines

Example without the restriction:

Input tree (valid):
package-githash/file1
package-githash/link -> ./file1

Output tree (broken):
package-githash/file1
package-githash/link -> package-githash/file1

---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/git +0 -0
index 17ca04eb98..1acb795c81 100755

support/download/git LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" · line 193

Quoted 5 lines
support/download/git +1 -1
@@ -190,7 +190,7 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
 
 # Create GNU-format tarballs, since that's the format of the tarballs on
 # sources.buildroot.org and used in the *.hash files
-tar cf - --transform="s#^\./#${basename}/#" \
+tar cf - --transform="flags=r;s#^\./#${basename}/#" \

OK, so this is a nice trick and all, but I can't find any reference to it in the man page. But I see it's in the git tree in doc/tar.texi...

Your proposal would exclude hardlinks too, which is not correct I think. In fact, we would need to use 'S' (upper-case 'S') here, because we want it to apply to anything but symlinks.

Also, the texi file states that "Default scope flags can also be changed using flags= statement in the transform expression." However, they are standard flags too, quoting:

Supported flags are:
g [...]
i [...]
x [...]
<number> [...]

[...]

In addition, several transformation scope flags are supported, that control to what files transformations apply. These are:
r Apply transformation to regular archive members.
R Do not apply transformation to regular archive members.
s Apply transformation to symbolic link targets.
S Do not apply transformation to symbolic link targets.
h Apply transformation to hard link targets.
H Do not apply transformation to hard link targets.

So I think we could simply write:

    --transform="s#^\./#${basename}/#S'

Regards,
Yann E. MORIN.

Quoted 7 lines
support/download/git +0 -0
          --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
          -T "${output}.list.sorted" >"${output}.tar"
 gzip -6 -n <"${output}.tar" >"${output}"

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

PATCH 1/1 Restrict tar transform to regular files Without this

Quoted 5 lines

Timothy, All,

On 2019-04-26 11:49 -0500, Timothy Pearson spake thusly:

Quoted 1 line

restriction, symlinks are rewritten and corrupted.

Quoted 8 lines

Something is amiss in your commit log. It shoul;d be:

titile on a single line

Some text on
multiple lines.

Looks like the signed off line wasn't part of the commit...sorry about that. I've re-uploaded it. The rest of the formatting was generated by git format-patch and should be correct?

Quoted 52 lines
Quoted 18 lines

Example without the restriction:

Input tree (valid):
package-githash/file1
package-githash/link -> ./file1

Output tree (broken):
package-githash/file1
package-githash/link -> package-githash/file1

---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/git +0 -0
index 17ca04eb98..1acb795c81 100755

support/download/git LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" · line 193

Quoted 5 lines
support/download/git +1 -1
@@ -190,7 +190,7 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
 
 # Create GNU-format tarballs, since that's the format of the tarballs on
 # sources.buildroot.org and used in the *.hash files
-tar cf - --transform="s#^\./#${basename}/#" \
+tar cf - --transform="flags=r;s#^\./#${basename}/#" \

OK, so this is a nice trick and all, but I can't find any reference to it in the man page. But I see it's in the git tree in doc/tar.texi...

Your proposal would exclude hardlinks too, which is not correct I think. In fact, we would need to use 'S' (upper-case 'S') here, because we want it to apply to anything but symlinks.

Also, the texi file states that "Default scope flags can also be changed using flags= statement in the transform expression." However, they are standard flags too, quoting:

Supported flags are:

       g [...]
       i [...]
       x [...]
       <number> [...]

[...]

In addition, several transformation scope flags are supported, that control to what files transformations apply. These are:

       r   Apply transformation to regular archive members.
       R   Do not apply transformation to regular archive members.
       s   Apply transformation to symbolic link targets.
       S   Do not apply transformation to symbolic link targets.
       h   Apply transformation to hard link targets.
       H   Do not apply transformation to hard link targets.
Quoted 3 lines

So I think we could simply write:

   --transform="s#^\./#${basename}/#S'

If that is functionally equivalent I'd be OK with it. Let me run a quick test against the original failing downloads first.

Quoted 18 lines

Regards,
Yann E. MORIN.

Quoted 7 lines
support/download/git +0 -0
          --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
          -T "${output}.list.sorted" >"${output}.tar"
 gzip -6 -n <"${output}.tar" >"${output}"

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

#3 lore raw

PATCH 1/1 Restrict tar transform to regular files Without this

Timothy, All,

[Please, wrap your messages at ~72 chars]

On 2019-04-27 03:49 -0500, Timothy Pearson spake thusly:

Quoted 7 lines
Quoted 7 lines

On 2019-04-26 11:49 -0500, Timothy Pearson spake thusly:

Quoted 1 line

restriction, symlinks are rewritten and corrupted.

Something is amiss in your commit log. It shoul;d be:

titile on a single line

Quoted 8 lines
Quoted 4 lines

Some text on
multiple lines.

Looks like the signed off line wasn't part of the commit...sorry about that. I've re-uploaded it. The rest of the formatting was generated by git format-patch and should be correct?

No it's not correct, see the subject of the mail which ends up in the middle of a sentence, and see how patchwork stored it (even your v2):

    https://patchwork.ozlabs.org/patch/1091900/

By the way, why did you resend a v2 that does not address my comment, which suggests putting the flag at the end of the expression?

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'