Makefile: Fix overlay overwriting everything

PATCH New Maxime Hadjinlian 4 participants 8 messages Feb 10, 2016 – Jun 2, 2016 Kbuild

lore

Share & tools

lore

Maxime Hadjinlian

author

2016-02-10 15:58 UTC #0 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

After bumping from a 2 years old Buildroot installation, I got the following messages from the kernel at boot:

Failed to execute /init (error -2)

It appears that the overlay mechanisms was causing my issues.

Description:
The overlay mechanisms works by rsync'ing files from a folder specified by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are symlinked into /usr.

Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content. Obviously, this causes a lot of troubles, starting with the kernel failing to start init (because of missing shared libraries).

Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Makefile +1 -1

index 71735b5..2f33ddf 100644
@@ -655,7 +655,7 @@ endif
 
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
-		rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+		rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
 			--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
 			$(d)/ $(TARGET_DIR)$(sep))

Signature · 1 line

2.7.0

Arnout Vandecappelle

2016-02-10 20:09 UTC #1 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

Quoted 20 lines · Maxime Hadjinlian

After bumping from a 2 years old Buildroot installation, I got the following messages from the kernel at boot:
Failed to execute /init (error -2)
It appears that the overlay mechanisms was causing my issues.
Description:
The overlay mechanisms works by rsync'ing files from a folder specified by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are symlinked into /usr.
Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content. Obviously, this causes a lot of troubles, starting with the kernel failing to start init (because of missing shared libraries).

Quoted 3 lines

Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.

I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some directories so it becomes useable.

That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.

Still, you _are_ breaking a valid use case with this patch.

What do the others think?

Regards,
Arnout

Quoted 20 lines

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Makefile +1 -1

index 71735b5..2f33ddf 100644
@@ -655,7 +655,7 @@ endif
 
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
-		rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+		rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
 			--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
 			$(d)/ $(TARGET_DIR)$(sep))

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Maxime Hadjinlian

author

2016-02-10 20:16 UTC #2 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

Hi Arnout, all

Quoted 21 lines · Arnout Vandecappelle

Quoted 20 lines · Maxime Hadjinlian
After bumping from a 2 years old Buildroot installation, I got the following messages from the kernel at boot:
Failed to execute /init (error -2)
It appears that the overlay mechanisms was causing my issues.
Description:
The overlay mechanisms works by rsync'ing files from a folder specified by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are symlinked into /usr.
Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content. Obviously, this causes a lot of troubles, starting with the kernel failing to start init (because of missing shared libraries).

Quoted 7 lines

Quoted 3 lines
Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.
I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some directories so it becomes useable.

Just a precision: it's not all the symlinks only the symlinks that points to a directory, like /var/spool

Quoted 5 lines

That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.
Still, you _are_ breaking a valid use case with this patch.

Another way to deal with this, is to test if in the overlay, there's a a folder that is a link is the default skeleton and in that case, treat it differently.
In the spur of the moment it seems too complicated. But maybe it's the right solution after all.

Quoted 35 lines

What do the others think?

Regards,
Arnout

Quoted 12 lines

Signed-off-by Maxime Hadjinlian <[email protected]>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +0 -0
index 71735b5..2f33ddf 100644

Makefile endif · line 661

Quoted 8 lines

Makefile +1 -1

@@ -655,7 +655,7 @@ endif
 
      @$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
              $(call MESSAGE,"Copying overlay $(d)"); \
-             rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+             rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
                      --chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
                      $(d)/ $(TARGET_DIR)$(sep))

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Arnout Vandecappelle

2016-02-10 20:35 UTC #3 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

Quoted 34 lines · Maxime Hadjinlian

Hi Arnout, all
Quoted 21 lines · Arnout Vandecappelle
Quoted 20 lines · Maxime Hadjinlian
After bumping from a 2 years old Buildroot installation, I got the following messages from the kernel at boot:
Failed to execute /init (error -2)
It appears that the overlay mechanisms was causing my issues.
Description:
The overlay mechanisms works by rsync'ing files from a folder specified by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are symlinked into /usr.
Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content. Obviously, this causes a lot of troubles, starting with the kernel failing to start init (because of missing shared libraries).
Quoted 7 lines
Quoted 3 lines
Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.
I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some directories so it becomes useable.
Just a precision: it's not all the symlinks only the symlinks that points to a directory, like /var/spool
Quoted 1 line

Quoted 7 lines

Quoted 4 lines
That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.
Still, you _are_ breaking a valid use case with this patch.
Another way to deal with this, is to test if in the overlay, there's a a folder that is a link is the default skeleton and in that case, treat it differently.

That is just dealing with the "accident" that the /bin and /lib symlinks are
created with an explicit ln instead of being present in the skeleton. It could just as well have been the reverse, that they are symlinks in the skeleton and mkdirs in the makefile. So I wouldn't rely on that.

Regards,
Arnout

Quoted 2 lines

In the spur of the moment it seems too complicated. But maybe it's the right solution after all.

[snip]

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Maxime Hadjinlian

author

2016-02-11 09:10 UTC #4 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

Quoted 39 lines · Arnout Vandecappelle

Quoted 38 lines · Maxime Hadjinlian
Hi Arnout, all
Quoted 21 lines · Arnout Vandecappelle
Quoted 20 lines · Maxime Hadjinlian
After bumping from a 2 years old Buildroot installation, I got the following messages from the kernel at boot:
Failed to execute /init (error -2)
It appears that the overlay mechanisms was causing my issues.
Description:
The overlay mechanisms works by rsync'ing files from a folder specified by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are symlinked into /usr.
Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content. Obviously, this causes a lot of troubles, starting with the kernel failing to start init (because of missing shared libraries).
Quoted 7 lines
Quoted 3 lines
Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.
I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some directories so it becomes useable.
Just a precision: it's not all the symlinks only the symlinks that points to a directory, like /var/spool
Quoted 5 lines
That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.
Still, you _are_ breaking a valid use case with this patch.

Quoted 8 lines

Quoted 3 lines
Another way to deal with this, is to test if in the overlay, there's a a folder that is a link is the default skeleton and in that case, treat it differently.
That is just dealing with the "accident" that the /bin and /lib symlinks are
created with an explicit ln instead of being present in the skeleton. It could just as well have been the reverse, that they are symlinks in the skeleton and mkdirs in the makefile. So I wouldn't rely on that.

Or we could do rsync twice, once where we exclude /sbin, /bin and /lib from the overlay and do it as it was, and once again after which include only /bin, /sbin and /lib from the overlay with the --keep-dirlinks ?

Quoted 16 lines

Regards,
Arnout

Quoted 2 lines

In the spur of the moment it seems too complicated. But maybe it's the right solution after all.

[snip]

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Peter Korsgaard

2016-02-11 22:26 UTC #5 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

In reply to Arnout Vandecappelle

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Hi,

Quoted 8 lines

Quoted 3 lines
Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.
I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some directories so it becomes useable.

Quoted 6 lines

That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.
Still, you _are_ breaking a valid use case with this patch.
What do the others think?

I'm also not really happy about it either. Another potential issue is if the build machine is so old that rsync doesn't support --keep-dirlinks, but according to:

https://download.samba.org/pub/unpacked/rsync/OLDNEWS

This was added in 2.6.3 (Sep 2004), so I guess even RHEL5 and similar should have it.

Signature · 1 line

Bye, Peter Korsgaard

Maxime Hadjinlian

author

2016-02-12 13:40 UTC #6 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

Hi Peter, all

Quoted 19 lines · Peter Korsgaard

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 8 lines
Quoted 3 lines
Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.
I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some directories so it becomes useable.
Quoted 6 lines
That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.
Still, you _are_ breaking a valid use case with this patch.
What do the others think?

Quoted 8 lines

I'm also not really happy about it either. Another potential issue is if the build machine is so old that rsync doesn't support --keep-dirlinks, but according to:
https://download.samba.org/pub/unpacked/rsync/OLDNEWS
This was added in 2.6.3 (Sep 2004), so I guess even RHEL5 and similar should have it.

So that's not really an issue then, is it ?

What do you think about the idea of doing rsync twice ? That should solve every use case pretty nicely I think.

Quoted 3 lines

--
Bye, Peter Korsgaard

Yann E. MORIN

2016-06-02 22:20 UTC #7 lore raw

PATCH 1/1 Makefile: Fix overlay overwriting everything

In reply to Maxime Hadjinlian

Maxime, All,

On 2016-02-10 16:58 +0100, Maxime Hadjinlian spake thusly:

Quoted 20 lines

After bumping from a 2 years old Buildroot installation, I got the following messages from the kernel at boot:
Failed to execute /init (error -2)
It appears that the overlay mechanisms was causing my issues.
Description:
The overlay mechanisms works by rsync'ing files from a folder specified by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are symlinked into /usr.
Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content. Obviously, this causes a lot of troubles, starting with the kernel failing to start init (because of missing shared libraries).

Quoted 3 lines

Solution:
Telling rsync to treat symlinked dir on the receiver side as directory allow users to be blissfully ignorant of that problem.

It seems that this has already been fixed by a patch by Peter Seiderer:

0db3452 Makefile: fix rootfs overlay with BR2_ROOTFS_MERGED_USR
enabled

Can you confirm that this also fixes your issue? If so, care to mark your patch as superseded in Patchwork, please?

If not, just chime in!

Thanks!

Regards,
Yann E. MORIN.

Quoted 25 lines

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Makefile +1 -1

index 71735b5..2f33ddf 100644
@@ -655,7 +655,7 @@ endif
 
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
-		rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+		rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
 			--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
 			$(d)/ $(TARGET_DIR)$(sep))

Signature · 6 lines

2.7.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'