Skip to contentSkip to conversation

utils/docker-run: make it compatible with SELinux

RFC PATCH In review Thomas Petazzoni via buildroot 3 participants 13 messages Jul 26, 2023 – Aug 9, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/2 utils/docker-run: make it compatible with SELinux

After switching to a fresh Fedora 38 installation with SELinux disabled, we noticed that utils/docker-run doesn't work as the applications running inside the container are not allowed to accept the data mounted through the bind mount.

Turns out that Docker has a "Z" option to do the appropriate magic for SELinux. However, this "Z" option is only available for --volume, not for --mount, as explained in
https://docs.docker.com/storage/bind-mounts/.

So, this commit partially reverts 7f2020f9040f ("utils/docker-run:
improve user experience") that switched from --volume to --mount. However, the justification in 7f2020f9040f to switch from --volume to --mount was "Docker will create the destination if it does not exist", but the current Docker documentation seems to say exactly the opposite:

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.

If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.

---
NOTE: I am not a Docker expert, and I certainly don't know if this is the right solution, and I would appreciate feedback from folks with more Docker experience.

---
 utils/docker-run | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
utils/docker-run +2 -2
index 17c587a484..eee1aad7a4 100755
@@ -12,8 +12,8 @@ declare -a docker_opts=(
     -i
     --rm
     --user "$(id -u):$(id -g)"
-    --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}"
-    --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}"
+    --volume "${MAIN_DIR}:${MAIN_DIR}:Z"
+    --volume "${GIT_DIR}:${GIT_DIR}:Z"
     --workdir "${MAIN_DIR}"
 )
 if tty -s; then
Signature · 6 lines
2.41.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

RFC PATCH 2/2 utils/docker-run: mount the download directory if specified

If the user has defined $BR2_DL_DIR in the environment, it would be nice to have it accessible inside the Docker container, and the BR2_DL_DIR environment variable set to access it.

This commit does exactly this: it mounts the host $BR2_DL_DIR as /dl in the container, and sets BR2_DL_DIR=/dl in the container.

---
Here as well, opinion from Docker expert would be useful. For example, with this change, the files added in the download directory are owned by $USER:docker and not $USER:$USER as would probably be expected.

---
 utils/docker-run | 4 ++++
 1 file changed, 4 insertions(+)
utils/docker-run +4 -0
index eee1aad7a4..6ea4311c68 100755
@@ -20,4 +20,8 @@ if tty -s; then
     docker_opts+=( -t )
 fi
 
+if test -n "${BR2_DL_DIR}"; then
+    docker_opts+=( --volume "${BR2_DL_DIR}:/dl:Z" -e "BR2_DL_DIR=/dl" )
+fi
+
 exec docker run "${docker_opts[@]}" "${IMAGE}" "${@}"
Signature · 6 lines
2.41.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH RFC 2/2] utils/docker-run: mount the download directory if specified

Thomas, All,

On 2023-07-26 23:20 +0200, Thomas Petazzoni via buildroot spake thusly:

Quoted 4 lines

If the user has defined $BR2_DL_DIR in the environment, it would be nice to have it accessible inside the Docker container, and the BR2_DL_DIR environment variable set to access it.

Quoted 8 lines

This commit does exactly this: it mounts the host $BR2_DL_DIR as /dl in the container, and sets BR2_DL_DIR=/dl in the container.

---
Here as well, opinion from Docker expert would be useful. For example, with this change, the files added in the download directory are owned by $USER:docker and not $USER:$USER as would probably be expected.

Here, with --mount, they do belong to $USER:$USER...

So, I guess your issue is that /dl does not exist in the container, so it is created by the docker runtime and thus group-belongs to docker, and the SELinux labelling means that everything beneath it also belongs to docker.

Also:

* what about files that already existed before: are the chgrp-ed to docker, or do they retain their group?

* and from within the container, whom do the files belong to?

Regards,
Yann E. MORIN.

Quoted 24 lines
---
 utils/docker-run | 4 ++++
 1 file changed, 4 insertions(+)
utils/docker-run +4 -0
index eee1aad7a4..6ea4311c68 100755
@@ -20,4 +20,8 @@ if tty -s; then
     docker_opts+=( -t )
 fi
 
+if test -n "${BR2_DL_DIR}"; then
+    docker_opts+=( --volume "${BR2_DL_DIR}:/dl:Z" -e "BR2_DL_DIR=/dl" )
+fi
+
 exec docker run "${docker_opts[@]}" "${IMAGE}" "${@}"
Signature · 6 lines
2.41.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

Re: [Buildroot] [PATCH RFC 2/2] utils/docker-run: mount the download directory if specified

Hello,

Quoted 9 lines · On Tue, 8 Aug 2023 22:50:47 +0200 "Yann E. MORIN" <[email protected]>…

So, I guess your issue is that /dl does not exist in the container, so it is created by the docker runtime and thus group-belongs to docker, and the SELinux labelling means that everything beneath it also belongs to docker.

Also:

* what about files that already existed before: are the chgrp-ed to docker, or do they retain their group?

Folders in /dl that existed before in the BR2_DL_DIR on my "host" look like this outside the container:

$ ls -l ~/dl/xz
drwxr-xr-x. 2 thomas thomas 4096  6 août  00:05 /home/thomas/dl/xz

and inside the container:

$ ./utils/docker-run ls -ld /dl/xz
drwxr-xr-x. 2 br-user br-user 4096 Aug  5 22:05 /dl/xz
Quoted 1 line

* and from within the container, whom do the files belong to?

The files downloaded in /dl from within the container (by running a Buildroot build in the container, that causes some downloads to take place). They appear outside the container as such:

$ ls -ld ~/dl/swig
drwxr-xr-x. 2 thomas docker 4096 26 juil. 23:00 /home/thomas/dl/swig/

And within the container they appear as such:

$ ./utils/docker-run ls -ld /dl/swig
drwxr-xr-x. 2 br-user 976 4096 Jul 26 21:00 /dl/swig

Inside the container, the /dl directory itself is owned by br-user:br-user:

$ ./utils/docker-run ls -ld /dl
drwxr-xr-x. 276 br-user br-user 12288 Aug  8 21:18 /dl

Does that answer your question?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw
In reply to Thomas Petazzoni via buildroot

Hi Thomas,

Quoting Thomas Petazzoni via buildroot (2023-07-26 23:20:07)

utils/docker-run declare -a docker_opts=( · line 16

Quoted 13 lines
utils/docker-run +2 -2
index 17c587a484..eee1aad7a4 100755
@@ -12,8 +12,8 @@ declare -a docker_opts=(
     -i
     --rm
     --user "$(id -u):$(id -g)"
-    --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}"
-    --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}"
+    --volume "${MAIN_DIR}:${MAIN_DIR}:Z"
+    --volume "${GIT_DIR}:${GIT_DIR}:Z"

Using Z will label all the files in MAIN_DIR and GIT_DIR with a private label and IIRC that means relabeling all files each time a new container is started; which can take quite some time if there are lots if files in there. However z can be used to label files with a shared label and they won't be relabeled after the first run.

Antoine
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#5 lore raw

Hello Antoine,

Quoted 8 lines · On Thu, 27 Jul 2023 10:13:53 +0200 Antoine Tenart <[email protected]> wrote:

utils/docker-run declare -a docker_opts=( · line 16

Quoted 2 lines
utils/docker-run +2 -0
+    --volume "${MAIN_DIR}:${MAIN_DIR}:Z"
+    --volume "${GIT_DIR}:${GIT_DIR}:Z"  

Using Z will label all the files in MAIN_DIR and GIT_DIR with a private label and IIRC that means relabeling all files each time a new container is started; which can take quite some time if there are lots if files in there. However z can be used to label files with a shared label and they won't be relabeled after the first run.

Thanks for the hint, makes sense!

Do you know why --mount, which is apparently "superior" and recommended over --volume, does not support this SELinux labeling mechanism?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#6 lore raw

Quoting Thomas Petazzoni via buildroot (2023-07-27 10:48:21)

Quoted 3 lines

Do you know why --mount, which is apparently "superior" and recommended over --volume, does not support this SELinux labeling mechanism?

IIRC --mount was introduced for better consistency in the docker args and supporting names parameters (I don't know the full reasoning but it's probably linked to supporting APIs -like yaml definitions- in a better way?). While doing so they tried to fix issues in --volume and labeling files on mount was one of them.

The issue is both logical and practical: files should be labeled when being created (not when being mounted) and relabeling files "magically" can cause issues (eg. don't -v /home:/home:z !). The reasoning with the new --mount option is a directory should be created and configured on the host and then only mounted by containers (hence the directory is not anymore created if not present when using --mount).

So here, idk, not a docker expert :) I'd say using relabeling of a directory that is under Buildroot's control is probably OK. While replying, had a quick look at this and it seems the preferred solution would be instead to use the --security-opt label=disable option:
labels would be kept in sync with the host and I guess the goal of using Docker here is not for isolating the build but to have a known environment. I never played with that option so please investigate before switching to it.

Antoine
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#7 lore raw

Hello,

Quoted 13 lines · On Thu, 27 Jul 2023 12:19:48 +0200 Antoine Tenart <[email protected]> wrote:

IIRC --mount was introduced for better consistency in the docker args and supporting names parameters (I don't know the full reasoning but it's probably linked to supporting APIs -like yaml definitions- in a better way?). While doing so they tried to fix issues in --volume and labeling files on mount was one of them.

The issue is both logical and practical: files should be labeled when being created (not when being mounted) and relabeling files "magically" can cause issues (eg. don't -v /home:/home:z !). The reasoning with the new --mount option is a directory should be created and configured on the host and then only mounted by containers (hence the directory is not anymore created if not present when using --mount).

Quoted 8 lines

So here, idk, not a docker expert :) I'd say using relabeling of a directory that is under Buildroot's control is probably OK. While replying, had a quick look at this and it seems the preferred solution would be instead to use the --security-opt label=disable option:
labels would be kept in sync with the host and I guess the goal of using Docker here is not for isolating the build but to have a known environment. I never played with that option so please investigate before switching to it.

Thanks for the hint!

I tried --security-opt label=disable, and it works, at least it fixes my permission issue, without having to do the z/Z thing. Probably it's a better option than using z/Z that has the side effect of adding SELinux labels on all files being mounted? To be honest, I'm not clear on the consequences of --security-opt label=disable.

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#8 lore raw

Quoting Thomas Petazzoni (2023-07-27 12:24:49)

Quoted 25 lines
Quoted 13 lines · On Thu, 27 Jul 2023 12:19:48 +0200 Antoine Tenart <[email protected]> wrote:

IIRC --mount was introduced for better consistency in the docker args and supporting names parameters (I don't know the full reasoning but it's probably linked to supporting APIs -like yaml definitions- in a better way?). While doing so they tried to fix issues in --volume and labeling files on mount was one of them.

The issue is both logical and practical: files should be labeled when being created (not when being mounted) and relabeling files "magically" can cause issues (eg. don't -v /home:/home:z !). The reasoning with the new --mount option is a directory should be created and configured on the host and then only mounted by containers (hence the directory is not anymore created if not present when using --mount).

Quoted 8 lines

So here, idk, not a docker expert :) I'd say using relabeling of a directory that is under Buildroot's control is probably OK. While replying, had a quick look at this and it seems the preferred solution would be instead to use the --security-opt label=disable option:
labels would be kept in sync with the host and I guess the goal of using Docker here is not for isolating the build but to have a known environment. I never played with that option so please investigate before switching to it.

Quoted 5 lines

I tried --security-opt label=disable, and it works, at least it fixes my permission issue, without having to do the z/Z thing. Probably it's a better option than using z/Z that has the side effect of adding SELinux labels on all files being mounted? To be honest, I'm not clear on the consequences of --security-opt label=disable.

I agree on not relabeling files if that's an option. Not sure about the consequences of --security-opt label=disable too. My understanding is the containerized process won't be labeled with a container-specific label, but they still could be labeled with a permissive one not 100% matching the user label.

Without having a full understanding of the implications we can still expect child processes not to have more rights than the parent and created file labels to match what would have happened on the host (could you check?). Given the goal is not to isolate the build, my guess is that is fine and the best option.

Antoine
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#9 lore raw

Quoting Antoine Tenart (2023-07-27 12:50:17)

Quoted 3 lines

Without having a full understanding of the implications we can still expect child processes not to have more rights than the parent

Hmm, my point doesn't work for non-rootless containers, the Docker deamon runs as root. Not that this changes the other points.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#10 lore raw
In reply to Thomas Petazzoni via buildroot

Thomas, All,

On 2023-07-26 23:20 +0200, Thomas Petazzoni spake thusly:

Quoted 9 lines

After switching to a fresh Fedora 38 installation with SELinux disabled, we noticed that utils/docker-run doesn't work as the applications running inside the container are not allowed to accept the data mounted through the bind mount.

Turns out that Docker has a "Z" option to do the appropriate magic for SELinux. However, this "Z" option is only available for --volume, not for --mount, as explained in
https://docs.docker.com/storage/bind-mounts/.

hat about those systems that do not have SELinux: how is z/Z going to work for those?

Quoted 7 lines

So, this commit partially reverts 7f2020f9040f ("utils/docker-run:
improve user experience") that switched from --volume to --mount. However, the justification in 7f2020f9040f to switch from --volume to --mount was "Docker will create the destination if it does not exist", but the current Docker documentation seems to say exactly the opposite:

Quoted 7 lines

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.

If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.

Note that this is about "on the docker host", so the source, while the commit mentions "the destination in the container".

I can't where I initially read that, since the pointer in 7f2020f9040f is no longer valid (the anchor no longer exists).

Quoted 27 lines

---
NOTE: I am not a Docker expert, and I certainly don't know if this is the right solution, and I would appreciate feedback from folks with more Docker experience.

---
 utils/docker-run | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
utils/docker-run +2 -2
index 17c587a484..eee1aad7a4 100755
@@ -12,8 +12,8 @@ declare -a docker_opts=(
     -i
     --rm
     --user "$(id -u):$(id -g)"
-    --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}"
-    --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}"
+    --volume "${MAIN_DIR}:${MAIN_DIR}:Z"
+    --volume "${GIT_DIR}:${GIT_DIR}:Z"
     --workdir "${MAIN_DIR}"
 )
 if tty -s; then
Signature · 1 line
2.41.0
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

A message that is not in the archive (sent off-list or not yet received): CA+h8R2qJF87Wi_w9DBjFZO__x=Kku+hfU1_-uhn2tLegFtc37g@mail.gmail.com

#12 lore raw

Hello Christian,

Quoted 9 lines · On Thu, 27 Jul 2023 20:13:08 -0700 Christian Stewart <[email protected]> wrote:

utils/docker-run declare -a docker_opts=( · line 17

Quoted 5 lines · Thomas Petazzoni
utils/docker-run +2 -2
-    --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}"
-    --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}"
+    --volume "${MAIN_DIR}:${MAIN_DIR}:Z"
+    --volume "${GIT_DIR}:${GIT_DIR}:Z"
     --workdir "${MAIN_DIR}"  

What is the purpose of the GIT_DIR mount here, doesn't MAIN_DIR contain .git?

This is explained in 791c163b2f9f07d4c02b18eabd9b195918e1c603:

commit 791c163b2f9f07d4c02b18eabd9b195918e1c603

Author: Yann E. MORIN <[email protected]>
Date:   Sat May 6 23:46:18 2023 +0200

utils/docker-run: make it work in workdirs/woktrees

It is quite customary to use a single repository with multiple workdirs, one for each active branch, with either the aging 'git new-workdir' or the more recent 'git worktree'.

However, in a workdir/worktree, most entries in .git/ are only symlinks to the actual files in the main repository.

Currently, utils/docker-run only bind-mounts the current working copy. If that is a workdir/worktree, then it is going to be missing the actual git data, resulting in errors like:

$ ./utils/docker-run make check-package
fatal: not a git repository (or any parent up to mount point [....]/buildroot) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). No files to check style
make: *** [Makefile:1257: check-package] Error 1

So, we also bind-mount the actual git directory. If that is a subdir of the current working copy, then it is already mounted and thus the bind-mount is superfluous but harmless; for simplicity, we mount it unconditionally.

Cc Ricardo Martincoski <[email protected]>
Quoted 2 lines

I think this mount might be causing some unpredictable behavior in the host-go package, but it's just a hunch at the moment.

Did you confirm this?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#13 lore raw
In reply to Thomas Petazzoni via buildroot

Thomas, All,

On 2023-07-26 23:20 +0200, Thomas Petazzoni via buildroot spake thusly:

Quoted 21 lines

After switching to a fresh Fedora 38 installation with SELinux disabled, we noticed that utils/docker-run doesn't work as the applications running inside the container are not allowed to accept the data mounted through the bind mount.

Turns out that Docker has a "Z" option to do the appropriate magic for SELinux. However, this "Z" option is only available for --volume, not for --mount, as explained in
https://docs.docker.com/storage/bind-mounts/.

So, this commit partially reverts 7f2020f9040f ("utils/docker-run:
improve user experience") that switched from --volume to --mount. However, the justification in 7f2020f9040f to switch from --volume to --mount was "Docker will create the destination if it does not exist", but the current Docker documentation seems to say exactly the opposite:

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.

Quoted 5 lines

If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.

I've included this change in my respin:

    https://patchwork.ozlabs.org/project/buildroot/list/?series=368134

In doing so, I switched to the proposal by Antoine, to use --security-opt label=disable

I did not see an adverse effect on a non-SELinux host, and you reported that it also worked for you, so let's ship it.

I've also included patch 2 in the respin.

Regards,
Yann E. MORIN.

Quoted 30 lines

---
NOTE: I am not a Docker expert, and I certainly don't know if this is the right solution, and I would appreciate feedback from folks with more Docker experience.

---
 utils/docker-run | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
utils/docker-run +2 -2
index 17c587a484..eee1aad7a4 100755
@@ -12,8 +12,8 @@ declare -a docker_opts=(
     -i
     --rm
     --user "$(id -u):$(id -g)"
-    --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}"
-    --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}"
+    --volume "${MAIN_DIR}:${MAIN_DIR}:Z"
+    --volume "${GIT_DIR}:${GIT_DIR}:Z"
     --workdir "${MAIN_DIR}"
 )
 if tty -s; then
Signature · 6 lines
2.41.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot