utils/docker-run: make it compatible with SELinux
RFC PATCH 1/2 utils/docker-run: make it compatible with SELinux
After switching to a fresh Fedora 38 installation with SELinux disabled, we noticed that utils/docker-run doesn't work as the applications running inside the container are not allowed to accept the data mounted through the bind mount.
Turns out that Docker has a "Z" option to do the appropriate magic for SELinux. However, this "Z" option is only available for --volume, not for --mount, as explained in
https://docs.docker.com/storage/bind-mounts/.
So, this commit partially reverts 7f2020f9040f ("utils/docker-run:
improve user experience") that switched from --volume to --mount. However, the justification in 7f2020f9040f to switch from --volume to --mount was "Docker will create the destination if it does not exist", but the current Docker documentation seems to say exactly the opposite:
If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.
If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.
---
NOTE: I am not a Docker expert, and I certainly don't know if this is the right solution, and I would appreciate feedback from folks with more Docker experience.
--- utils/docker-run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
utils/docker-run +2 -2
@@ -12,8 +12,8 @@ declare -a docker_opts=( -i --rm --user "$(id -u):$(id -g)" - --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}" - --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}" + --volume "${MAIN_DIR}:${MAIN_DIR}:Z" + --volume "${GIT_DIR}:${GIT_DIR}:Z" --workdir "${MAIN_DIR}" ) if tty -s; then
Signature · 6 lines
2.41.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
RFC PATCH 2/2 utils/docker-run: mount the download directory if specified
If the user has defined $BR2_DL_DIR in the environment, it would be nice to have it accessible inside the Docker container, and the BR2_DL_DIR environment variable set to access it.
This commit does exactly this: it mounts the host $BR2_DL_DIR as /dl in the container, and sets BR2_DL_DIR=/dl in the container.
---
Here as well, opinion from Docker expert would be useful. For example, with this change, the files added in the download directory are owned by $USER:docker and not $USER:$USER as would probably be expected.
--- utils/docker-run | 4 ++++ 1 file changed, 4 insertions(+)
utils/docker-run +4 -0
@@ -20,4 +20,8 @@ if tty -s; then docker_opts+=( -t ) fi +if test -n "${BR2_DL_DIR}"; then + docker_opts+=( --volume "${BR2_DL_DIR}:/dl:Z" -e "BR2_DL_DIR=/dl" ) +fi + exec docker run "${docker_opts[@]}" "${IMAGE}" "${@}"
Signature · 6 lines
2.41.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH RFC 2/2] utils/docker-run: mount the download directory if specified
Thomas, All,
On 2023-07-26 23:20 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 4 lines
If the user has defined $BR2_DL_DIR in the environment, it would be nice to have it accessible inside the Docker container, and the
BR2_DL_DIRenvironment variable set to access it.
Quoted 8 lines
This commit does exactly this: it mounts the host $BR2_DL_DIR as /dl in the container, and sets BR2_DL_DIR=/dl in the container.
Signed-off-by Thomas Petazzoni <[email protected]>---
Here as well, opinion from Docker expert would be useful. For example, with this change, the files added in the download directory are owned by $USER:docker and not $USER:$USER as would probably be expected.
Here, with --mount, they do belong to $USER:$USER...
So, I guess your issue is that /dl does not exist in the container, so it is created by the docker runtime and thus group-belongs to docker, and the SELinux labelling means that everything beneath it also belongs to docker.
Also:
* what about files that already existed before: are the chgrp-ed to docker, or do they retain their group?
* and from within the container, whom do the files belong to?
Regards,
Yann E. MORIN.
Quoted 24 lines
--- utils/docker-run | 4 ++++ 1 file changed, 4 insertions(+)utils/docker-run +4
-0@@ -20,4 +20,8 @@ if tty -s; then docker_opts+=( -t ) fi +if test -n "${BR2_DL_DIR}"; then + docker_opts+=( --volume "${BR2_DL_DIR}:/dl:Z" -e "BR2_DL_DIR=/dl" ) +fi + exec docker run "${docker_opts[@]}" "${IMAGE}" "${@}"Signature · 6 lines
2.41.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH RFC 2/2] utils/docker-run: mount the download directory if specified
Hello,
Quoted 9 lines · On Tue, 8 Aug 2023 22:50:47 +0200 "Yann E. MORIN" <[email protected]>…
So, I guess your issue is that /dl does not exist in the container, so it is created by the docker runtime and thus group-belongs to docker, and the SELinux labelling means that everything beneath it also belongs to docker.
Also:
* what about files that already existed before: are the chgrp-ed to docker, or do they retain their group?
Folders in /dl that existed before in the BR2_DL_DIR on my "host" look like this outside the container:
$ ls -l ~/dl/xz drwxr-xr-x. 2 thomas thomas 4096 6 août 00:05 /home/thomas/dl/xz
and inside the container:
$ ./utils/docker-run ls -ld /dl/xz drwxr-xr-x. 2 br-user br-user 4096 Aug 5 22:05 /dl/xz
Quoted 1 line
* and from within the container, whom do the files belong to?
The files downloaded in /dl from within the container (by running a Buildroot build in the container, that causes some downloads to take place). They appear outside the container as such:
$ ls -ld ~/dl/swig drwxr-xr-x. 2 thomas docker 4096 26 juil. 23:00 /home/thomas/dl/swig/
And within the container they appear as such:
$ ./utils/docker-run ls -ld /dl/swig drwxr-xr-x. 2 br-user 976 4096 Jul 26 21:00 /dl/swig
Inside the container, the /dl directory itself is owned by br-user:br-user:
$ ./utils/docker-run ls -ld /dl drwxr-xr-x. 276 br-user br-user 12288 Aug 8 21:18 /dl
Does that answer your question?
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Thomas,
Quoting Thomas Petazzoni via buildroot (2023-07-26 23:20:07)
utils/docker-run declare -a docker_opts=( · line 16
Quoted 13 lines
utils/docker-run +2
-2@@ -12,8 +12,8 @@ declare -a docker_opts=( -i --rm --user "$(id -u):$(id -g)" - --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}" - --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}" + --volume "${MAIN_DIR}:${MAIN_DIR}:Z" + --volume "${GIT_DIR}:${GIT_DIR}:Z"
Using Z will label all the files in MAIN_DIR and GIT_DIR with a private label and IIRC that means relabeling all files each time a new container is started; which can take quite some time if there are lots if files in there. However z can be used to label files with a shared label and they won't be relabeled after the first run.
Antoine
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Antoine,
Quoted 8 lines · On Thu, 27 Jul 2023 10:13:53 +0200 Antoine Tenart <[email protected]> wrote:
utils/docker-run declare -a docker_opts=( · line 16
Quoted 2 lines
utils/docker-run +2
-0+ --volume "${MAIN_DIR}:${MAIN_DIR}:Z" + --volume "${GIT_DIR}:${GIT_DIR}:Z"Using Z will label all the files in
MAIN_DIRandGIT_DIRwith a private label and IIRC that means relabeling all files each time a new container is started; which can take quite some time if there are lots if files in there. However z can be used to label files with a shared label and they won't be relabeled after the first run.
Thanks for the hint, makes sense!
Do you know why --mount, which is apparently "superior" and recommended over --volume, does not support this SELinux labeling mechanism?
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoting Thomas Petazzoni via buildroot (2023-07-27 10:48:21)
Quoted 3 lines
Do you know why --mount, which is apparently "superior" and recommended over --volume, does not support this SELinux labeling mechanism?
IIRC --mount was introduced for better consistency in the docker args and supporting names parameters (I don't know the full reasoning but it's probably linked to supporting APIs -like yaml definitions- in a better way?). While doing so they tried to fix issues in --volume and labeling files on mount was one of them.
The issue is both logical and practical: files should be labeled when being created (not when being mounted) and relabeling files "magically" can cause issues (eg. don't -v /home:/home:z !). The reasoning with the new --mount option is a directory should be created and configured on the host and then only mounted by containers (hence the directory is not anymore created if not present when using --mount).
So here, idk, not a docker expert :) I'd say using relabeling of a directory that is under Buildroot's control is probably OK. While replying, had a quick look at this and it seems the preferred solution would be instead to use the --security-opt label=disable option:
labels would be kept in sync with the host and I guess the goal of using Docker here is not for isolating the build but to have a known environment. I never played with that option so please investigate before switching to it.
Antoine
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello,
Quoted 13 lines · On Thu, 27 Jul 2023 12:19:48 +0200 Antoine Tenart <[email protected]> wrote:
IIRC --mount was introduced for better consistency in the docker args and supporting names parameters (I don't know the full reasoning but it's probably linked to supporting APIs -like yaml definitions- in a better way?). While doing so they tried to fix issues in --volume and labeling files on mount was one of them.
The issue is both logical and practical: files should be labeled when being created (not when being mounted) and relabeling files "magically" can cause issues (eg. don't
-v /home:/home:z!). The reasoning with the new --mount option is a directory should be created and configured on the host and then only mounted by containers (hence the directory is not anymore created if not present when using --mount).
Quoted 8 lines
So here, idk, not a docker expert :) I'd say using relabeling of a directory that is under Buildroot's control is probably OK. While replying, had a quick look at this and it seems the preferred solution would be instead to use the
--security-opt label=disableoption:
labels would be kept in sync with the host and I guess the goal of using Docker here is not for isolating the build but to have a known environment. I never played with that option so please investigate before switching to it.
Thanks for the hint!
I tried --security-opt label=disable, and it works, at least it fixes my permission issue, without having to do the z/Z thing. Probably it's a better option than using z/Z that has the side effect of adding SELinux labels on all files being mounted? To be honest, I'm not clear on the consequences of --security-opt label=disable.
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoting Thomas Petazzoni (2023-07-27 12:24:49)
Quoted 25 lines
Quoted 13 lines · On Thu, 27 Jul 2023 12:19:48 +0200 Antoine Tenart <[email protected]> wrote:
IIRC --mount was introduced for better consistency in the docker args and supporting names parameters (I don't know the full reasoning but it's probably linked to supporting APIs -like yaml definitions- in a better way?). While doing so they tried to fix issues in --volume and labeling files on mount was one of them.
The issue is both logical and practical: files should be labeled when being created (not when being mounted) and relabeling files "magically" can cause issues (eg. don't
-v /home:/home:z!). The reasoning with the new --mount option is a directory should be created and configured on the host and then only mounted by containers (hence the directory is not anymore created if not present when using --mount).Quoted 8 lines
So here, idk, not a docker expert :) I'd say using relabeling of a directory that is under Buildroot's control is probably OK. While replying, had a quick look at this and it seems the preferred solution would be instead to use the
--security-opt label=disableoption:
labels would be kept in sync with the host and I guess the goal of using Docker here is not for isolating the build but to have a known environment. I never played with that option so please investigate before switching to it.
Quoted 5 lines
I tried --security-opt label=disable, and it works, at least it fixes my permission issue, without having to do the z/Z thing. Probably it's a better option than using z/Z that has the side effect of adding SELinux labels on all files being mounted? To be honest, I'm not clear on the consequences of --security-opt label=disable.
I agree on not relabeling files if that's an option. Not sure about the consequences of --security-opt label=disable too. My understanding is the containerized process won't be labeled with a container-specific label, but they still could be labeled with a permissive one not 100% matching the user label.
Without having a full understanding of the implications we can still expect child processes not to have more rights than the parent and created file labels to match what would have happened on the host (could you check?). Given the goal is not to isolate the build, my guess is that is fine and the best option.
Antoine
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoting Antoine Tenart (2023-07-27 12:50:17)
Quoted 3 lines
Without having a full understanding of the implications we can still expect child processes not to have more rights than the parent
Hmm, my point doesn't work for non-rootless containers, the Docker deamon runs as root. Not that this changes the other points.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas, All,
On 2023-07-26 23:20 +0200, Thomas Petazzoni spake thusly:
Quoted 9 lines
After switching to a fresh Fedora 38 installation with SELinux disabled, we noticed that utils/docker-run doesn't work as the applications running inside the container are not allowed to accept the data mounted through the bind mount.
Turns out that Docker has a "Z" option to do the appropriate magic for SELinux. However, this "Z" option is only available for --volume, not for --mount, as explained in
https://docs.docker.com/storage/bind-mounts/.
hat about those systems that do not have SELinux: how is z/Z going to work for those?
Quoted 7 lines
So, this commit partially reverts
7f2020f9040f("utils/docker-run:
improve user experience") that switched from --volume to --mount. However, the justification in7f2020f9040fto switch from --volume to --mount was "Docker will create the destination if it does not exist", but the current Docker documentation seems to say exactly the opposite:
Quoted 7 lines
If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.
If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.
Note that this is about "on the docker host", so the source, while the commit mentions "the destination in the container".
I can't where I initially read that, since the pointer in 7f2020f9040f is no longer valid (the anchor no longer exists).
Quoted 27 lines
Signed-off-by Thomas Petazzoni <[email protected]>---
NOTE: I am not a Docker expert, and I certainly don't know if this is the right solution, and I would appreciate feedback from folks with more Docker experience.--- utils/docker-run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)utils/docker-run +2
-2@@ -12,8 +12,8 @@ declare -a docker_opts=( -i --rm --user "$(id -u):$(id -g)" - --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}" - --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}" + --volume "${MAIN_DIR}:${MAIN_DIR}:Z" + --volume "${GIT_DIR}:${GIT_DIR}:Z" --workdir "${MAIN_DIR}" ) if tty -s; thenSignature · 1 line
2.41.0
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
A message that is not in the archive (sent off-list or not yet received): CA+h8R2qJF87Wi_w9DBjFZO__x=Kku+hfU1_-uhn2tLegFtc37g@mail.gmail.com
Hello Christian,
Quoted 9 lines · On Thu, 27 Jul 2023 20:13:08 -0700 Christian Stewart <[email protected]> wrote:
utils/docker-run declare -a docker_opts=( · line 17
Quoted 5 lines · Thomas Petazzoni
utils/docker-run +2
-2- --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}" - --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}" + --volume "${MAIN_DIR}:${MAIN_DIR}:Z" + --volume "${GIT_DIR}:${GIT_DIR}:Z" --workdir "${MAIN_DIR}"What is the purpose of the
GIT_DIRmount here, doesn'tMAIN_DIRcontain .git?
This is explained in 791c163b2f9f07d4c02b18eabd9b195918e1c603:
commit 791c163b2f9f07d4c02b18eabd9b195918e1c603
Author: Yann E. MORIN <[email protected]> Date: Sat May 6 23:46:18 2023 +0200
utils/docker-run: make it work in workdirs/woktrees
It is quite customary to use a single repository with multiple workdirs, one for each active branch, with either the aging 'git new-workdir' or the more recent 'git worktree'.
However, in a workdir/worktree, most entries in .git/ are only symlinks to the actual files in the main repository.
Currently, utils/docker-run only bind-mounts the current working copy. If that is a workdir/worktree, then it is going to be missing the actual git data, resulting in errors like:
$ ./utils/docker-run make check-package
fatal: not a git repository (or any parent up to mount point [....]/buildroot) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). No files to check style
make: *** [Makefile:1257: check-package] Error 1
So, we also bind-mount the actual git directory. If that is a subdir of the current working copy, then it is already mounted and thus the bind-mount is superfluous but harmless; for simplicity, we mount it unconditionally.
Quoted 2 lines
I think this mount might be causing some unpredictable behavior in the host-go package, but it's just a hunch at the moment.
Did you confirm this?
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas, All,
On 2023-07-26 23:20 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 21 lines
After switching to a fresh Fedora 38 installation with SELinux disabled, we noticed that utils/docker-run doesn't work as the applications running inside the container are not allowed to accept the data mounted through the bind mount.
Turns out that Docker has a "Z" option to do the appropriate magic for SELinux. However, this "Z" option is only available for --volume, not for --mount, as explained in
https://docs.docker.com/storage/bind-mounts/.So, this commit partially reverts
7f2020f9040f("utils/docker-run:
improve user experience") that switched from --volume to --mount. However, the justification in7f2020f9040fto switch from --volume to --mount was "Docker will create the destination if it does not exist", but the current Docker documentation seems to say exactly the opposite:If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.
Quoted 5 lines
If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.
Signed-off-by Thomas Petazzoni <[email protected]>
I've included this change in my respin:
https://patchwork.ozlabs.org/project/buildroot/list/?series=368134
In doing so, I switched to the proposal by Antoine, to use --security-opt label=disable
I did not see an adverse effect on a non-SELinux host, and you reported that it also worked for you, so let's ship it.
I've also included patch 2 in the respin.
Regards,
Yann E. MORIN.
Quoted 30 lines
---
NOTE: I am not a Docker expert, and I certainly don't know if this is the right solution, and I would appreciate feedback from folks with more Docker experience.--- utils/docker-run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)utils/docker-run +2
-2@@ -12,8 +12,8 @@ declare -a docker_opts=( -i --rm --user "$(id -u):$(id -g)" - --mount "type=bind,src=${MAIN_DIR},dst=${MAIN_DIR}" - --mount "type=bind,src=${GIT_DIR},dst=${GIT_DIR}" + --volume "${MAIN_DIR}:${MAIN_DIR}:Z" + --volume "${GIT_DIR}:${GIT_DIR}:Z" --workdir "${MAIN_DIR}" ) if tty -s; thenSignature · 6 lines
2.41.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot