Skip to contentSkip to conversation

virtiofsd: drop Linux capabilities(7)

PATCH In review Stefan Hajnoczi 8 participants 39 messages Apr 16, 2020 – Jul 14, 2020

lore
Share & tools
lore
#0 lore raw

cover virtiofsd: drop Linux capabilities(7)

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Stefan Hajnoczi (2):
virtiofsd: only retain file system capabilities
virtiofsd: drop all capabilities in the wait parent process

 tools/virtiofsd/passthrough_ll.c | 51 ++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 virtiofsd: only retain file system capabilities

virtiofsd runs as root but only needs a subset of root's Linux capabilities(7). As a file server its purpose is to create and access files on behalf of a client. It needs to be able to access files with arbitrary uid/gid owners. It also needs to be create device nodes.

Introduce a Linux capabilities(7) whitelist and drop all capabilities that we don't need, making the virtiofsd process less powerful than a regular uid root process.

  # cat /proc/PID/status
  ...
          Before           After
  CapInh: 0000000000000000 0000000000000000
  CapPrm: 0000003fffffffff 00000000880000df
  CapEff: 0000003fffffffff 00000000880000df
  CapBnd: 0000003fffffffff 0000000000000000
  CapAmb: 0000000000000000 0000000000000000

Note that file capabilities cannot be used to achieve the same effect on the virtiofsd executable because mount is used during sandbox setup. Therefore we drop capabilities programmatically at the right point during startup.

This patch only affects the sandboxed child process. The parent process that sits in waitpid(2) still has full root capabilities and will be addressed in the next patch.

---
 tools/virtiofsd/passthrough_ll.c | 38 ++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
tools/virtiofsd/passthrough_ll.c +38 -0
index 4c35c95b25..af97ba1c41 100644
@@ -2695,6 +2695,43 @@ static void setup_mounts(const char *source)
     close(oldroot);
 }
 
+/*
+ * Only keep whitelisted capabilities that are needed for file system operation
+ */
+static void setup_capabilities(void)
+{
+    pthread_mutex_lock(&cap.mutex);
+    capng_restore_state(&cap.saved);
+
+    /*
+     * Whitelist file system-related capabilities that are needed for a file
+     * server to act like root.  Drop everything else like networking and
+     * sysadmin capabilities.
+     *
+     * Exclusions:
+     * 1. CAP_LINUX_IMMUTABLE is not included because it's only used via ioctl
+     *    and we don't support that.
+     * 2. CAP_MAC_OVERRIDE is not included because it only seems to be
+     *    used by the Smack LSM.  Omit it until there is demand for it.
+     */
+    capng_setpid(syscall(SYS_gettid));
+    capng_clear(CAPNG_SELECT_BOTH);
+    capng_updatev(CAPNG_ADD, CAPNG_PERMITTED | CAPNG_EFFECTIVE,
+            CAP_CHOWN,
+            CAP_DAC_OVERRIDE,
+            CAP_DAC_READ_SEARCH,
+            CAP_FOWNER,
+            CAP_FSETID,
+            CAP_SETGID,
+            CAP_SETUID,
+            CAP_MKNOD,
+            CAP_SETFCAP);
+    capng_apply(CAPNG_SELECT_BOTH);
+
+    cap.saved = capng_save_state();
+    pthread_mutex_unlock(&cap.mutex);
+}
+
 /*
  * Lock down this process to prevent access to other processes or files outside
  * source directory.  This reduces the impact of arbitrary code execution bugs.
@@ -2705,6 +2742,7 @@ static void setup_sandbox(struct lo_data *lo, struct fuse_session *se,
     setup_namespaces(lo, se);
     setup_mounts(lo->source);
     setup_seccomp(enable_syslog);
+    setup_capabilities();
 }
 
 /* Raise the maximum number of open file descriptors */
Signature · 1 line
2.25.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] virtiofsd: only retain file system capabilities

Quoted 23 lines · * Stefan Hajnoczi ([email protected]) wrote:

virtiofsd runs as root but only needs a subset of root's Linux capabilities(7). As a file server its purpose is to create and access files on behalf of a client. It needs to be able to access files with arbitrary uid/gid owners. It also needs to be create device nodes.

Introduce a Linux capabilities(7) whitelist and drop all capabilities that we don't need, making the virtiofsd process less powerful than a regular uid root process.

  # cat /proc/PID/status
  ...
          Before           After
  CapInh: 0000000000000000 0000000000000000
  CapPrm: 0000003fffffffff 00000000880000df
  CapEff: 0000003fffffffff 00000000880000df
  CapBnd: 0000003fffffffff 0000000000000000
  CapAmb: 0000000000000000 0000000000000000

Note that file capabilities cannot be used to achieve the same effect on the virtiofsd executable because mount is used during sandbox setup. Therefore we drop capabilities programmatically at the right point during startup.

Quoted 5 lines

This patch only affects the sandboxed child process. The parent process that sits in waitpid(2) still has full root capabilities and will be addressed in the next patch.

Looks reasonable to me; I can't see any capabilities in the manpage that you're missing that make sense.
They also look old enough not to be a problem with reasonably old systems.

Quoted 63 lines
---
 tools/virtiofsd/passthrough_ll.c | 38 ++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
tools/virtiofsd/passthrough_ll.c +38 -0
index 4c35c95b25..af97ba1c41 100644
@@ -2695,6 +2695,43 @@ static void setup_mounts(const char *source)
     close(oldroot);
 }
 
+/*
+ * Only keep whitelisted capabilities that are needed for file system operation
+ */
+static void setup_capabilities(void)
+{
+    pthread_mutex_lock(&cap.mutex);
+    capng_restore_state(&cap.saved);
+
+    /*
+     * Whitelist file system-related capabilities that are needed for a file
+     * server to act like root.  Drop everything else like networking and
+     * sysadmin capabilities.
+     *
+     * Exclusions:
+     * 1. CAP_LINUX_IMMUTABLE is not included because it's only used via ioctl
+     *    and we don't support that.
+     * 2. CAP_MAC_OVERRIDE is not included because it only seems to be
+     *    used by the Smack LSM.  Omit it until there is demand for it.
+     */
+    capng_setpid(syscall(SYS_gettid));
+    capng_clear(CAPNG_SELECT_BOTH);
+    capng_updatev(CAPNG_ADD, CAPNG_PERMITTED | CAPNG_EFFECTIVE,
+            CAP_CHOWN,
+            CAP_DAC_OVERRIDE,
+            CAP_DAC_READ_SEARCH,
+            CAP_FOWNER,
+            CAP_FSETID,
+            CAP_SETGID,
+            CAP_SETUID,
+            CAP_MKNOD,
+            CAP_SETFCAP);
+    capng_apply(CAPNG_SELECT_BOTH);
+
+    cap.saved = capng_save_state();
+    pthread_mutex_unlock(&cap.mutex);
+}
+
 /*
  * Lock down this process to prevent access to other processes or files outside
  * source directory.  This reduces the impact of arbitrary code execution bugs.
@@ -2705,6 +2742,7 @@ static void setup_sandbox(struct lo_data *lo, struct fuse_session *se,
     setup_namespaces(lo, se);
     setup_mounts(lo->source);
     setup_seccomp(enable_syslog);
+    setup_capabilities();
 }
 
 /* Raise the maximum number of open file descriptors */
Signature · 1 line
2.25.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#3 lore raw

PATCH 2/2 virtiofsd: drop all capabilities in the wait parent process

In reply to Stefan Hajnoczi

All this process does is wait for its child. No capabilities are needed.

---
 tools/virtiofsd/passthrough_ll.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
tools/virtiofsd/passthrough_ll.c +13 -0
index af97ba1c41..0c3f33b074 100644
@@ -2530,6 +2530,17 @@ static void print_capabilities(void)
     printf("}\n");
 }
 
+/*
+ * Drop all Linux capabilities because the wait parent process only needs to
+ * sit in waitpid(2) and terminate.
+ */
+static void setup_wait_parent_capabilities(void)
+{
+    capng_setpid(syscall(SYS_gettid));
+    capng_clear(CAPNG_SELECT_BOTH);
+    capng_apply(CAPNG_SELECT_BOTH);
+}
+
 /*
  * Move to a new mount, net, and pid namespaces to isolate this process.
  */
@@ -2561,6 +2572,8 @@ static void setup_namespaces(struct lo_data *lo, struct fuse_session *se)
         pid_t waited;
         int wstatus;
 
+        setup_wait_parent_capabilities();
+
         /* The parent waits for the child */
         do {
             waited = waitpid(child, &wstatus, 0);
Signature · 1 line
2.25.1
Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] virtiofsd: drop all capabilities in the wait parent process

Quoted 13 lines · Stefan Hajnoczi

All this process does is wait for its child. No capabilities are needed.

---
  tools/virtiofsd/passthrough_ll.c | 13 +++++++++++++
  1 file changed, 13 insertions(+)
tools/virtiofsd/passthrough_ll.c +0 -0
index af97ba1c41..0c3f33b074 100644

tools/virtiofsd/passthrough_ll.c static void print_capabilities(void) · line 2539

Quoted 10 lines
tools/virtiofsd/passthrough_ll.c +7 -0
@@ -2530,6 +2530,17 @@ static void print_capabilities(void)
      printf("}\n");
  }
 
+/*
+ * Drop all Linux capabilities because the wait parent process only needs to
+ * sit in waitpid(2) and terminate.
+ */
+static void setup_wait_parent_capabilities(void)
+{
+    capng_setpid(syscall(SYS_gettid));

Maybe worth a /* Drop all capabilities */ comment here.

Quoted 17 lines
tools/virtiofsd/passthrough_ll.c +6 -0
+    capng_clear(CAPNG_SELECT_BOTH);
+    capng_apply(CAPNG_SELECT_BOTH);
+}
+
  /*
   * Move to a new mount, net, and pid namespaces to isolate this process.
   */
@@ -2561,6 +2572,8 @@ static void setup_namespaces(struct lo_data *lo, struct fuse_session *se)
          pid_t waited;
          int wstatus;
 
+        setup_wait_parent_capabilities();
+
          /* The parent waits for the child */
          do {
              waited = waitpid(child, &wstatus, 0);
#5 lore raw
In reply to Stefan Hajnoczi
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

Good to see this patch. We needed to limit capabilities to reduce attack surface.

What tests have you run to make sure this current set of whitelisted capabilities is good enough.

Vivek

Quoted 11 lines

Stefan Hajnoczi (2):
virtiofsd: only retain file system capabilities
virtiofsd: drop all capabilities in the wait parent process

 tools/virtiofsd/passthrough_ll.c | 51 ++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
Signature · 1 line
2.25.1
#6 lore raw
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 7 lines

Hi Stefan,

Good to see this patch. We needed to limit capabilities to reduce attack surface.

What tests have you run to make sure this current set of whitelisted capabilities is good enough.

Booting and light usage of Fedora 29 and running blogbench.

I would appreciate it if others could try it out with their tests/workloads.

Stefan

#7 lore raw
In reply to Stefan Hajnoczi
Quoted 4 lines · * Stefan Hajnoczi ([email protected]) wrote:

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Queued.

Quoted 10 lines

Stefan Hajnoczi (2):
virtiofsd: only retain file system capabilities
virtiofsd: drop all capabilities in the wait parent process

 tools/virtiofsd/passthrough_ll.c | 51 ++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
Signature · 1 line
2.25.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#8 lore raw
In reply to Stefan Hajnoczi
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Thanks
Vivek

Quoted 15 lines

Stefan Hajnoczi (2):
virtiofsd: only retain file system capabilities
virtiofsd: drop all capabilities in the wait parent process

 tools/virtiofsd/passthrough_ll.c | 51 ++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
Signature · 6 lines
2.25.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
#9 lore raw
Quoted 22 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

Quoted 7 lines

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?
b) If something nasty was to write junk into the trusted attributes, what would happen?
c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

Dave

Quoted 19 lines

Thanks
Vivek

Quoted 15 lines

Stefan Hajnoczi (2):
virtiofsd: only retain file system capabilities
virtiofsd: drop all capabilities in the wait parent process

 tools/virtiofsd/passthrough_ll.c | 51 ++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
Signature · 6 lines
2.25.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
Quoted 4 lines

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#10 lore raw
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 26 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

Quoted 7 lines
Quoted 3 lines

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Quoted 2 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

Thanks
Vivek

#11 lore raw
Quoted 9 lines · Vivek Goyal
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

Not just that but it needs CAP_SYS_ADMIN in the init namespace[1]. We have the same problem. Our virtiofs process has CAP_SYS_ADMIN in a user namespace and it cannot set any trusted or security xattrs. The security xattrs check is at least namespace aware so you only need CAP_SYS_ADMIN in the namespace that mounted the fs but that doesn't help us much.

We ended up working around it by prefixing "user.virtiofs." to the xattr name[2], which has its own problems but there was pretty much no chance that we would be able to give the fs device CAP_SYS_ADMIN in the init namespace.

[1]: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/+/5e857ce6eae7ca21b2055cca4885545e29228fe2/fs/xattr.c#116
[2]: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2243111

#12 lore raw
Quoted 18 lines · * Chirantan Ekbote ([email protected]) wrote:
Quoted 9 lines · Vivek Goyal
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

Quoted 6 lines

Not just that but it needs CAP_SYS_ADMIN in the init namespace[1]. We have the same problem. Our virtiofs process has CAP_SYS_ADMIN in a user namespace and it cannot set any trusted or security xattrs. The security xattrs check is at least namespace aware so you only need CAP_SYS_ADMIN in the namespace that mounted the fs but that doesn't help us much.

It would have been good if you'd mentioned that; it would have saved Vivek some confusion!

Quoted 4 lines

We ended up working around it by prefixing "user.virtiofs." to the xattr name[2], which has its own problems but there was pretty much no chance that we would be able to give the fs device CAP_SYS_ADMIN in the init namespace.

What problems did you hit with that? We should standardise the renaming so we make an on-disc format that's compatible.

Dave

Quoted 4 lines

[1]: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/+/5e857ce6eae7ca21b2055cca4885545e29228fe2/fs/xattr.c#116
[2]: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2243111

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#13 lore raw
Quoted 12 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Chirantan Ekbote ([email protected]) wrote:

We ended up working around it by prefixing "user.virtiofs." to the xattr name[2], which has its own problems but there was pretty much no chance that we would be able to give the fs device CAP_SYS_ADMIN in the init namespace.

What problems did you hit with that? We should standardise the renaming so we make an on-disc format that's compatible.

I guess what I meant by problems is that it made what was previously a simple and straightforward implementation into something more complex and added some limitations. For example, we now need to parse the result of the listxattr system call and strip out the prefix from any name in the list. It also means that we cannot allow the guest to directly set or remove any "user.virtiofs." xattr as this would allow an unprivileged process in the vm to modify an xattr that it wouldn't otherwise be allowed to modify. On top of being a somewhat arbitrary restriction this also means that you can't have stacked virtiofs instances as the lower instance would reject attempts by the upper one to set those xattrs. These limitations aren't really a problem for us but I can see how they might be a problem for others.

The change was also merged just yesterday so there may be other problems with it that haven't surfaced yet.

I didn't mention it before because I figured this was something that we brought upon ourselves as chrome os is a bit extreme about sandboxing. If we can come up with a standardized way to handle this I think we'll gladly switch the chrome os implementation to use it.

Chirantan

#14 lore raw
Quoted 11 lines · * Chirantan Ekbote ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Chirantan Ekbote ([email protected]) wrote:

We ended up working around it by prefixing "user.virtiofs." to the xattr name[2], which has its own problems but there was pretty much no chance that we would be able to give the fs device CAP_SYS_ADMIN in the init namespace.

Quoted 7 lines
Quoted 3 lines

What problems did you hit with that? We should standardise the renaming so we make an on-disc format that's compatible.

I guess what I meant by problems is that it made what was previously a simple and straightforward implementation into something more complex and added some limitations.

Yeh.

Quoted 10 lines

For example, we now need to parse the
result of the listxattr system call and strip out the prefix from any name in the list. It also means that we cannot allow the guest to directly set or remove any "user.virtiofs." xattr as this would allow an unprivileged process in the vm to modify an xattr that it wouldn't otherwise be allowed to modify. On top of being a somewhat arbitrary restriction this also means that you can't have stacked virtiofs instances as the lower instance would reject attempts by the upper one to set those xattrs. These limitations aren't really a problem for us but I can see how they might be a problem for others.

Isn't this a classic escaping problem?
Would it work if you prepended 'user.virtiofs.' onto any xattr that started with 'trusted' or 'user.virtiofs.' ?

Quoted 6 lines

The change was also merged just yesterday so there may be other problems with it that haven't surfaced yet.

I didn't mention it before because I figured this was something that we brought upon ourselves as chrome os is a bit extreme about sandboxing.

I think we're trying to be as extreme in virtiofsd, but it is causing us similar problems.

Quoted 2 lines

If we can come up with a standardized way to handle this I think we'll gladly switch the chrome os implementation to use it.

Great!

Dave

Quoted 3 lines

Chirantan

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#15 lore raw
In reply to Chirantan Ekbote
Quoted 25 lines · Chirantan Ekbote
Quoted 9 lines · Vivek Goyal
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

Not just that but it needs CAP_SYS_ADMIN in the init namespace[1]. We have the same problem. Our virtiofs process has CAP_SYS_ADMIN in a user namespace and it cannot set any trusted or security xattrs. The security xattrs check is at least namespace aware so you only need CAP_SYS_ADMIN in the namespace that mounted the fs but that doesn't help us much.

Quoted 4 lines

We ended up working around it by prefixing "user.virtiofs." to the xattr name[2], which has its own problems but there was pretty much no chance that we would be able to give the fs device CAP_SYS_ADMIN in the init namespace.

Chirantan,

So you ended up renaming all "trusted", "security" and "system" xattrs? Only "user" xattrs are complete passthrough?

IOW, security.selinux will be renamed to user.virtiofs.security.selinux on host?

Thanks
Vivek

#16 lore raw
Quoted 32 lines · Vivek Goyal
Quoted 25 lines · Chirantan Ekbote
Quoted 9 lines · Vivek Goyal
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

Not just that but it needs CAP_SYS_ADMIN in the init namespace[1]. We have the same problem. Our virtiofs process has CAP_SYS_ADMIN in a user namespace and it cannot set any trusted or security xattrs. The security xattrs check is at least namespace aware so you only need CAP_SYS_ADMIN in the namespace that mounted the fs but that doesn't help us much.

Quoted 4 lines

We ended up working around it by prefixing "user.virtiofs." to the xattr name[2], which has its own problems but there was pretty much no chance that we would be able to give the fs device CAP_SYS_ADMIN in the init namespace.

Quoted 5 lines

Chirantan,

So you ended up renaming all "trusted", "security" and "system" xattrs? Only "user" xattrs are complete passthrough?

No, we only rename "security" xattrs (except for selinux).

Quoted 4 lines

IOW, security.selinux will be renamed to user.virtiofs.security.selinux on host?

We don't relabel security.selinux because it only requires CAP_FOWNER in the process's user namespace and it also does its own MAC-based checks. Also we have some tools that label files beforehand so it seemed easier to leave them unchanged.

Chirantan

#17 lore raw

On Thu, Jun 25, 2020 at 12:19:39PM +0900, Chirantan Ekbote wrote:
[..]

Quoted 9 lines
Quoted 5 lines

Chirantan,

So you ended up renaming all "trusted", "security" and "system" xattrs? Only "user" xattrs are complete passthrough?

No, we only rename "security" xattrs (except for selinux).

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

IOW, security.selinux will be renamed to user.virtiofs.security.selinux on host?

We don't relabel security.selinux because it only requires CAP_FOWNER in the process's user namespace and it also does its own MAC-based checks. Also we have some tools that label files beforehand so it seemed easier to leave them unchanged.

If we rename selinux xattr also, then we can support selinux both in guest and host and they both can have their own independent policies.

Otherwise we either have to disable selinux on host (if we want to support it in guest) or somehow guest and how policies will have to know about each other and be able to work together (which will be hard for a generic use case).

Vivek

#18 lore raw
Quoted 16 lines · Vivek Goyal

On Thu, Jun 25, 2020 at 12:19:39PM +0900, Chirantan Ekbote wrote:
[..]

Quoted 13 lines
Quoted 5 lines

Chirantan,

So you ended up renaming all "trusted", "security" and "system" xattrs? Only "user" xattrs are complete passthrough?

No, we only rename "security" xattrs (except for selinux).

Quoted 4 lines

IOW, security.selinux will be renamed to user.virtiofs.security.selinux on host?

Quoted 8 lines
Quoted 4 lines

We don't relabel security.selinux because it only requires CAP_FOWNER in the process's user namespace and it also does its own MAC-based checks. Also we have some tools that label files beforehand so it seemed easier to leave them unchanged.

If we rename selinux xattr also, then we can support selinux both in guest and host and they both can have their own independent policies.

This works as long as you don't need to support setfscreatecon, which exists to guarantee that an inode is atomically created with the correct selinux context. It's kind of possible for files with O_TMPFILE + fsetxattr + linkat but there is no equivalent for directories. You're basically forced to create the directory and then set the xattr and while it's possible to prevent other threads in the server from seeing the unfinished directories with a rwlock or similar there is no protection against power loss. If the machine loses power after the directory is created but before the selinux xattr is set then that directory will have the wrong selinux context and the guest would need to run restorecon at boot to assign the correct label.

Quoted 4 lines

Otherwise we either have to disable selinux on host (if we want to support it in guest) or somehow guest and how policies will have to know about each other and be able to work together (which will be hard for a generic use case).

Yes, I agree this is hard to do for a generic case but unfortunately the more I understand how selinux works the less I feel that it works well with a passthrough style file system. As you said it either needs to be turned off on the host or the host and guest need to work together.

Chirantan

#19 lore raw
Quoted 26 lines · Chirantan Ekbote
Quoted 16 lines · Vivek Goyal

On Thu, Jun 25, 2020 at 12:19:39PM +0900, Chirantan Ekbote wrote:
[..]

Quoted 13 lines
Quoted 5 lines

Chirantan,

So you ended up renaming all "trusted", "security" and "system" xattrs? Only "user" xattrs are complete passthrough?

No, we only rename "security" xattrs (except for selinux).

Quoted 4 lines

IOW, security.selinux will be renamed to user.virtiofs.security.selinux on host?

Quoted 8 lines
Quoted 4 lines

We don't relabel security.selinux because it only requires CAP_FOWNER in the process's user namespace and it also does its own MAC-based checks. Also we have some tools that label files beforehand so it seemed easier to leave them unchanged.

If we rename selinux xattr also, then we can support selinux both in guest and host and they both can have their own independent policies.

Quoted 11 lines

This works as long as you don't need to support setfscreatecon, which exists to guarantee that an inode is atomically created with the correct selinux context. It's kind of possible for files with O_TMPFILE + fsetxattr + linkat but there is no equivalent for directories. You're basically forced to create the directory and then set the xattr and while it's possible to prevent other threads in the server from seeing the unfinished directories with a rwlock or similar there is no protection against power loss. If the machine loses power after the directory is created but before the selinux xattr is set then that directory will have the wrong selinux context and the guest would need to run restorecon at boot to assign the correct label.

Overlayfs has this notion of work directory where they create directory (and file if O_TMPFILE is not supported), and then rename it to correct place. I am wondering if we could do something similar. If server is given a work directory where.

A. Create new directory
B. set selinux lables
C. rename directory

That way if machine crashes after step A or step B, that directory never becomes visible to guest and no relabeling is required.

Thanks
Vivek

#20 lore raw
In reply to Chirantan Ekbote
Quoted 36 lines · Chirantan Ekbote
Quoted 12 lines · Vivek Goyal

On Thu, Jun 25, 2020 at 12:19:39PM +0900, Chirantan Ekbote wrote:
[..]

Quoted 10 lines
Quoted 5 lines

Chirantan,

So you ended up renaming all "trusted", "security" and "system" xattrs? Only "user" xattrs are complete passthrough?

No, we only rename "security" xattrs (except for selinux).

Quoted 3 lines

IOW, security.selinux will be renamed to user.virtiofs.security.selinux on host?

Quoted 7 lines
Quoted 4 lines

We don't relabel security.selinux because it only requires CAP_FOWNER in the process's user namespace and it also does its own MAC-based checks. Also we have some tools that label files beforehand so it seemed easier to leave them unchanged.

If we rename selinux xattr also, then we can support selinux both in guest and host and they both can have their own independent policies.

This works as long as you don't need to support setfscreatecon, which exists to guarantee that an inode is atomically created with the correct selinux context. It's kind of possible for files with O_TMPFILE + fsetxattr + linkat but there is no equivalent for directories. You're basically forced to create the directory and then set the xattr and while it's possible to prevent other threads in the server from seeing the unfinished directories with a rwlock or similar there is no protection against power loss. If the machine loses power after the directory is created but before the selinux xattr is set then that directory will have the wrong selinux context and the guest would need to run restorecon at boot to assign the correct label.

Quoted 4 lines

Otherwise we either have to disable selinux on host (if we want to support it in guest) or somehow guest and how policies will have to know about each other and be able to work together (which will be hard for a generic use case).

Quoted 5 lines

Yes, I agree this is hard to do for a generic case but unfortunately the more I understand how selinux works the less I feel that it works well with a passthrough style file system. As you said it either needs to be turned off on the host or the host and guest need to work together.

Correct both kernels need to understand the labels, or one of the kernels has to have SELinux disabled.

That is the bottom line.  Same issue exists for labeled NFS so I don't see this as a problem.

Quoted 6 lines

Chirantan

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

#21 lore raw

On Mon, Jul 13, 2020 at 05:39:05PM -0400, Daniel Walsh wrote:

[..]

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

Otherwise we either have to disable selinux on host (if we want to support it in guest) or somehow guest and how policies will have to know about each other and be able to work together (which will be hard for a generic use case).

Yes, I agree this is hard to do for a generic case but unfortunately the more I understand how selinux works the less I feel that it works well with a passthrough style file system. As you said it either needs to be turned off on the host or the host and guest need to work together.

Quoted 5 lines

Correct both kernels need to understand the labels, or one of the kernels has to have SELinux disabled.

That is the bottom line.  Same issue exists for labeled NFS so I don't see this as a problem.

Dan,

So what does labeled NFS do. Server disables SELinux so that it can be enabled on client?

Thanks
Vivek

#22 lore raw
Quoted 17 lines · Vivek Goyal

On Mon, Jul 13, 2020 at 05:39:05PM -0400, Daniel Walsh wrote:

[..]

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

Otherwise we either have to disable selinux on host (if we want to support it in guest) or somehow guest and how policies will have to know about each other and be able to work together (which will be hard for a generic use case).

Yes, I agree this is hard to do for a generic case but unfortunately the more I understand how selinux works the less I feel that it works well with a passthrough style file system. As you said it either needs to be turned off on the host or the host and guest need to work together.

Quoted 5 lines

Correct both kernels need to understand the labels, or one of the kernels has to have SELinux disabled.

That is the bottom line.  Same issue exists for labeled NFS so I don't see this as a problem.

Quoted 7 lines

Dan,

So what does labeled NFS do. Server disables SELinux so that it can be enabled on client?

Thanks
Vivek

Labeled NFS does all three

Server Disabled, Client Enabled - Server does not care about the labels

Sever Enabled, Client Disabled  - Server selects the labels, Client ignores them

Sever Enabled, Client Enabled  - All labels are seen on the client and server and must be understood by the Server, if the client does not understand the label, it will be marked as unlabeled_t on the client, if the Client tries to create a label that is not understood on the Server, the Server will block the creation.

The most usual case of labeled NFS is the Client and Server use the same SELinux policy. Or one or the other is disabled.  The cases where each is enabled with different policy is rare, in my experience.

#23 lore raw
In reply to Vivek Goyal
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do? Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

Quoted 4 lines
Quoted 2 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");

but I don't know what index and metacopy are.

Quoted 4 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

Dave

Quoted 2 lines

Thanks
Vivek

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#24 lore raw
Quoted 75 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do? Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

Quoted 4 lines
Quoted 2 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");

but I don't know what index and metacopy are.

Quoted 4 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

Quoted 4 lines

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

The CAP_SYS_ADMIN requirement for 'trusted.' xattrs is simply a useful mechanism for applications to control access. The host kernel doesn' tuse this namespace itself. Linux has four namespaces for xattrs:

 -  user - for userspace apps. accessible based on read/write permissions
 -  trusted - for userspace apps. accessible by CAP_SYS_ADMIN processes only
 -  system - for kernel only. used by ACLs
 -  security - for kernel only. used by SELinux

The use case for "trusted" xattrs is thus where a privileged management application or service wants to store metadata against the file, but also needs to grant an unprivileged process access to write to this file while not allowing that unprivileged process the ability to change the metadata. This is mentioned in the man page:

[man xattr(7)]
Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐ cesses that have the CAP_SYS_ADMIN capability. Attributes in this class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

User extended attributes
User extended attributes may be assigned to files and directories for storing arbitrary additional information such as the mime type, character set or encoding of a file. The access permissions for user attributes are defined by the file permission bits: read per‐ mission is required to retrieve the attribute value, and writer per‐ mission is required to change it.
[/man]

Libvirtd uses the "trusted." xattr namespace to record information against disk images for QEMU, because we need to grant QEMU access to read/write the disk iamges, but don't want QEMU to be able to alter our xattrs.

It is unfortunate that this namespace is tied to the CAP_SYS_ADMIN cap. It really ought to have had its own dedicated capability :-( Such is life with anything that uses CAP_SYS_ADMIN...

With this in mind we really should have both trusted. & user. xattrs allowed to the guest by default.

Conversely, we'll need to block usage of the security. and system. namespaces.

Unfortuntely this implies allowing CAP_SYS_ADMIN by default. We could have a "--trusted-xattrs=on|off" flag to allow people to run in a more locked down state if they knowingly want to block trusted xattrs.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#25 lore raw
Quoted 96 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 75 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do? Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

Quoted 4 lines
Quoted 2 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");

but I don't know what index and metacopy are.

Quoted 4 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

Quoted 4 lines

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

The CAP_SYS_ADMIN requirement for 'trusted.' xattrs is simply a useful mechanism for applications to control access. The host kernel doesn' tuse this namespace itself. Linux has four namespaces for xattrs:

 -  user - for userspace apps. accessible based on read/write permissions
 -  trusted - for userspace apps. accessible by CAP_SYS_ADMIN processes only
 -  system - for kernel only. used by ACLs
 -  security - for kernel only. used by SELinux

The use case for "trusted" xattrs is thus where a privileged management application or service wants to store metadata against the file, but also needs to grant an unprivileged process access to write to this file while not allowing that unprivileged process the ability to change the metadata. This is mentioned in the man page:

Quoted 7 lines

[man xattr(7)]
Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐ cesses that have the CAP_SYS_ADMIN capability. Attributes in this class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

overlayfs inside the kernel is using trusted though which is the case Vivek ran into.

Quoted 20 lines

User extended attributes
User extended attributes may be assigned to files and directories for storing arbitrary additional information such as the mime type, character set or encoding of a file. The access permissions for user attributes are defined by the file permission bits: read per‐ mission is required to retrieve the attribute value, and writer per‐ mission is required to change it.
[/man]

Libvirtd uses the "trusted." xattr namespace to record information against disk images for QEMU, because we need to grant QEMU access to read/write the disk iamges, but don't want QEMU to be able to alter our xattrs.

It is unfortunate that this namespace is tied to the CAP_SYS_ADMIN cap. It really ought to have had its own dedicated capability :-( Such is life with anything that uses CAP_SYS_ADMIN...

With this in mind we really should have both trusted. & user. xattrs allowed to the guest by default.

Quoted 6 lines

Conversely, we'll need to block usage of the security. and system. namespaces.

Unfortuntely this implies allowing CAP_SYS_ADMIN by default. We could have a "--trusted-xattrs=on|off" flag to allow people to run in a more locked down state if they knowingly want to block trusted xattrs.

Does the trick described by Chirantan work for you, where the clients 'trusted' view is different from the host?

Dave

Quoted 6 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#26 lore raw
Quoted 105 lines · Dr. David Alan Gilbert
Quoted 96 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 75 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do? Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

Quoted 4 lines
Quoted 2 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");

but I don't know what index and metacopy are.

Quoted 4 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

Quoted 4 lines

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

The CAP_SYS_ADMIN requirement for 'trusted.' xattrs is simply a useful mechanism for applications to control access. The host kernel doesn' tuse this namespace itself. Linux has four namespaces for xattrs:

 -  user - for userspace apps. accessible based on read/write permissions
 -  trusted - for userspace apps. accessible by CAP_SYS_ADMIN processes only
 -  system - for kernel only. used by ACLs
 -  security - for kernel only. used by SELinux

The use case for "trusted" xattrs is thus where a privileged management application or service wants to store metadata against the file, but also needs to grant an unprivileged process access to write to this file while not allowing that unprivileged process the ability to change the metadata. This is mentioned in the man page:

Quoted 7 lines

[man xattr(7)]
Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐ cesses that have the CAP_SYS_ADMIN capability. Attributes in this class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

Quoted 2 lines

overlayfs inside the kernel is using trusted though which is the case Vivek ran into.

IIUC, this is need on the lower-layer FS. If you're running overlayfs on the host, then virtiofsd is exposing the upper layer FS, so guest use of "trusted." on the upper layer won't clash with overlayfs in the host.

If you're running overlayfs in the guest, then virtiofs is the lower layer so needs to support the "trusted." on the FS its exposing. Again this should not clash with anything on the host.

Quoted 23 lines
Quoted 23 lines

User extended attributes
User extended attributes may be assigned to files and directories for storing arbitrary additional information such as the mime type, character set or encoding of a file. The access permissions for user attributes are defined by the file permission bits: read per‐ mission is required to retrieve the attribute value, and writer per‐ mission is required to change it.
[/man]

Libvirtd uses the "trusted." xattr namespace to record information against disk images for QEMU, because we need to grant QEMU access to read/write the disk iamges, but don't want QEMU to be able to alter our xattrs.

It is unfortunate that this namespace is tied to the CAP_SYS_ADMIN cap. It really ought to have had its own dedicated capability :-( Such is life with anything that uses CAP_SYS_ADMIN...

With this in mind we really should have both trusted. & user. xattrs allowed to the guest by default.

Conversely, we'll need to block usage of the security. and system. namespaces.

Quoted 6 lines
Quoted 3 lines

Unfortuntely this implies allowing CAP_SYS_ADMIN by default. We could have a "--trusted-xattrs=on|off" flag to allow people to run in a more locked down state if they knowingly want to block trusted xattrs.

Does the trick described by Chirantan work for you, where the clients 'trusted' view is different from the host?

That works as long as you don't require any interoperability with processes that have setup the exposed file tree on the host, or might need to concurrently access it.

This probably suggests a need various modes of exposing xattrs

 - "passthrough"  - run CAP_SYS_ADMIN and pass directly
 - "mapped"       - no capabilities, rewrite everything into user. namespace
 - "disable"      - don't allow xattr

As mentioned in the other thread though, CAP_SYS_ADMIN needs to be present in the initial namespace, so to support "passthrough" we'd have to NOT use user namespace too. Something to be aware of when we accept the user namespace patches.

Such "mapping" of attributes could also be used if virtiofs was running as an unprivileged user account, to expose a full range of file owner, group and permissions to the guest, while having everything be owned
by the normal user on the host. QEMU's current 9p filesystem has
this kind of mapping support for owner/permissions, so guest view can be distinct from the host view.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#27 lore raw
Quoted 151 lines · Daniel P. Berrangé
Quoted 105 lines · Dr. David Alan Gilbert
Quoted 96 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 75 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do? Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

Quoted 4 lines
Quoted 2 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");

but I don't know what index and metacopy are.

Quoted 4 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

Quoted 4 lines

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

The CAP_SYS_ADMIN requirement for 'trusted.' xattrs is simply a useful mechanism for applications to control access. The host kernel doesn' tuse this namespace itself. Linux has four namespaces for xattrs:

 -  user - for userspace apps. accessible based on read/write permissions
 -  trusted - for userspace apps. accessible by CAP_SYS_ADMIN processes only
 -  system - for kernel only. used by ACLs
 -  security - for kernel only. used by SELinux

The use case for "trusted" xattrs is thus where a privileged management application or service wants to store metadata against the file, but also needs to grant an unprivileged process access to write to this file while not allowing that unprivileged process the ability to change the metadata. This is mentioned in the man page:

Quoted 7 lines

[man xattr(7)]
Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐ cesses that have the CAP_SYS_ADMIN capability. Attributes in this class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

Quoted 2 lines

overlayfs inside the kernel is using trusted though which is the case Vivek ran into.

IIUC, this is need on the lower-layer FS. If you're running overlayfs on the host, then virtiofsd is exposing the upper layer FS, so guest use of "trusted." on the upper layer won't clash with overlayfs in the host.

If you're running overlayfs in the guest, then virtiofs is the lower layer so needs to support the "trusted." on the FS its exposing. Again this should not clash with anything on the host.

Quoted 23 lines
Quoted 23 lines

User extended attributes
User extended attributes may be assigned to files and directories for storing arbitrary additional information such as the mime type, character set or encoding of a file. The access permissions for user attributes are defined by the file permission bits: read per‐ mission is required to retrieve the attribute value, and writer per‐ mission is required to change it.
[/man]

Libvirtd uses the "trusted." xattr namespace to record information against disk images for QEMU, because we need to grant QEMU access to read/write the disk iamges, but don't want QEMU to be able to alter our xattrs.

It is unfortunate that this namespace is tied to the CAP_SYS_ADMIN cap. It really ought to have had its own dedicated capability :-( Such is life with anything that uses CAP_SYS_ADMIN...

With this in mind we really should have both trusted. & user. xattrs allowed to the guest by default.

Conversely, we'll need to block usage of the security. and system. namespaces.

Quoted 6 lines
Quoted 3 lines

Unfortuntely this implies allowing CAP_SYS_ADMIN by default. We could have a "--trusted-xattrs=on|off" flag to allow people to run in a more locked down state if they knowingly want to block trusted xattrs.

Does the trick described by Chirantan work for you, where the clients 'trusted' view is different from the host?

That works as long as you don't require any interoperability with processes that have setup the exposed file tree on the host, or might need to concurrently access it.

Quoted 6 lines

This probably suggests a need various modes of exposing xattrs

 - "passthrough"  - run CAP_SYS_ADMIN and pass directly
 - "mapped"       - no capabilities, rewrite everything into user. namespace
 - "disable"      - don't allow xattr

We already do "passthrough" (this is our default) and "disable" (-o no_xattr). We probably need to implement this "mapped" mode and that will allow us to run without CAP_SYS_ADMIN in init_user_ns.

Quoted 6 lines

As mentioned in the other thread though, CAP_SYS_ADMIN needs to be present in the initial namespace, so to support "passthrough" we'd have to NOT use user namespace too. Something to be aware of when we accept the user namespace patches.

Quoted 6 lines

Such "mapping" of attributes could also be used if virtiofs was running as an unprivileged user account, to expose a full range of file owner, group and permissions to the guest, while having everything be owned
by the normal user on the host. QEMU's current 9p filesystem has
this kind of mapping support for owner/permissions, so guest view can be distinct from the host view.

If we figure out a way to run in a user namespace, then we don't necessarily require this mapping functionality, right? User namespace will automatically provide that mapping. We can just run as root of that user namespace.

When 9p wrote this remapping method, we probably did not have user namespace as popular as they are now. So we probably should implement remapping of trusted xattr and then run in a user namesapce.

And complete passthrough mode will have to run in init user namespace and this will be least securre mode (from host security point of view).

Thanks
Vivek

#28 lore raw
In reply to Daniel P. Berrangé

On Fri, Jun 19, 2020 at 12:39:14PM +0100, Daniel P. Berrangé wrote:
[..]

Quoted 40 lines

The CAP_SYS_ADMIN requirement for 'trusted.' xattrs is simply a useful mechanism for applications to control access. The host kernel doesn' tuse this namespace itself. Linux has four namespaces for xattrs:

 -  user - for userspace apps. accessible based on read/write permissions
 -  trusted - for userspace apps. accessible by CAP_SYS_ADMIN processes only
 -  system - for kernel only. used by ACLs
 -  security - for kernel only. used by SELinux

The use case for "trusted" xattrs is thus where a privileged management application or service wants to store metadata against the file, but also needs to grant an unprivileged process access to write to this file while not allowing that unprivileged process the ability to change the metadata. This is mentioned in the man page:

[man xattr(7)]
Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐ cesses that have the CAP_SYS_ADMIN capability. Attributes in this class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

User extended attributes
User extended attributes may be assigned to files and directories for storing arbitrary additional information such as the mime type, character set or encoding of a file. The access permissions for user attributes are defined by the file permission bits: read per‐ mission is required to retrieve the attribute value, and writer per‐ mission is required to change it.
[/man]

Libvirtd uses the "trusted." xattr namespace to record information against disk images for QEMU, because we need to grant QEMU access to read/write the disk iamges, but don't want QEMU to be able to alter our xattrs.

It is unfortunate that this namespace is tied to the CAP_SYS_ADMIN cap. It really ought to have had its own dedicated capability :-( Such is life with anything that uses CAP_SYS_ADMIN...

Quoted 5 lines

With this in mind we really should have both trusted. & user. xattrs allowed to the guest by default.

Conversely, we'll need to block usage of the security. and system. namespaces.

I am wondering can we block usage of "system" and "security"? What about guest setting acls over virtiofs files. These will have to go through and that means we need to allow system xattrs.

Similarly setting file capabilities inside should trigger setxattr(security.capability) and that means we need to allow security xattr as well.

Thanks
Vivek

#29 lore raw
Quoted 48 lines · * Vivek Goyal ([email protected]) wrote:

On Fri, Jun 19, 2020 at 12:39:14PM +0100, Daniel P. Berrangé wrote:
[..]

Quoted 40 lines

The CAP_SYS_ADMIN requirement for 'trusted.' xattrs is simply a useful mechanism for applications to control access. The host kernel doesn' tuse this namespace itself. Linux has four namespaces for xattrs:

 -  user - for userspace apps. accessible based on read/write permissions
 -  trusted - for userspace apps. accessible by CAP_SYS_ADMIN processes only
 -  system - for kernel only. used by ACLs
 -  security - for kernel only. used by SELinux

The use case for "trusted" xattrs is thus where a privileged management application or service wants to store metadata against the file, but also needs to grant an unprivileged process access to write to this file while not allowing that unprivileged process the ability to change the metadata. This is mentioned in the man page:

[man xattr(7)]
Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐ cesses that have the CAP_SYS_ADMIN capability. Attributes in this class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

User extended attributes
User extended attributes may be assigned to files and directories for storing arbitrary additional information such as the mime type, character set or encoding of a file. The access permissions for user attributes are defined by the file permission bits: read per‐ mission is required to retrieve the attribute value, and writer per‐ mission is required to change it.
[/man]

Libvirtd uses the "trusted." xattr namespace to record information against disk images for QEMU, because we need to grant QEMU access to read/write the disk iamges, but don't want QEMU to be able to alter our xattrs.

It is unfortunate that this namespace is tied to the CAP_SYS_ADMIN cap. It really ought to have had its own dedicated capability :-( Such is life with anything that uses CAP_SYS_ADMIN...

Quoted 5 lines

With this in mind we really should have both trusted. & user. xattrs allowed to the guest by default.

Conversely, we'll need to block usage of the security. and system. namespaces.

Quoted 7 lines

I am wondering can we block usage of "system" and "security"? What about guest setting acls over virtiofs files. These will have to go through and that means we need to allow system xattrs.

Similarly setting file capabilities inside should trigger setxattr(security.capability) and that means we need to allow security xattr as well.

Yep, we see that when people install Fedora packages, when rpm unpackgs /usr/bin/newgidmap which has:

$ getfattr -d '--match=.*' /usr/bin/newgidmap 
getfattr: Removing leading '/' from absolute path names
# file: usr/bin/newgidmap
security.capability=0sAQAAAkAAAAAAAAAAAAAAAAAAAAA=
security.selinux="system_u:object_r:bin_t:s0"

Dave

Quoted 2 lines

Thanks
Vivek

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#30 lore raw
In reply to Dr. David Alan Gilbert
Quoted 33 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do?

It contains path information which is used for lookup into lower layer.

Quoted 1 line

Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Overlay is storing its metadata in trusted.* xattrs. If a user modifies metadata, then various kind of bad things can happen. I think one can do some kind of checks on metadata to make sure it does not crash atleast.

And that's true for any filesystem. Isn't. If user manages to modify metadata outside of filesystem, then lot of bad things can happen. I thought that's the reason that people are not comfortable with the idea of allowing mounting filesystem from inside user namespace because it makes it easy to mount a hand crafted filesystem.

Anyway, I think overlayfs is just one use case of trusted xattr. Even if overlayfs moves away from trusted xattr, what about other users. We need to have a story around how will we support trusted xattrs safely.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

I thought qemu and kvm are the one who should ensure we should not be able to break out of sandbox. Kernel implementation could be as buggy as it wanted to be. We are working with this security model that kernel is completely untrusted.

Quoted 17 lines
Quoted 5 lines
Quoted 3 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");
Quoted 1 line

but I don't know what index and metacopy are.

They enable certain features in overlayfs. In fact, we fall back to lesser capability on if we are running on ext4/xfs. For virtiofs, we deny the mount completely.

        /*
         * We allowed sub-optimal upper fs configuration and don't want to break
         * users over kernel upgrade, but we never allowed remote upper fs, so
         * we can enforce strict requirements for remote upper fs.
         */
        if (ovl_dentry_remote(ofs->workdir) &&
            (!d_type || !rename_whiteout || ofs->noxattr)) {
                pr_err("upper fs missing required features.\n");
                err = -EINVAL;
                goto out;
        }
Quoted 10 lines
Quoted 5 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

Agreed that we need to look into whether having CAP_SYS_ADMIN allow virtiofsd to break out of jail.

May be we need to provide that remapping trusted xattr feature so that we don't have to have CAP_SYS_ADMIN in init_user_ns and can provide this emulation even when running in user namespace.

Vivek

#31 lore raw
Quoted 71 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 33 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 31 lines · Dr. David Alan Gilbert
Quoted 29 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

overlayfs sets "trusted.overlay.*" and xattrs in trusted domain need CAP_SYS_ADMIN.

man xattr says.

Trusted extended attributes
Trusted extended attributes are visible and accessible only to pro‐
cesses that have the CAP_SYS_ADMIN capability. Attributes in this
class are used to implement mechanisms in user space (i.e., outside the kernel) which keep information in extended attributes to which ordinary processes should not have access.

There is a chance that overlay moves away from trusted xattr in future. But for now we need to make it work. This is an important use case for kata docker in docker build.

May be we can add an option to virtiofsd say "--add-cap <capability>" and ask user to pass in "--add-cap cap_sys_admin" if they need to run daemon with this capaibility.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I'll admit I don't like the idea of giving it cap_sys_admin. Can you explain:
a) What overlayfs uses trusted for?

overlayfs stores bunch of metadata and uses "trusted" xattrs for it.

Tell me more about this metadata.
Taking a juicy looking one, what does OVL_XATTR_REDIRECT do?

It contains path information which is used for lookup into lower layer.

Quoted 1 line

Or what happens if I was to write random numbers into OVL_XATTR_NLINK?

Overlay is storing its metadata in trusted.* xattrs. If a user modifies metadata, then various kind of bad things can happen. I think one can do some kind of checks on metadata to make sure it does not crash atleast.

And that's true for any filesystem. Isn't. If user manages to modify metadata outside of filesystem, then lot of bad things can happen. I thought that's the reason that people are not comfortable with the idea of allowing mounting filesystem from inside user namespace because it makes it easy to mount a hand crafted filesystem.

Anyway, I think overlayfs is just one use case of trusted xattr. Even if overlayfs moves away from trusted xattr, what about other users. We need to have a story around how will we support trusted xattrs safely.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Quoted 7 lines
Quoted 2 lines

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

I thought qemu and kvm are the one who should ensure we should not be able to break out of sandbox. Kernel implementation could be as buggy as it wanted to be. We are working with this security model that kernel is completely untrusted.

But with virtiofs we allow the guest to do a lot of filesystem operations on the host. It's the virtiofsd that has to ensure that these are safe and contained within the fs it's exposed; the qemu/kvm can't protect us from that.

That's why we sandbox the virtiofsd like we do - if we allow a priviliged guest to perform calls to an unconstrained virtiofsd it would be able to escape. That's what I want to check.

Dave

Quoted 46 lines
Quoted 17 lines
Quoted 5 lines
Quoted 3 lines

c) I see overlayfs has a fallback check if xattr isn't supported at
all - what is the consequence?

It falls back to I think read only mode.

It looks like the fallback is more subtle to me:

        /*
         * Check if upper/work fs supports trusted.overlay.* xattr
         */
        err = ovl_do_setxattr(ofs->workdir, OVL_XATTR_OPAQUE, "0", 1, 0);
        if (err) {
                ofs->noxattr = true;
                ofs->config.index = false;
                ofs->config.metacopy = false;
                pr_warn("upper fs does not support xattr, falling back to index=off and metacopy=off.\n");
Quoted 1 line

but I don't know what index and metacopy are.

They enable certain features in overlayfs. In fact, we fall back to lesser capability on if we are running on ext4/xfs. For virtiofs, we deny the mount completely.

        /*
         * We allowed sub-optimal upper fs configuration and don't want to break
         * users over kernel upgrade, but we never allowed remote upper fs, so
         * we can enforce strict requirements for remote upper fs.
         */
        if (ovl_dentry_remote(ofs->workdir) &&
            (!d_type || !rename_whiteout || ofs->noxattr)) {
                pr_err("upper fs missing required features.\n");
                err = -EINVAL;
                goto out;
        }
Quoted 10 lines
Quoted 5 lines

For a moment forget about overlayfs. Say a user process in guest with CAP_SYS_ADMIN is writing trusted.foo. Should that succeed? Is a passthrough filesystem, so it should go through. But currently it wont.

As long as any effects of what it writes are contained to the area of the filesystem exposed to the guest, yes - however it worries me what the consequences of broken trusted metadata is. If it's delicate enough that it's guarded by CAP_SYS_ADMIN someone must have worried about it.

Quoted 8 lines

Agreed that we need to look into whether having CAP_SYS_ADMIN allow virtiofsd to break out of jail.

May be we need to provide that remapping trusted xattr feature so that we don't have to have CAP_SYS_ADMIN in init_user_ns and can provide this emulation even when running in user namespace.

Vivek

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#32 lore raw

On Fri, Jun 19, 2020 at 05:16:48PM +0100, Dr. David Alan Gilbert wrote:

[..]

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Quoted 7 lines
Quoted 2 lines

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

I thought qemu and kvm are the one who should ensure we should not be able to break out of sandbox. Kernel implementation could be as buggy as it wanted to be. We are working with this security model that kernel is completely untrusted.

Quoted 4 lines

But with virtiofs we allow the guest to do a lot of filesystem operations on the host. It's the virtiofsd that has to ensure that these are safe and contained within the fs it's exposed; the qemu/kvm can't protect us from that.

Fair enough. I should have added virtiofsd to list. Its an attack vector and is of concern.

Quoted 4 lines

That's why we sandbox the virtiofsd like we do - if we allow a priviliged guest to perform calls to an unconstrained virtiofsd it would be able to escape. That's what I want to check.

Sure. So does giving CAP_SYS_ADMIN to virtiofsd allow daemon to escape the jail. If it does we need to implement what crossvm folks did, remapping of trusted xattr. That will also allow us to run inside user namespace and still be able to support trusted xattr emulation for guest.

Vivek

#33 lore raw
Quoted 26 lines · * Vivek Goyal ([email protected]) wrote:

On Fri, Jun 19, 2020 at 05:16:48PM +0100, Dr. David Alan Gilbert wrote:

[..]

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

b) If something nasty was to write junk into the trusted attributes, what would happen?

This directory is owned by guest. So it should be able to write anything it wants, as long as process in guest has CAP_SYS_ADMIN, right?

Quoted 7 lines
Quoted 2 lines

Well, we shouldn't be able to break/crash/escape into the host; how much does overlayfs validate trusted.* it uses?

I thought qemu and kvm are the one who should ensure we should not be able to break out of sandbox. Kernel implementation could be as buggy as it wanted to be. We are working with this security model that kernel is completely untrusted.

Quoted 4 lines

But with virtiofs we allow the guest to do a lot of filesystem operations on the host. It's the virtiofsd that has to ensure that these are safe and contained within the fs it's exposed; the qemu/kvm can't protect us from that.

Fair enough. I should have added virtiofsd to list. Its an attack vector and is of concern.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

That's why we sandbox the virtiofsd like we do - if we allow a priviliged guest to perform calls to an unconstrained virtiofsd it would be able to escape. That's what I want to check.

Sure. So does giving CAP_SYS_ADMIN to virtiofsd allow daemon to escape the jail.

So that's *my* question - what bad things can someone do by setting attributes (trusted/system/security) - it's fine if they break they screwup the security inside the container, because they'd need to be CAP_SYS_ADMIN inside the container to do it - as long as they can't break the host. So what happens if someone starts doing bad things to trusted.* attributes on an overlayfs - no other fs uses them as far as I know.

Quoted 4 lines

If it does we need to implement what crossvm folks did, remapping of trusted xattr. That will also allow us to run inside user namespace and still be able to support trusted xattr emulation for guest.

I think we need to do that anyway, it's just going to take a while to figure out.

Dave

Quoted 2 lines

Vivek

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#34 lore raw
In reply to Vivek Goyal
Quoted 10 lines · Vivek Goyal
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

How so? Virtiofs isn't mounting overlayfs, is it? Only the mounter requires CAP_SYS_ADMIN, not the accessor.

Thanks,
Miklos

#35 lore raw
Quoted 8 lines · Miklos Szeredi
Quoted 7 lines · Vivek Goyal
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 6 lines
Quoted 3 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

How so? Virtiofs isn't mounting overlayfs, is it? Only the mounter requires CAP_SYS_ADMIN, not the accessor.

virtiofsd needs CAP_SYS_ADMIN, otherwise fsetxattr(trusted.overlay.opaque) fails in lo_setxattr().

This is triggered when we mount overlayfs on top of virtiofs and overlayfs tries to set OVL_XATTR_OPAQUE on upper to check if trusted xattrs are supported or not.

Thanks
Vivek

#36 lore raw
Quoted 17 lines · Vivek Goyal
Quoted 8 lines · Miklos Szeredi
Quoted 7 lines · Vivek Goyal
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 6 lines
Quoted 3 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

How so? Virtiofs isn't mounting overlayfs, is it? Only the mounter requires CAP_SYS_ADMIN, not the accessor.

Quoted 6 lines

virtiofsd needs CAP_SYS_ADMIN, otherwise fsetxattr(trusted.overlay.opaque) fails in lo_setxattr().

This is triggered when we mount overlayfs on top of virtiofs and overlayfs tries to set OVL_XATTR_OPAQUE on upper to check if trusted xattrs are supported or not.

Ah, right.

Plan is to use "user.*" xattr for unprivileged overlay. This would be a good way to eliminate this attack surface in the overlay on virtiofs case as well.

Other ways to minimize risk is to separate operations requiring CAP_SYS_ADMIN into a separate process, preferably a separate executable, that communicates with virtiofsd using a pipe and contains the minimum amount of code.

Thanks,
Miklos

#37 lore raw
Quoted 25 lines · Miklos Szeredi
Quoted 17 lines · Vivek Goyal
Quoted 8 lines · Miklos Szeredi
Quoted 7 lines · Vivek Goyal
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 6 lines
Quoted 3 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

How so? Virtiofs isn't mounting overlayfs, is it? Only the mounter requires CAP_SYS_ADMIN, not the accessor.

Quoted 6 lines

virtiofsd needs CAP_SYS_ADMIN, otherwise fsetxattr(trusted.overlay.opaque) fails in lo_setxattr().

This is triggered when we mount overlayfs on top of virtiofs and overlayfs tries to set OVL_XATTR_OPAQUE on upper to check if trusted xattrs are supported or not.

Quoted 5 lines

Ah, right.

Plan is to use "user.*" xattr for unprivileged overlay. This would be a good way to eliminate this attack surface in the overlay on virtiofs case as well.

So unpriviliged overlay is one which is mounted from inside a user namespace. But in this case we might be mounting it from init_user_ns of guest. So from overlayfs perspective this will still be treated as priviliged overlay instance and it will use trusted xattrs, IIUC?

Thanks
Vivek

Quoted 10 lines

Other ways to minimize risk is to separate operations requiring CAP_SYS_ADMIN into a separate process, preferably a separate executable, that communicates with virtiofsd using a pipe and contains the minimum amount of code.

Thanks,
Miklos

#38 lore raw
In reply to Miklos Szeredi
Quoted 8 lines · Miklos Szeredi
Quoted 7 lines · Vivek Goyal
Quoted 4 lines · Stefan Hajnoczi

virtiofsd doesn't need of all Linux capabilities(7) available to root. Keep a whitelisted set of capabilities that we require. This improves security in case virtiofsd is compromised by making it hard for an attacker to gain further access to the system.

Quoted 6 lines
Quoted 3 lines

Hi Stefan,

I just noticed that this patch set breaks overlayfs on top of virtiofs.

How so? Virtiofs isn't mounting overlayfs, is it? Only the mounter requires CAP_SYS_ADMIN, not the accessor.

Are you thinking of virtiofs on top of overlayfs? I was referring to mounting overlayfs on top of virtiofs inside VM.

Thanks
Vivek