Skip to contentSkip to conversation

hostmem: don't use mbind() if host-nodes is epmty

PATCH Applied Igor Mammedov 5 participants 9 messages Apr 30, 2020 – May 11, 2020 Arm STM32

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hostmem: don't use mbind() if host-nodes is epmty

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 backends/hostmem.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
backends/hostmem.c +4 -2
index 327f9eebc3..0efd7b7bd6 100644
@@ -383,8 +383,10 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)
         assert(sizeof(backend->host_nodes) >=
                BITS_TO_LONGS(MAX_NODES + 1) * sizeof(unsigned long));
         assert(maxnode <= MAX_NODES);
-        if (mbind(ptr, sz, backend->policy,
-                  maxnode ? backend->host_nodes : NULL, maxnode + 1, flags)) {
+
+        if (maxnode &&
+            mbind(ptr, sz, backend->policy, backend->host_nodes, maxnode + 1,
+                  flags)) {
             if (backend->policy != MPOL_DEFAULT || errno != ENOSYS) {
                 error_setg_errno(errp, errno,
                                  "cannot bind memory to host NUMA nodes");
Signature · 1 line
2.18.1
#1 lore raw

Typo "empty" in patch subject.

Quoted 46 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
  backends/hostmem.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
backends/hostmem.c +4 -2
index 327f9eebc3..0efd7b7bd6 100644
@@ -383,8 +383,10 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)
          assert(sizeof(backend->host_nodes) >=
                 BITS_TO_LONGS(MAX_NODES + 1) * sizeof(unsigned long));
          assert(maxnode <= MAX_NODES);
-        if (mbind(ptr, sz, backend->policy,
-                  maxnode ? backend->host_nodes : NULL, maxnode + 1, flags)) {
+
+        if (maxnode &&
+            mbind(ptr, sz, backend->policy, backend->host_nodes, maxnode + 1,
+                  flags)) {
              if (backend->policy != MPOL_DEFAULT || errno != ENOSYS) {
                  error_setg_errno(errp, errno,
                                   "cannot bind memory to host NUMA nodes");
#2 lore raw

Thanks! I applied the patch, and now it works also inside the docker container, for all architectures (i386, x86_64, arm, aarch64) for which I have test cases at hand.

Indeed, since the container is configured by a public cloud service, there is no possibility to change any security settings. Disabling mbind unless explicitly requested seems to be the best way to go here.

Quoted 50 lines · Philippe Mathieu-Daudé

Typo "empty" in patch subject.

Quoted 46 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
  "cannot bind memory to host NUMA nodes: Operation not permitted" even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
---
  backends/hostmem.c | 6 ++++--

  1 file changed, 4 insertions(+), 2 deletions(-)
backends/hostmem.c +0 -0
index 327f9eebc3..0efd7b7bd6 100644
@@ -383,8 +383,10 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)

          assert(sizeof(backend->host_nodes) >= BITS_TO_LONGS(MAX_NODES + 1) * sizeof(unsigned long)); assert(maxnode <= MAX_NODES);
-        if (mbind(ptr, sz, backend->policy,
-                  maxnode ? backend->host_nodes : NULL, maxnode + 1, flags)) { +
+        if (maxnode &&
+            mbind(ptr, sz, backend->policy, backend->host_nodes, maxnode + 1, +                  flags)) {
              if (backend->policy != MPOL_DEFAULT || errno != ENOSYS) { error_setg_errno(errp, errno,
                                   "cannot bind memory to host NUMA nodes");

Reviewed-by on v1 1/1
#3 lore raw
In reply to Igor Mammedov
Quoted 35 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
 backends/hostmem.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
backends/hostmem.c +0 -0
index 327f9eebc3..0efd7b7bd6 100644
@@ -383,8 +383,10 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)
         assert(sizeof(backend->host_nodes) >=
                BITS_TO_LONGS(MAX_NODES + 1) * sizeof(unsigned long));

backends/hostmem.c host_memory_backend_memory_complete(UserCreatable *uc, Error **errp) · line 392

Quoted 10 lines
backends/hostmem.c +4 -2
         assert(maxnode <= MAX_NODES);
-        if (mbind(ptr, sz, backend->policy,
-                  maxnode ? backend->host_nodes : NULL, maxnode + 1, flags)) {
+
+        if (maxnode &&
+            mbind(ptr, sz, backend->policy, backend->host_nodes, maxnode + 1,
+                  flags)) {
             if (backend->policy != MPOL_DEFAULT || errno != ENOSYS) {
                 error_setg_errno(errp, errno,
                                  "cannot bind memory to host NUMA nodes");

personally I would have found this code clearer if the check had been "if (backend->policy != MPOL_DEFAULT && ..." as I had to read quite a few lines to understand that the 'maxnode' is zero if-and-only-if policy == MPOL_DEFAULT

Regardless though, this is functionally correct so

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
Quoted 52 lines · Daniel P. Berrangé
Quoted 35 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

---
  backends/hostmem.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
backends/hostmem.c +0 -0
index 327f9eebc3..0efd7b7bd6 100644
@@ -383,8 +383,10 @@ host_memory_backend_memory_complete(UserCreatable *uc, Error **errp)
          assert(sizeof(backend->host_nodes) >=
                 BITS_TO_LONGS(MAX_NODES + 1) * sizeof(unsigned long));

backends/hostmem.c host_memory_backend_memory_complete(UserCreatable *uc, Error **errp) · line 392

Quoted 10 lines
backends/hostmem.c +4 -2
          assert(maxnode <= MAX_NODES);
-        if (mbind(ptr, sz, backend->policy,
-                  maxnode ? backend->host_nodes : NULL, maxnode + 1, flags)) {
+
+        if (maxnode &&
+            mbind(ptr, sz, backend->policy, backend->host_nodes, maxnode + 1,
+                  flags)) {
              if (backend->policy != MPOL_DEFAULT || errno != ENOSYS) {
                  error_setg_errno(errp, errno,
                                   "cannot bind memory to host NUMA nodes");

personally I would have found this code clearer if the check had been "if (backend->policy != MPOL_DEFAULT && ..." as I had to read quite a few lines to understand that the 'maxnode' is zero if-and-only-if policy == MPOL_DEFAULT

Quoted 3 lines

Regardless though, this is functionally correct so

I could reproduce running 'make check-qtest-hppa' on the qemu:fedora image:

TEST check-qtest-hppa: tests/qtest/boot-serial-test
qemu-system-hppa: cannot bind memory to host NUMA nodes: Operation not permitted
Broken pipe
tests/qtest/libqtest.c:166: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0) ERROR - too few tests run (expected 1, got 0)
make: *** [tests/Makefile.include:637: check-qtest-hppa] Error 1

Quoted 4 lines

Regards,
Daniel

#5 lore raw
In reply to Igor Mammedov
Quoted 16 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

Quoted 2 lines

Queued on machine-next, thanks!

Signature · 1 line
Eduardo
#6 lore raw

Hi Eduardo,

Quoted 17 lines · Eduardo Habkost
Quoted 16 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

Quoted 4 lines
Quoted 2 lines

Queued on machine-next, thanks!

I've been debugging this issue again today and figured it was not merged, if possible can you add the "Cc: [email protected]" tag before sending your pull request?

Thanks,

Phil.

#7 lore raw
Quoted 20 lines · On Mon, 11 May 2020 18:00:01 +0200 Philippe Mathieu-Daudé <[email protected]>…

Hi Eduardo,

Quoted 17 lines · Eduardo Habkost
Quoted 16 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Queued on machine-next, thanks!

I've been debugging this issue again today and figured it was not merged, if possible can you add the "Cc: [email protected]" tag before sending your pull request?

it's CCed already, so my impression was that will should picked up once it was reviewed.

Quoted 5 lines

Thanks,

Phil.

#8 lore raw
Quoted 26 lines · Igor Mammedov
Quoted 23 lines · On Mon, 11 May 2020 18:00:01 +0200 Philippe Mathieu-Daudé <[email protected]>…

Hi Eduardo,

Quoted 20 lines · Eduardo Habkost
Quoted 18 lines · Igor Mammedov

Since 5.0 QEMU uses hostmem backend for allocating main guest RAM. The backend however calls mbind() which is typically NOP in case of default policy/absent host-nodes bitmap. However when runing in container with black-listed mbind() syscall, QEMU fails to start with error
"cannot bind memory to host NUMA nodes: Operation not permitted"
even when user hasn't provided host-nodes to pin to explictly (which is the case with -m option)

To fix issue, call mbind() only in case when user has provided host-nodes explicitly (i.e. host_nodes bitmap is not empty). That should allow to run QEMU in containers with black-listed mbind() without memory pinning. If QEMU provided memory-pinning is required user still has to white-list mbind() in container configuration.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Queued on machine-next, thanks!

I've been debugging this issue again today and figured it was not merged, if possible can you add the "Cc: [email protected]" tag before sending your pull request?

it's CCed already, so my impression was that will should picked up once it was reviewed.

Correct, however some distributions find easier to grep for the 'Cc: [email protected]' merged tag before qemu-stable is released.

Quoted 7 lines
Quoted 6 lines

Thanks,

Phil.