Skip to contentSkip to conversation

KVM: Move VM's worker kthreads back to the original cgroups before exiting.

PATCH Superseded Vipin Sharma 5 participants 9 messages Dec 14, 2021 – Dec 22, 2021 KVM

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: Move VM's worker kthreads back to the original cgroups before exiting.

VM worker kthreads can linger in the VM process's cgroup for sometime after KVM temrinates the VM process.

KVM terminates the worker kthreads by calling kthread_stop() which waits on the signal generated by exit_mm() in do_exit() during kthread's exit. However, these kthreads are removed from the cgroup using cgroup_exit() call which happens after exit_mm() in do_exit(). A VM process can terminate between the time window of exit_mm() to cgroup_exit(), leaving only worker kthreads in the cgroup.

Moving worker kthreads back to the original cgroup (kthreadd_task's cgroup) makes sure that cgroup is empty as soon as the main VM process is terminated.

---
 virt/kvm/kvm_main.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
virt/kvm/kvm_main.c +10 -1
index b0f7e6eb00ff..edd304a18f16 100644
@@ -5785,7 +5785,7 @@ static int kvm_vm_worker_thread(void *context)
 	init_context = NULL;
 
 	if (err)
-		return err;
+		goto out;
 
 	/* Wait to be woken up by the spawner before proceeding. */
 	kthread_parkme();
@@ -5793,6 +5793,15 @@ static int kvm_vm_worker_thread(void *context)
 	if (!kthread_should_stop())
 		err = thread_fn(kvm, data);
 
+out:
+	/*
+	 * We need to move the kthread back to its original cgroups, so that it
+	 * doesn't linger in the cgroups of the user process after that has
+	 * already terminated. exit_mm() in do_exit() signals kthread_stop() to
+	 * return, whereas, removal of the task from the cgroups happens in
+	 * cgroup_exit() which happens after exit_mm().
+	 */
+	WARN_ON(cgroup_attach_task_all(kthreadd_task, current));
 	return err;
 }
 

base-commit: d8f6ef45a623d650f9b97e11553adb4978f6aa70

Signature · 1 line
2.34.1.173.g76aa8bc2d0-goog
#1 lore raw
Thank you for the patch! Yet something to improve:

Hi Vipin,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on d8f6ef45a623d650f9b97e11553adb4978f6aa70]

url:    https://github.com/0day-ci/linux/commits/Vipin-Sharma/KVM-Move-VM-s-worker-kthreads-back-to-the-original-cgroups-before-exiting/20211214-130827
base:   d8f6ef45a623d650f9b97e11553adb4978f6aa70
config: riscv-defconfig (https://download.01.org/0day-ci/archive/20211214/202112142328.a9ebDmd7-lkp(a)intel.com/config)
compiler: riscv64-linux-gcc (GCC) 11.2.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/fd29d23507ef3f06b61d9de1b7ecd1a0d70136f3
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Vipin-Sharma/KVM-Move-VM-s-worker-kthreads-back-to-the-original-cgroups-before-exiting/20211214-130827 git checkout fd29d23507ef3f06b61d9de1b7ecd1a0d70136f3
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=riscv SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> ERROR: modpost: "kthreadd_task" [arch/riscv/kvm/kvm.ko] undefined!

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#2 lore raw
In reply to Vipin Sharma
Quoted 11 lines · Vipin Sharma

VM worker kthreads can linger in the VM process's cgroup for sometime after KVM temrinates the VM process.

KVM terminates the worker kthreads by calling kthread_stop() which waits on the signal generated by exit_mm() in do_exit() during kthread's exit. However, these kthreads are removed from the cgroup using cgroup_exit() call which happens after exit_mm() in do_exit(). A VM process can terminate between the time window of exit_mm() to cgroup_exit(), leaving only worker kthreads in the cgroup.

Quoted 8 lines

Moving worker kthreads back to the original cgroup (kthreadd_task's cgroup) makes sure that cgroup is empty as soon as the main VM process is terminated.

---
 virt/kvm/kvm_main.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

Hello

Off-topic, can this kvm worker_thread and the thread to do async pagefault be possibly changed to use something like io_uring's IOWQ (fs/io-wq.c) created by create_io_thread()?

So that every resource the threads used are credited to the process of the vm.

Thanks
Lai

#3 lore raw
Quoted 10 lines · Lai Jiangshan

Hello

Off-topic, can this kvm worker_thread and the thread to do async pagefault be possibly changed to use something like io_uring's IOWQ (fs/io-wq.c) created by create_io_thread()?

So that every resource the threads used are credited to the process of the vm.

Sorry, I am not very familiar with it. Maybe someone from the community knows better about this.

#4 lore raw
In reply to Vipin Sharma
Quoted 35 lines · Vipin Sharma

VM worker kthreads can linger in the VM process's cgroup for sometime after KVM temrinates the VM process.

KVM terminates the worker kthreads by calling kthread_stop() which waits on the signal generated by exit_mm() in do_exit() during kthread's exit. However, these kthreads are removed from the cgroup using cgroup_exit() call which happens after exit_mm() in do_exit(). A VM process can terminate between the time window of exit_mm() to cgroup_exit(), leaving only worker kthreads in the cgroup.

Moving worker kthreads back to the original cgroup (kthreadd_task's cgroup) makes sure that cgroup is empty as soon as the main VM process is terminated.

---
 virt/kvm/kvm_main.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
virt/kvm/kvm_main.c +1 -1
index b0f7e6eb00ff..edd304a18f16 100644
@@ -5785,7 +5785,7 @@ static int kvm_vm_worker_thread(void *context)
 	init_context = NULL;
 
 	if (err)
-		return err;
+		goto out;
 
 	/* Wait to be woken up by the spawner before proceeding. */
 	kthread_parkme();
@@ -5793,6 +5793,15 @@ static int kvm_vm_worker_thread(void *context)
 	if (!kthread_should_stop())
 		err = thread_fn(kvm, data);

virt/kvm/kvm_main.c static int kvm_vm_worker_thread(void *context) · line 5804

Quoted 10 lines
virt/kvm/kvm_main.c +9 -0
 
+out:
+	/*
+	 * We need to move the kthread back to its original cgroups, so that it
+	 * doesn't linger in the cgroups of the user process after that has
+	 * already terminated. exit_mm() in do_exit() signals kthread_stop() to
+	 * return, whereas, removal of the task from the cgroups happens in
+	 * cgroup_exit() which happens after exit_mm().
+	 */
+	WARN_ON(cgroup_attach_task_all(kthreadd_task, current));

As the build bot noted, kthreadd_task isn't exported, and I doubt you'll convince folks to let you export it.

Why is it problematic for the kthread to linger in the cgroup? Conceptually, it's not really wrong.

Quoted 8 lines
virt/kvm/kvm_main.c +0 -0
 	return err;
 }
 

base-commit: d8f6ef45a623d650f9b97e11553adb4978f6aa70

Signature · 1 line
2.34.1.173.g76aa8bc2d0-goog
#5 lore raw
Quoted 9 lines · Sean Christopherson
Quoted 1 line · Vipin Sharma

+ WARN_ON(cgroup_attach_task_all(kthreadd_task, current));

As the build bot noted, kthreadd_task isn't exported, and I doubt you'll convince folks to let you export it.

Why is it problematic for the kthread to linger in the cgroup? Conceptually, it's not really wrong.

Issue comes when a process tries to clear up the resources when a VM shutdown/dies. The process sometimes get an EBUSY error when it tries to delete the cgroup directories which were created for that VM. It is also difficult to know how many times to retry or how much time to wait before the cgroup is empty. This issue is not always happening.

#6 lore raw
In reply to Vipin Sharma
Thank you for the patch! Yet something to improve:

Hi Vipin,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on d8f6ef45a623d650f9b97e11553adb4978f6aa70]

url:    https://github.com/0day-ci/linux/commits/Vipin-Sharma/KVM-Move-VM-s-worker-kthreads-back-to-the-original-cgroups-before-exiting/20211214-130827
base:   d8f6ef45a623d650f9b97e11553adb4978f6aa70
config: x86_64-rhel-8.3 (https://download.01.org/0day-ci/archive/20211215/202112150131.MaZ9xOJx-lkp(a)intel.com/config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/fd29d23507ef3f06b61d9de1b7ecd1a0d70136f3
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Vipin-Sharma/KVM-Move-VM-s-worker-kthreads-back-to-the-original-cgroups-before-exiting/20211214-130827
        git checkout fd29d23507ef3f06b61d9de1b7ecd1a0d70136f3
        # save the config file to linux build tree
        mkdir build_dir
        make W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> ERROR: modpost: "kthreadd_task" [arch/x86/kvm/kvm.ko] undefined!

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

maintainer
#7 lore raw
In reply to Vipin Sharma
Quoted 3 lines · Vipin Sharma

KVM terminates the worker kthreads by calling kthread_stop() which waits on the signal generated by exit_mm() in do_exit() during kthread's exit.

Instead of "signal", please spell it as "the 'exited' completion, triggered by exit_mm(), via mm_release(), during the kthread's exit". That makes things a bit clearer.

So the issue is that the kthread_stop happens around the time exit_task_work() destroys the VM, but the process can go on and signal its demise to the parent process before the kthread has been completely dropped. Not even close() can fix it, though it may reduce the window completely, so I agree that this is a bug and vhost has the same bug too.

Due to the issue with kthreadd_task not being exported, perhaps you can change cgroup_attach_task_all to use kthreadd_task if the "from" argument is NULL?

Paolo

#8 lore raw
Quoted 10 lines · Paolo Bonzini

So the issue is that the kthread_stop happens around the time exit_task_work() destroys the VM, but the process can go on and signal its demise to the parent process before the kthread has been completely dropped. Not even close() can fix it, though it may reduce the window completely, so I agree that this is a bug and vhost has the same bug too.

Due to the issue with kthreadd_task not being exported, perhaps you can change cgroup_attach_task_all to use kthreadd_task if the "from" argument is NULL?

Thanks for the suggestion. This also works, I will send out a patch.