Skip to contentSkip to conversation

KVM:x86: Let kvm-pit thread inherit the cgroups of the calling process

PATCH In review Jietao Xiao 2 participants 2 messages Jan 2, 2022 – Jan 6, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM:x86: Let kvm-pit thread inherit the cgroups of the calling process

Qemu-kvm will create several kernel threads for each VM including kvm-nx-lpage-re, vhost, and so on. Both of them properly inherit the cgroups of the calling process,so they are easy to attach to the VMM process's cgroups.

Kubernetes has a feature Pod Overhead for accounting for the resources consumed by the Pod infrastructure(e.g overhead brought by qemu-kvm), and sandbox container runtime usually creates a sandbox or sandbox overhead cgroup for this feature. By just simply adding the runtime or the VMM process to the sandbox's cgroup, vhost and kvm-nx-lpage-re thread can successfully attach to the sanbox's cgroup but kvm-pit thread cannot. Besides, in some scenarios, kvm-pit thread can bring some CPU overhead. So it's better to let the kvm-pit inherit the cgroups of the calling userspace process.

By queuing the attach cgroup work as the first work after the creation of the kvm-pit worker thread, the worker thread can successfully attach to the callings process's cgroups.

---
 arch/x86/kvm/i8254.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
arch/x86/kvm/i8254.c +32 -0
index 0b65a764ed3a..c8dcfd6a9ed4 100644
@@ -34,6 +34,7 @@
 
 #include <linux/kvm_host.h>
 #include <linux/slab.h>
+#include <linux/cgroup.h>
 
 #include "ioapic.h"
 #include "irq.h"
@@ -647,6 +648,32 @@ static void pit_mask_notifer(struct kvm_irq_mask_notifier *kimn, bool mask)
 		kvm_pit_reset_reinject(pit);
 }
 
+struct pit_attach_cgroups_struct {
+	struct kthread_work work;
+	struct task_struct *owner;
+	int ret;
+};
+
+static void pit_attach_cgroups_work(struct kthread_work *work)
+{
+	struct pit_attach_cgroups_struct *attach;
+
+	attach = container_of(work, struct pit_attach_cgroups_struct, work);
+	attach->ret = cgroup_attach_task_all(attach->owner, current);
+}
+
+
+static int pit_attach_cgroups(struct kvm_pit *pit)
+{
+	struct pit_attach_cgroups_struct attach;
+
+	attach.owner = current;
+	kthread_init_work(&attach.work, pit_attach_cgroups_work);
+	kthread_queue_work(pit->worker, &attach.work);
+	kthread_flush_work(&attach.work);
+	return attach.ret;
+}
+
 static const struct kvm_io_device_ops pit_dev_ops = {
 	.read     = pit_ioport_read,
 	.write    = pit_ioport_write,
@@ -683,6 +710,10 @@ struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags)
 	if (IS_ERR(pit->worker))
 		goto fail_kthread;
 
+	ret = pit_attach_cgroups(pit);
+	if (ret < 0)
+		goto fail_attach_cgroups;
+
 	kthread_init_work(&pit->expired, pit_do_work);
 
 	pit->kvm = kvm;
@@ -723,6 +754,7 @@ struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags)
 fail_register_pit:
 	mutex_unlock(&kvm->slots_lock);
 	kvm_pit_set_reinject(pit, false);
+fail_attach_cgroups:
 	kthread_destroy_worker(pit->worker);
 fail_kthread:
 	kvm_free_irq_source_id(kvm, pit->irq_source_id);
Signature · 1 line
2.20.1
#1 lore raw
Quoted 11 lines · Jietao Xiao

Qemu-kvm will create several kernel threads for each VM including kvm-nx-lpage-re, vhost, and so on. Both of them properly inherit the cgroups of the calling process,so they are easy to attach to the VMM process's cgroups.

Kubernetes has a feature Pod Overhead for accounting for the resources consumed by the Pod infrastructure(e.g overhead brought by qemu-kvm), and sandbox container runtime usually creates a sandbox or sandbox overhead cgroup for this feature. By just simply adding the runtime or the VMM process to the sandbox's cgroup, vhost and kvm-nx-lpage-re thread can successfully attach to the sanbox's cgroup but kvm-pit thread cannot.

Emm, it seems to be true for kvm-pit kthread.

Quoted 3 lines

Besides, in some scenarios, kvm-pit thread can bring some CPU overhead. So it's better to let the kvm-pit inherit the cgroups of the calling userspace process.

As a side note, there is about ~3% overhead in the firecracker scenario.

Quoted 28 lines

By queuing the attach cgroup work as the first work after the creation of the kvm-pit worker thread, the worker thread can successfully attach to the callings process's cgroups.

---
  arch/x86/kvm/i8254.c | 32 ++++++++++++++++++++++++++++++++
  1 file changed, 32 insertions(+)
arch/x86/kvm/i8254.c +3 -0
index 0b65a764ed3a..c8dcfd6a9ed4 100644
@@ -34,6 +34,7 @@
 
  #include <linux/kvm_host.h>
  #include <linux/slab.h>
+#include <linux/cgroup.h>
 
  #include "ioapic.h"
  #include "irq.h"
@@ -647,6 +648,32 @@ static void pit_mask_notifer(struct kvm_irq_mask_notifier *kimn, bool mask)
  		kvm_pit_reset_reinject(pit);
  }
 
+struct pit_attach_cgroups_struct {
+	struct kthread_work work;

arch/x86/kvm/i8254.c static void pit_mask_notifer(struct kvm_irq_mask_notifier *kimn, bool mask) · line 662

Quoted 10 lines
arch/x86/kvm/i8254.c +10 -0
+	struct task_struct *owner;
+	int ret;
+};
+
+static void pit_attach_cgroups_work(struct kthread_work *work)
+{
+	struct pit_attach_cgroups_struct *attach;
+
+	attach = container_of(work, struct pit_attach_cgroups_struct, work);
+	attach->ret = cgroup_attach_task_all(attach->owner, current);

This cgroup_v1 interface is also called by the vhost_attach_cgroups_work(), as well as the kvm_vm_worker_thread() in the KVM context.

This part of the code may be a bit redundant as the number of kthreads increases.

Quoted 18 lines
arch/x86/kvm/i8254.c +14 -0
+}
+
+
+static int pit_attach_cgroups(struct kvm_pit *pit)
+{
+	struct pit_attach_cgroups_struct attach;
+
+	attach.owner = current;
+	kthread_init_work(&attach.work, pit_attach_cgroups_work);
+	kthread_queue_work(pit->worker, &attach.work);
+	kthread_flush_work(&attach.work);
+	return attach.ret;
+}
+
  static const struct kvm_io_device_ops pit_dev_ops = {
  	.read     = pit_ioport_read,
  	.write    = pit_ioport_write,

arch/x86/kvm/i8254.c struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags) · line 711

Quoted 2 lines
arch/x86/kvm/i8254.c +0 -0
@@ -683,6 +710,10 @@ struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags)
  	if (IS_ERR(pit->worker))
  		goto fail_kthread;

I wonder if we could unify the kthread_create method for both vhost and kvm-pit so that all kthreds from kvm_arch_vm agent could share the cgroup_attach_task_all() code base and more stuff like set_user_nice().

Quoted 9 lines
arch/x86/kvm/i8254.c +4 -0
+	ret = pit_attach_cgroups(pit);
+	if (ret < 0)
+		goto fail_attach_cgroups;
+
  	kthread_init_work(&pit->expired, pit_do_work);
 
  	pit->kvm = kvm;

arch/x86/kvm/i8254.c struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags) · line 758

Quoted 5 lines
arch/x86/kvm/i8254.c +1 -0
@@ -723,6 +754,7 @@ struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags)
  fail_register_pit:
  	mutex_unlock(&kvm->slots_lock);
  	kvm_pit_set_reinject(pit, false);
+fail_attach_cgroups:
  	kthread_destroy_worker(pit->worker);

If it fails, could we keep it at least alive and functional ?

Quoted 2 lines
arch/x86/kvm/i8254.c +0 -0
  fail_kthread:
  	kvm_free_irq_source_id(kvm, pit->irq_source_id);