Skip to contentSkip to conversation

Add a new acquire resource to query vcpu statistics

RFC PATCH Superseded Matias Ezequiel Vara Larsen 2 participants 5 messages Aug 24, 2022 – Sep 30, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Add a new acquire resource to query vcpu statistics

Hello all,

The purpose of this RFC is to get feedback about a new acquire resource that exposes vcpu statistics for a given domain. The current mechanism to get those statistics is by querying the hypervisor. This mechanism relies on a hypercall and holds the domctl spinlock during its execution. When a pv tool like xcp-rrdd periodically samples these counters, it ends up affecting other paths that share that spinlock. By using acquire resources, the pv tool only requires a few hypercalls to set the shared memory region and samples are got without issuing any other hypercall. The original idea has been suggested by Andrew Cooper to which I have been discussing about how to implement the current PoC. You can find the RFC patch series at [1]. The series is rebased on top of stable-4.15.

I am currently a bit blocked on 1) what to expose and 2) how to expose it. For 1), I decided to expose what xcp-rrdd is querying, e.g., XEN_DOMCTL_getvcpuinfo. More precisely, xcp-rrd gets runstate.time[RUNSTATE_running]. This is a uint64_t counter. However, the time spent in other states may be interesting too. Regarding 2), I am not sure if simply using an array of uint64_t is enough or if a different interface should be exposed. The remaining question is when to get new values. For the moment, I am updating this counter during vcpu_runstate_change().

The current series includes a simple pv tool that shows how this new interface is used. This tool maps the counter and periodically samples it.

Any feedback/help would be appreciated.

Thanks, Matias.

[1] https://github.com/MatiasVara/xen/tree/feature_stats

Changes in v1:
- rework how the resource is allocated and released
- rework when the resource is allocated that happens only when the resource is requested
- rework the structure shared between the tool and Xen to make it extensible to new counters and declare it in a public header

There are still the following questions:
- consumer may fetch inconsistency data
- resource shall be released when there are no more readers otherwise we keep updating it during a hot path
- one frame can host up to 512 vcpus. Should I check to this limit when updating? Should it be possible to allocate more than one frame for vcpu counters?

Matias Ezequiel Vara Larsen (2):
xen/memory : Add a stats_table resource type
tools/misc: Add xen-vcpus-stats tool

 tools/misc/Makefile          |  6 +++
 tools/misc/xen-vcpus-stats.c | 76 +++++++++++++++++++++++++++++
 xen/arch/x86/hvm/hvm.c       |  2 +
 xen/common/memory.c          | 94 ++++++++++++++++++++++++++++++++++++
 xen/common/sched/core.c      |  7 +++
 xen/include/public/memory.h  |  3 ++
 xen/include/public/vcpu.h    | 10 ++++
 xen/include/xen/mm.h         |  2 +
 xen/include/xen/sched.h      |  5 ++
 9 files changed, 205 insertions(+)
 create mode 100644 tools/misc/xen-vcpus-stats.c
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH 1/2 xen/memory : Add a stats_table resource type

This commit proposes a new mechanism to query the RUNSTATE_running counter for a given vcpu from a dom0 userspace application. This commit proposes to expose that counter by using the acquire_resource interface. The current mechanism relies on the XEN_DOMCTL_getvcpuinfo and holds a single global domctl_lock for the entire hypercall; and iterate over every vcpu in the system for every update thus impacting operations that share that lock.

This commit proposes to expose vcpu RUNSTATE_running via the xenforeignmemory interface thus preventing to issue the hypercall and holding the lock. For that purpose, a new resource type named stats_table is added. The first frame of this resource stores per-vcpu counters. The frame has one entry of type struct vcpu_stats per vcpu. The allocation of this frame only happens if the resource is requested. The frame is released after the domain is destroyed.

Note that the updating of this counter is in a hot path, thus, in this commit, copying only happens if it is specifically required.

Note that the exposed structure is extensible in two ways. First, the structure vcpu_stats can be extended with new per-vcpu counters while it fits in a frame. Second, new frames can be added in case new counters are required.

---
Changes in v1:
- rework the allocation and releasing of the frames
- use the zero frame for per-vcpu counters that are listed as an array
- allocate vcpu stats frames only when the resource is requested
- rewrite commit message
- add the vcpu_stats structure to keep per-vcpu counters
- add the shared_vcpustatspage to keep an array of per-vcpu counters for a given domain
- declare the structures in a public header
- define the vcpustats_page in the domain structure

---
 xen/arch/x86/hvm/hvm.c      |  2 +
 xen/common/memory.c         | 94 +++++++++++++++++++++++++++++++++++++
 xen/common/sched/core.c     |  7 +++
 xen/include/public/memory.h |  3 ++
 xen/include/public/vcpu.h   | 10 ++++
 xen/include/xen/mm.h        |  2 +
 xen/include/xen/sched.h     |  5 ++
 7 files changed, 123 insertions(+)
xen/arch/x86/hvm/hvm.c +2 -0
index ddd001a6ad..1ef6cb5ff0 100644
@@ -741,6 +741,8 @@ void hvm_domain_relinquish_resources(struct domain *d)
 
     ioreq_server_destroy_all(d);
 
+    stats_free_vcpu_mfn(d);
+
     msixtbl_pt_cleanup(d);
 
     /* Stop all asynchronous timer actions. */
xen/common/memory.c +94 -0

118 lines not on this page · raw message

xen/common/sched/core.c +7 -0
index 8f4b1ca10d..87cfa172ad 100644
@@ -264,6 +264,8 @@ static inline void vcpu_runstate_change(
 {
     s_time_t delta;
     struct sched_unit *unit = v->sched_unit;
+    shared_vcpustatspage_t * vcpustats_va;
+    struct domain *d = v->domain;
 
     ASSERT(spin_is_locked(get_sched_res(v->processor)->schedule_lock));
     if ( v->runstate.state == new_state )
@@ -287,6 +289,11 @@ static inline void vcpu_runstate_change(
     }
 
     v->runstate.state = new_state;
+
+    vcpustats_va = (shared_vcpustatspage_t*)d->vcpustats_page.va;
+    if ( vcpustats_va )
+        memcpy(&vcpustats_va->vcpu_info[v->vcpu_id].runstate_running_time,
+               &v->runstate.time[RUNSTATE_running], sizeof(v->runstate.time[RUNSTATE_running]));
 }
 
 void sched_guest_idle(void (*idle) (void), unsigned int cpu)
xen/include/public/memory.h +3 -0
index 50e73eef98..e1a10b8b97 100644
@@ -626,6 +626,7 @@ struct xen_mem_acquire_resource {
 #define XENMEM_resource_ioreq_server 0
 #define XENMEM_resource_grant_table 1
 #define XENMEM_resource_vmtrace_buf 2
+#define XENMEM_resource_stats_table 3
 
     /*
      * IN - a type-specific resource identifier, which must be zero
@@ -683,6 +684,8 @@ struct xen_mem_acquire_resource {
 typedef struct xen_mem_acquire_resource xen_mem_acquire_resource_t;
 DEFINE_XEN_GUEST_HANDLE(xen_mem_acquire_resource_t);
 
+#define XENMEM_resource_stats_frame_vcpustats 0
+
 /*
  * XENMEM_get_vnumainfo used by guest to get
  * vNUMA topology from hypervisor.
xen/include/public/vcpu.h +10 -0
index 3623af932f..57a5ca777c 100644
@@ -235,6 +235,16 @@ struct vcpu_register_time_memory_area {
 typedef struct vcpu_register_time_memory_area vcpu_register_time_memory_area_t;
 DEFINE_XEN_GUEST_HANDLE(vcpu_register_time_memory_area_t);
 
+struct vcpu_stats{
+   uint64_t runstate_running_time;
+};
+
+struct shared_vcpustatspage {
+   struct vcpu_stats vcpu_info[1];
+};
+
+typedef struct shared_vcpustatspage shared_vcpustatspage_t;
+
 #endif /* __XEN_PUBLIC_VCPU_H__ */
 
 /*
xen/include/xen/mm.h +2 -0
index 667f9dac83..d1ca8b9aa8 100644
@@ -134,6 +134,8 @@ int assign_pages(
 /* Dump info to serial console */
 void arch_dump_shared_mem_info(void);
 
+void stats_free_vcpu_mfn(struct domain * d);
+
 /*
  * Extra fault info types which are used to further describe
  * the source of an access violation.
xen/include/xen/sched.h +5 -0
index 5485d08afb..d9551ce35f 100644
@@ -577,6 +577,11 @@ struct domain
         struct ioreq_server     *server[MAX_NR_IOREQ_SERVERS];
     } ioreq_server;
 #endif
+    /* Page that hosts vcpu stats */
+    struct {
+        struct page_info *pg;
+        void *va;
+    } vcpustats_page;
 };
 
 static inline struct page_list_head *page_to_list(
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH 2/2 tools/misc: Add xen-vcpus-stats tool

In reply to Matias Ezequiel Vara Larsen

Add a demostration tool that uses the stats_table resource to query vcpus' RUNSTATE_running counter for a DomU.

---
Changes in v1:
- change the name of the tool to xen-vcpus-stats
- set command line parameters in the same order that are passed
- remove header libs.h
- build by default
- remove errno, strerrno, "\n", and identation
- use errx when errno is not needed
- address better the number of pages requested and error msgs
- use the shared_vcpustatspage_t structure
- use the correct frame id when requesting the resource

---
 tools/misc/Makefile          |  6 +++
 tools/misc/xen-vcpus-stats.c | 76 ++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)
 create mode 100644 tools/misc/xen-vcpus-stats.c
tools/misc/Makefile +6 -0
index 2b683819d4..837e4b50da 100644
@@ -49,6 +49,7 @@ TARGETS_COPY += xenpvnetboot
 
 # Everything which needs to be built
 TARGETS_BUILD := $(filter-out $(TARGETS_COPY),$(TARGETS_ALL))
+TARGETS_BUILD += xen-vcpus-stats
 
 # ... including build-only targets
 TARGETS_BUILD-$(CONFIG_X86)    += xen-vmtrace
@@ -135,4 +136,9 @@ xencov: xencov.o
 xen-ucode: xen-ucode.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-vcpus-stats.o: CFLAGS += $(CFLAGS_libxenforeginmemory)
+
+xen-vcpus-stats: xen-vcpus-stats.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(LDLIBS_libxenforeignmemory) $(APPEND_LDFLAGS)
+
 -include $(DEPS_INCLUDE)
tools/misc/xen-vcpus-stats.c new file +76 -0

76 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw
In reply to Matias Ezequiel Vara Larsen
Quoted 20 lines · Matias Ezequiel Vara Larsen

The purpose of this RFC is to get feedback about a new acquire resource that exposes vcpu statistics for a given domain. The current mechanism to get those statistics is by querying the hypervisor. This mechanism relies on a hypercall and holds the domctl spinlock during its execution. When a pv tool like xcp-rrdd periodically samples these counters, it ends up affecting other paths that share that spinlock. By using acquire resources, the pv tool only requires a few hypercalls to set the shared memory region and samples are got without issuing any other hypercall. The original idea has been suggested by Andrew Cooper to which I have been discussing about how to implement the current PoC. You can find the RFC patch series at [1]. The series is rebased on top of stable-4.15.

I am currently a bit blocked on 1) what to expose and 2) how to expose it. For 1), I decided to expose what xcp-rrdd is querying, e.g., XEN_DOMCTL_getvcpuinfo. More precisely, xcp-rrd gets runstate.time[RUNSTATE_running]. This is a uint64_t counter. However, the time spent in other states may be interesting too. Regarding 2), I am not sure if simply using an array of uint64_t is enough or if a different interface should be exposed. The remaining question is when to get new values. For the moment, I am updating this counter during vcpu_runstate_change().

Quoted 4 lines

The current series includes a simple pv tool that shows how this new interface is used. This tool maps the counter and periodically samples it.

Any feedback/help would be appreciated.

Before looking more closely - was there perhaps kind-of-review feedback during the summit, which would make it more reasonable to look through this once a v2 has appeared?

Jan

#4 lore raw
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Matias Ezequiel Vara Larsen

The purpose of this RFC is to get feedback about a new acquire resource that exposes vcpu statistics for a given domain. The current mechanism to get those statistics is by querying the hypervisor. This mechanism relies on a hypercall and holds the domctl spinlock during its execution. When a pv tool like xcp-rrdd periodically samples these counters, it ends up affecting other paths that share that spinlock. By using acquire resources, the pv tool only requires a few hypercalls to set the shared memory region and samples are got without issuing any other hypercall. The original idea has been suggested by Andrew Cooper to which I have been discussing about how to implement the current PoC. You can find the RFC patch series at [1]. The series is rebased on top of stable-4.15.

I am currently a bit blocked on 1) what to expose and 2) how to expose it. For 1), I decided to expose what xcp-rrdd is querying, e.g., XEN_DOMCTL_getvcpuinfo. More precisely, xcp-rrd gets runstate.time[RUNSTATE_running]. This is a uint64_t counter. However, the time spent in other states may be interesting too. Regarding 2), I am not sure if simply using an array of uint64_t is enough or if a different interface should be exposed. The remaining question is when to get new values. For the moment, I am updating this counter during vcpu_runstate_change().

Quoted 9 lines
Quoted 4 lines

The current series includes a simple pv tool that shows how this new interface is used. This tool maps the counter and periodically samples it.

Any feedback/help would be appreciated.

Before looking more closely - was there perhaps kind-of-review feedback during the summit, which would make it more reasonable to look through this once a v2 has appeared?

Yes, there was. I will submit v2 from feedback during summit. Thanks for point it.

Matias