Skip to contentSkip to conversation

plugins/cache: multicore cache emulation and minor

PATCH Applied Mahmoud Mandour 2 participants 17 messages Jul 14, 2021 – Jul 20, 2021

v1 of 3 Applied Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover plugins/cache: multicore cache emulation and minor

Hello,

This series introduce some minor improvements/bug fixes in the cache plugins and multicore cache modelling.

One prominent bug is the use-after-free bug induced by linux-user multithreaded programs. Since plugin_exit is not guaranteed to be called after all callbacks, it may free data that subsequent callbacks may try to use.

Trying to uninstall the plugin after exiting does not solve the problem since it will unregister the callbacks but it won't prevent the already-fired callback instances from continuing.

To mitigate this issue, the data is NULLified on exitting so that callbacks can check whether plugin_exit has been called already or not.

Also, raising the levels of warnings (by the time this is sent, it's not yet upstreamed but it's in the process of getting merged) induced some warnings, this is fixed in its own patch.

Mahmoud Mandour (6):
plugins/cache: Fixed a bug with destroying FIFO metadata plugins/cache: limited the scope of a mutex lock plugins/cache: Fixed a use-after-free bug with multithreaded usermode plugins/cache: Supported multicore cache modelling docs/devel/tcg-plugins: added cores arg to cache plugin plugins/cache: Fixed "function decl. is not a prototype" warnings

 contrib/plugins/cache.c    | 188 +++++++++++++++++++++++++++++--------
 docs/devel/tcg-plugins.rst |  13 ++-
 2 files changed, 155 insertions(+), 46 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/6 plugins/cache: Fixed a bug with destroying FIFO metadata

This manifests itself when associativity degree is greater than the number of sets and FIFO is used, otherwise it's also a memory leak whenever FIFO was used.

---
 contrib/plugins/cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
contrib/plugins/cache.c +1 -1
index bf0d2f6097..4a71602639 100644
@@ -200,7 +200,7 @@ static void fifo_destroy(Cache *cache)
 {
     int i;
 
-    for (i = 0; i < cache->assoc; i++) {
+    for (i = 0; i < cache->num_sets; i++) {
         g_queue_free(cache->sets[i].fifo_queue);
     }
 }
Signature · 1 line
2.25.1
Reviewed-by on v1 1/6
#2 lore raw

Re: [PATCH 1/6] plugins/cache: Fixed a bug with destroying FIFO metadata

Quoted 5 lines · Mahmoud Mandour

This manifests itself when associativity degree is greater than the number of sets and FIFO is used, otherwise it's also a memory leak whenever FIFO was used.

Quoted 17 lines
---
 contrib/plugins/cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
contrib/plugins/cache.c +1 -1
index bf0d2f6097..4a71602639 100644
@@ -200,7 +200,7 @@ static void fifo_destroy(Cache *cache)
 {
     int i;
 
-    for (i = 0; i < cache->assoc; i++) {
+    for (i = 0; i < cache->num_sets; i++) {
         g_queue_free(cache->sets[i].fifo_queue);
     }
 }
Signature · 1 line
Alex Bennée
#3 lore raw

PATCH 2/6 plugins/cache: limited the scope of a mutex lock

In reply to Mahmoud Mandour

It's not necessary to lock the address translation portion of the vcpu_mem_access callback.

---
 contrib/plugins/cache.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
contrib/plugins/cache.c +1 -2
index 4a71602639..695fb969dc 100644
@@ -355,15 +355,14 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     struct qemu_plugin_hwaddr *hwaddr;
     InsnData *insn;
 
-    g_mutex_lock(&mtx);
     hwaddr = qemu_plugin_get_hwaddr(info, vaddr);
     if (hwaddr && qemu_plugin_hwaddr_is_io(hwaddr)) {
-        g_mutex_unlock(&mtx);
         return;
     }
 
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
+    g_mutex_lock(&mtx);
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
Signature · 1 line
2.25.1
Reviewed-by on v1 2/6
#4 lore raw

Re: [PATCH 2/6] plugins/cache: limited the scope of a mutex lock

Quoted 19 lines · Mahmoud Mandour

It's not necessary to lock the address translation portion of the vcpu_mem_access callback.

---
 contrib/plugins/cache.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
contrib/plugins/cache.c +0 -1
index 4a71602639..695fb969dc 100644
@@ -355,15 +355,14 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     struct qemu_plugin_hwaddr *hwaddr;
     InsnData *insn;
 
-    g_mutex_lock(&mtx);
     hwaddr = qemu_plugin_get_hwaddr(info, vaddr);
     if (hwaddr && qemu_plugin_hwaddr_is_io(hwaddr)) {

contrib/plugins/cache.c static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, · line 368

Quoted 10 lines
contrib/plugins/cache.c +1 -1
-        g_mutex_unlock(&mtx);
         return;
     }
 
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
+    g_mutex_lock(&mtx);
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;

This is fine, but I see an exit leg creeps in later which I think we can eliminate. I'll comment there:

Signature · 1 line
Alex Bennée
#5 lore raw

PATCH 3/6 plugins/cache: Fixed a use-after-free bug with multithreaded usermode

In reply to Mahmoud Mandour

Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.

This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.

It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is already called once and reported the statistics.

---
 contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
contrib/plugins/cache.c +30 -1
index 695fb969dc..a452aba01c 100644
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
     g_mutex_lock(&mtx);
+    if (dcache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
+    if (icache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(icache, insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
             effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
         }
 
+        g_mutex_lock(&mtx);
+
+        /*
+         * is the plugin_exit callback called? If so, any further callback
+         * registration is useless as it won't get accounted for after calling
+         * plugin_exit once already, and also will use miss_ht after it's freed
+         */
+        if (miss_ht == NULL) {
+            g_mutex_unlock(&mtx);
+            return;
+        }
+
         /*
          * Instructions might get translated multiple times, we do not create
          * new entries for those instructions. Instead, we fetch the same
          * entry from the hash table and register it for the callback again.
          */
-        g_mutex_lock(&mtx);
+
         data = g_hash_table_lookup(miss_ht, GUINT_TO_POINTER(effective_addr));
         if (data == NULL) {
             data = g_new0(InsnData, 1);
@@ -527,13 +549,20 @@ static void log_top_insns()
 
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
+    g_mutex_lock(&mtx);
     log_stats();
     log_top_insns();
 
     cache_free(dcache);
+    dcache = NULL;
+
     cache_free(icache);
+    icache = NULL;
 
     g_hash_table_destroy(miss_ht);
+    miss_ht = NULL;
+
+    g_mutex_unlock(&mtx);
 }
 
 static void policy_init()
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode

Quoted 5 lines · Mahmoud Mandour

Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.

What was your test case for this because I wonder if it would be worth coming up with one for check-tcg? From what I remember the race is in between preexit_cleanup and the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?

Quoted 90 lines

This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.

It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is already called once and reported the statistics.

---
 contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
contrib/plugins/cache.c +30 -1
index 695fb969dc..a452aba01c 100644
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
     g_mutex_lock(&mtx);
+    if (dcache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
+    if (icache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(icache, insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
             effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
         }
 
+        g_mutex_lock(&mtx);
+
+        /*
+         * is the plugin_exit callback called? If so, any further callback
+         * registration is useless as it won't get accounted for after calling
+         * plugin_exit once already, and also will use miss_ht after it's freed
+         */
+        if (miss_ht == NULL) {
+            g_mutex_unlock(&mtx);
+            return;
+        }
+
         /*
          * Instructions might get translated multiple times, we do not create
          * new entries for those instructions. Instead, we fetch the same
          * entry from the hash table and register it for the callback again.
          */
-        g_mutex_lock(&mtx);
+
         data = g_hash_table_lookup(miss_ht, GUINT_TO_POINTER(effective_addr));
         if (data == NULL) {
             data = g_new0(InsnData, 1);
@@ -527,13 +549,20 @@ static void log_top_insns()
 
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
+    g_mutex_lock(&mtx);
     log_stats();
     log_top_insns();
 
     cache_free(dcache);
+    dcache = NULL;
+
     cache_free(icache);
+    icache = NULL;
 
     g_hash_table_destroy(miss_ht);
+    miss_ht = NULL;
+
+    g_mutex_unlock(&mtx);
 }
 
 static void policy_init()
Signature · 1 line
Alex Bennée
#7 lore raw

Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode

Quoted 11 lines · Alex Bennée
Quoted 5 lines · Mahmoud Mandour

Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.

What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?

I think just any ad-hoc multithreaded execution will evoke the race pretty much
consistently.

Quoted 5 lines

From what I remember the race is
in between preexit_cleanup and the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?

I think so. This remedies the bug for this particular plugin and I think there
would be a better solution of course. However, I just can't ever get plugin_exit
callback to be called more than once so I think that's probably not the problem?

The problem is that we *use* the data in translation/mem_access/exec callbacks
after a plugin_exit call is already called (this can be easily verified by having a
boolean set to true once plugin_exit is called and then g_assert this boolean is
false in the callbacks)

Quoted 106 lines

contrib/plugins/cache.c static void vcpu_mem_access(unsigned int

Quoted 20 lines

This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.

It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is already called once and reported the statistics.

---
 contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
contrib/plugins/cache.c +0 -0
index 695fb969dc..a452aba01c 100644
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int

vcpu_index, qemu_plugin_meminfo_t info,

contrib/plugins/cache.c static void vcpu_mem_access(unsigned int · line 363

Quoted 1 line
contrib/plugins/cache.c +0 -0
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) :

vaddr;

contrib/plugins/cache.c static void vcpu_insn_exec(unsigned int vcpu_index,

Quoted 11 lines
contrib/plugins/cache.c +5 -0
     g_mutex_lock(&mtx);
+    if (dcache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index,

void *userdata)

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id,

Quoted 12 lines
contrib/plugins/cache.c +5 -0
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
+    if (icache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(icache, insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id,

struct qemu_plugin_tb *tb)

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)

Quoted 7 lines
contrib/plugins/cache.c +4 -0
             effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
         }
 
+        g_mutex_lock(&mtx);
+
+        /*
+         * is the plugin_exit callback called? If so, any further

callback

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 423

Quoted 1 line
contrib/plugins/cache.c +1 -0
+         * registration is useless as it won't get accounted for after

calling

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 424

Quoted 1 line
contrib/plugins/cache.c +1 -0
+         * plugin_exit once already, and also will use miss_ht after

it's freed

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)

Quoted 8 lines
contrib/plugins/cache.c +6 -0
+         */
+        if (miss_ht == NULL) {
+            g_mutex_unlock(&mtx);
+            return;
+        }
+
         /*
          * Instructions might get translated multiple times, we do not

create

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 433

Quoted 1 line
contrib/plugins/cache.c +0 -0
          * new entries for those instructions. Instead, we fetch the

same

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 434

Quoted 1 line
contrib/plugins/cache.c +0 -0
          * entry from the hash table and register it for the callback

again.

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)

Quoted 4 lines
contrib/plugins/cache.c +1 -1
          */
-        g_mutex_lock(&mtx);
+
         data = g_hash_table_lookup(miss_ht,

GUINT_TO_POINTER(effective_addr));

Quoted 13 lines
contrib/plugins/cache.c +3 -0
         if (data == NULL) {
             data = g_new0(InsnData, 1);
@@ -527,13 +549,20 @@ static void log_top_insns()
 
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
+    g_mutex_lock(&mtx);
     log_stats();
     log_top_insns();
 
     cache_free(dcache);
+    dcache = NULL;
+

contrib/plugins/cache.c static void log_top_insns() · line 568

Quoted 10 lines
contrib/plugins/cache.c +4 -0
     cache_free(icache);
+    icache = NULL;
 
     g_hash_table_destroy(miss_ht);
+    miss_ht = NULL;
+
+    g_mutex_unlock(&mtx);
 }
 
 static void policy_init()

--
Alex Bennée

#8 lore raw

Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode

Quoted 10 lines · Mahmoud Mandour

On Mon, Jul 19, 2021 at 11:48 AM Alex Bennée <[email protected]> wrote:

Quoted 5 lines · Mahmoud Mandour

Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.

Quoted 5 lines

What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?

I think just any ad-hoc multithreaded execution will evoke the race pretty much consistently.

I haven't managed to trigger it with testthread but of course my libcache is trying to to defend against it.

Quoted 6 lines

From what I remember the race is
in between preexit_cleanup and the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?

Quoted 8 lines

I think so. This remedies the bug for this particular plugin and I think there would be a better solution of course. However, I just can't ever get plugin_exit callback to be called more than once so I think that's probably not the problem?

The problem is that we *use* the data in translation/mem_access/exec callbacks after a plugin_exit call is already called (this can be easily verified by having a boolean set to true once plugin_exit is called and then g_assert this boolean is false in the callbacks)

We have mechanisms for safely unloading plugins during running so I think we should be able to do something cleanly here. I'll cook up an RFC.

Quoted 94 lines
Quoted 81 lines

This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.

It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is already called once and reported the statistics.

---
 contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
contrib/plugins/cache.c +26 -1
index 695fb969dc..a452aba01c 100644
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
     g_mutex_lock(&mtx);
+    if (dcache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
+    if (icache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(icache, insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
             effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
         }
 
+        g_mutex_lock(&mtx);
+
+        /*
+         * is the plugin_exit callback called? If so, any further callback
+         * registration is useless as it won't get accounted for after calling
+         * plugin_exit once already, and also will use miss_ht after it's freed
+         */
+        if (miss_ht == NULL) {
+            g_mutex_unlock(&mtx);
+            return;
+        }
+
         /*
          * Instructions might get translated multiple times, we do not create
          * new entries for those instructions. Instead, we fetch the same
          * entry from the hash table and register it for the callback again.
          */
-        g_mutex_lock(&mtx);
+
         data = g_hash_table_lookup(miss_ht, GUINT_TO_POINTER(effective_addr));
         if (data == NULL) {
             data = g_new0(InsnData, 1);
@@ -527,13 +549,20 @@ static void log_top_insns()
 
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
+    g_mutex_lock(&mtx);
     log_stats();
     log_top_insns();
 
     cache_free(dcache);
+    dcache = NULL;
+

contrib/plugins/cache.c static void log_top_insns() · line 568

Quoted 10 lines
contrib/plugins/cache.c +4 -0
     cache_free(icache);
+    icache = NULL;
 
     g_hash_table_destroy(miss_ht);
+    miss_ht = NULL;
+
+    g_mutex_unlock(&mtx);
 }
 
 static void policy_init()

--
Alex Bennée

Signature · 1 line
Alex Bennée
#9 lore raw

Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode

Quoted 18 lines · Alex Bennée
Quoted 1 line · Mahmoud Mandour

On Mon, Jul 19, 2021 at 11:48 AM Alex Bennée <[email protected]>

Quoted 7 lines · wrote:
Quoted 4 lines · Mahmoud Mandour

Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will

not

Quoted 5 lines
Quoted 1 line

be called after it and hence use already-freed data structures.

What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?

Quoted 7 lines
Quoted 1 line

I think just any ad-hoc multithreaded execution will evoke the race

pretty much

Quoted 1 line

consistently.

I haven't managed to trigger it with testthread but of course my libcache is trying to to defend against it.

https://pastebin.com/X4Xazemh
That's a minimal program that reproduces the bug consistently for me (to my observation, a simple
program that creates a bunch of threads that immediately exit does not produce the bug as frequently)

You can also make hotblocks produce a similar crash (but make sure to add a g_hash_table_destroy(hotblocks)
at the end of plugin_exit.)

Quoted 17 lines
Quoted 8 lines

From what I remember the race is
in between preexit_cleanup and the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?

I think so. This remedies the bug for this particular plugin and I think

there

Quoted 1 line

would be a better solution of course. However, I just can't ever get

plugin_exit

Quoted 1 line

callback to be called more than once so I think that's probably not the

problem?

Quoted 2 lines

The problem is that we *use* the data in translation/mem_access/exec

callbacks

Quoted 1 line

after a plugin_exit call is already called (this can be easily verified

Quoted 9 lines

by having a

Quoted 1 line

boolean set to true once plugin_exit is called and then g_assert this

boolean is

Quoted 1 line

false in the callbacks)

We have mechanisms for safely unloading plugins during running so I think we should be able to do something cleanly here. I'll cook up an RFC.

I'll be waiting for it. Note that as I think I mentioned in the cover letter, it's that plugin_uninstall
is probably problematic since it unregisters callbacks but already-fired callbacks will continue executing.
So calling plugin_uninstall at the end of plugin_exit does not relieve the bug...

Quoted 112 lines
Quoted 10 lines
Quoted 10 lines

This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.

It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is

already

Quoted 7 lines
Quoted 7 lines

called once and reported the statistics.

---
 contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
Quoted 5 lines

contrib/plugins/cache.c static void vcpu_mem_access(unsigned int

Quoted 5 lines
contrib/plugins/cache.c +0 -0
index 695fb969dc..a452aba01c 100644
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int

vcpu_index, qemu_plugin_meminfo_t info,

Quoted 1 line

contrib/plugins/cache.c static void vcpu_mem_access(unsigned int · line 363

Quoted 1 line
contrib/plugins/cache.c +0 -0
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) :

vaddr;

Quoted 11 lines

contrib/plugins/cache.c static void vcpu_insn_exec(unsigned int

Quoted 11 lines
contrib/plugins/cache.c +5 -0
     g_mutex_lock(&mtx);
+    if (dcache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int

vcpu_index, void *userdata)

Quoted 12 lines

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id,

Quoted 12 lines
contrib/plugins/cache.c +5 -0
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
+    if (icache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(icache, insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id,

struct qemu_plugin_tb *tb)

Quoted 7 lines

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)

Quoted 7 lines
contrib/plugins/cache.c +4 -0
             effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
         }
 
+        g_mutex_lock(&mtx);
+
+        /*
+         * is the plugin_exit callback called? If so, any further

callback

Quoted 1 line

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 423

Quoted 1 line
contrib/plugins/cache.c +1 -0
+         * registration is useless as it won't get accounted for

after calling

Quoted 1 line

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 424

Quoted 1 line
contrib/plugins/cache.c +1 -0
+         * plugin_exit once already, and also will use miss_ht after

it's freed

Quoted 8 lines

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)

Quoted 8 lines
contrib/plugins/cache.c +6 -0
+         */
+        if (miss_ht == NULL) {
+            g_mutex_unlock(&mtx);
+            return;
+        }
+
         /*
          * Instructions might get translated multiple times, we do

not create

Quoted 1 line

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 433

Quoted 1 line
contrib/plugins/cache.c +0 -0
          * new entries for those instructions. Instead, we fetch the

same

Quoted 1 line

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 434

Quoted 1 line
contrib/plugins/cache.c +0 -0
          * entry from the hash table and register it for the

callback again.

Quoted 4 lines

contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)

Quoted 4 lines
contrib/plugins/cache.c +1 -1
          */
-        g_mutex_lock(&mtx);
+
         data = g_hash_table_lookup(miss_ht,

GUINT_TO_POINTER(effective_addr));

Quoted 22 lines
Quoted 22 lines
contrib/plugins/cache.c +7 -0
         if (data == NULL) {
             data = g_new0(InsnData, 1);
@@ -527,13 +549,20 @@ static void log_top_insns()
 
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
+    g_mutex_lock(&mtx);
     log_stats();
     log_top_insns();
 
     cache_free(dcache);
+    dcache = NULL;
+
     cache_free(icache);
+    icache = NULL;
 
     g_hash_table_destroy(miss_ht);
+    miss_ht = NULL;
+
+    g_mutex_unlock(&mtx);
 }
 
Quoted 4 lines

contrib/plugins/cache.c static void log_top_insns() · line 568

Quoted 1 line
contrib/plugins/cache.c +0 -0
 static void policy_init()

--
Alex Bennée

--
Alex Bennée

#10 lore raw

Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode

Quoted 23 lines · Mahmoud Mandour

On Mon, Jul 19, 2021 at 1:08 PM Alex Bennée <[email protected]> wrote:

Quoted 10 lines · Mahmoud Mandour

On Mon, Jul 19, 2021 at 11:48 AM Alex Bennée <[email protected]> wrote:

Quoted 5 lines · Mahmoud Mandour

Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.

Quoted 5 lines

What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?

I think just any ad-hoc multithreaded execution will evoke the race pretty much consistently.

I haven't managed to trigger it with testthread but of course my libcache is trying to to defend against it.

Quoted 7 lines

https://pastebin.com/X4Xazemh
That's a minimal program that reproduces the bug consistently for me (to my observation, a simple program that creates a bunch of threads that immediately exit does not produce the bug as frequently)

You can also make hotblocks produce a similar crash (but make sure to add a g_hash_table_destroy(hotblocks) at the end of plugin_exit.)

Thanks, I'll give that a try.

Quoted 16 lines
Quoted 7 lines

From what I remember the race is
in between preexit_cleanup and the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?

Quoted 8 lines

I think so. This remedies the bug for this particular plugin and I think there would be a better solution of course. However, I just can't ever get plugin_exit callback to be called more than once so I think that's probably not the problem?

The problem is that we *use* the data in translation/mem_access/exec callbacks after a plugin_exit call is already called (this can be easily verified by having a boolean set to true once plugin_exit is called and then g_assert this boolean is false in the callbacks)

Quoted 8 lines

We have mechanisms for safely unloading plugins during running so I think we should be able to do something cleanly here. I'll cook up an RFC.

I'll be waiting for it. Note that as I think I mentioned in the cover letter, it's that plugin_uninstall is probably problematic since it unregisters callbacks but already-fired callbacks will continue executing. So calling plugin_uninstall at the end of plugin_exit does not relieve
the bug...

That's OK - the plugin_uninstall contract explicitly says that callbacks may occur until the callback is called.

Quoted 99 lines
Quoted 92 lines
Quoted 92 lines

This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.

It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is already called once and reported the statistics.

---
 contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
contrib/plugins/cache.c +30 -1
index 695fb969dc..a452aba01c 100644
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
     g_mutex_lock(&mtx);
+    if (dcache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(dcache, effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
+    if (icache == NULL) {
+        g_mutex_unlock(&mtx);
+        return;
+    }
+
     if (!access_cache(icache, insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
             effective_addr = (uint64_t) qemu_plugin_insn_vaddr(insn);
         }
 
+        g_mutex_lock(&mtx);
+
+        /*
+         * is the plugin_exit callback called? If so, any further callback
+         * registration is useless as it won't get accounted for after calling
+         * plugin_exit once already, and also will use miss_ht after it's freed
+         */
+        if (miss_ht == NULL) {
+            g_mutex_unlock(&mtx);
+            return;
+        }
+
         /*
          * Instructions might get translated multiple times, we do not create
          * new entries for those instructions. Instead, we fetch the same
          * entry from the hash table and register it for the callback again.
          */
-        g_mutex_lock(&mtx);
+
         data = g_hash_table_lookup(miss_ht, GUINT_TO_POINTER(effective_addr));
         if (data == NULL) {
             data = g_new0(InsnData, 1);
@@ -527,13 +549,20 @@ static void log_top_insns()
 
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
+    g_mutex_lock(&mtx);
     log_stats();
     log_top_insns();
 
     cache_free(dcache);
+    dcache = NULL;
+
     cache_free(icache);
+    icache = NULL;
 
     g_hash_table_destroy(miss_ht);
+    miss_ht = NULL;
+
+    g_mutex_unlock(&mtx);
 }
 
Quoted 4 lines

contrib/plugins/cache.c static void log_top_insns() · line 568

Quoted 1 line
contrib/plugins/cache.c +0 -0
 static void policy_init()

--
Alex Bennée

--
Alex Bennée

Signature · 1 line
Alex Bennée
#11 lore raw

PATCH 4/6 plugins/cache: Supported multicore cache modelling

In reply to Mahmoud Mandour

Multicore L1 cache modelling is introduced and is supported for both full system emulation and linux-user.

For full-system emulation, L1 icache and dcache are maintained for each available core, since this information is exposed to the plugin through qemu_plugin_n_vcpus().

For linux-user, a static number of cores is assumed (default 1 core, and can be provided as a plugin argument cores=N). Every memory access goes through one of these caches, this approach is taken as it's somewhat akin to what happens on real setup, where a program that dispatches more threads than the available cores, they'll thrash each other

---
 contrib/plugins/cache.c | 156 ++++++++++++++++++++++++++++++----------
 1 file changed, 117 insertions(+), 39 deletions(-)
contrib/plugins/cache.c +117 -39
index a452aba01c..60f7be208b 100644
@@ -23,11 +23,6 @@ static GRand *rng;
 static int limit;
 static bool sys;
 
-static uint64_t dmem_accesses;
-static uint64_t dmisses;
-
-static uint64_t imem_accesses;
-static uint64_t imisses;
 
 enum EvictionPolicy {
     LRU,
@@ -90,13 +85,22 @@ typedef struct {
     uint64_t imisses;
 } InsnData;
 
+typedef struct {
+    uint64_t dmem_accesses;
+    uint64_t dmisses;
+    uint64_t imem_accesses;
+    uint64_t imisses;
+} CoreStats;
+
 void (*update_hit)(Cache *cache, int set, int blk);
 void (*update_miss)(Cache *cache, int set, int blk);
 
 void (*metadata_init)(Cache *cache);
 void (*metadata_destroy)(Cache *cache);
 
-Cache *dcache, *icache;
+static int cores;
+CoreStats *stats;
+Cache **dcaches, **icaches;
 
 static int pow_of_two(int num)
 {
@@ -233,10 +237,6 @@ static bool bad_cache_params(int blksize, int assoc, int cachesize)
 
 static Cache *cache_init(int blksize, int assoc, int cachesize)
 {
-    if (bad_cache_params(blksize, assoc, cachesize)) {
-        return NULL;
-    }
-
     Cache *cache;
     int i;
     uint64_t blk_mask;
@@ -263,6 +263,24 @@ static Cache *cache_init(int blksize, int assoc, int cachesize)
     return cache;
 }
 
+static Cache **caches_init(int blksize, int assoc, int cachesize)
+{
+    Cache **caches;
+    int i;
+
+    if (bad_cache_params(blksize, assoc, cachesize)) {
+        return NULL;
+    }
+
+    caches = g_new(Cache *, cores);
+
+    for (i = 0; i < cores; i++) {
+        caches[i] = cache_init(blksize, assoc, cachesize);
+    }
+
+    return caches;
+}
+
 static int get_invalid_block(Cache *cache, uint64_t set)
 {
     int i;
@@ -353,6 +371,7 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
 {
     uint64_t effective_addr;
     struct qemu_plugin_hwaddr *hwaddr;
+    int cache_idx;
     InsnData *insn;
 
     hwaddr = qemu_plugin_get_hwaddr(info, vaddr);
@@ -363,17 +382,24 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
     g_mutex_lock(&mtx);
-    if (dcache == NULL) {
+    cache_idx = vcpu_index % cores;
+    if (dcaches[cache_idx] == NULL) {
         g_mutex_unlock(&mtx);
         return;
     }
 
-    if (!access_cache(dcache, effective_addr)) {
+    if (!access_cache(dcaches[cache_idx], effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
-        dmisses++;
+        if (cores > 1) {
+            stats[cores].dmisses++;
+        }
+        stats[cache_idx].dmisses++;
+    }
+    if (cores > 1) {
+        stats[cores].dmem_accesses++;
     }
-    dmem_accesses++;
+    stats[cache_idx].dmem_accesses++;
     g_mutex_unlock(&mtx);
 }
 
@@ -381,21 +407,29 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
 {
     uint64_t insn_addr;
     InsnData *insn;
+    uint64_t cache_idx;
 
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
-    if (icache == NULL) {
+    cache_idx = vcpu_index % cores;
+    if (icaches[cache_idx] == NULL) {
         g_mutex_unlock(&mtx);
         return;
     }
 
-    if (!access_cache(icache, insn_addr)) {
+    if (!access_cache(icaches[cache_idx], insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;
-        imisses++;
+        if (cores > 1) {
+            stats[cores].imisses++;
+        }
+        stats[cache_idx].imisses++;
     }
-    imem_accesses++;
+    if (cores > 1) {
+        stats[cores].imem_accesses++;
+    }
+    stats[cache_idx].imem_accesses++;
     g_mutex_unlock(&mtx);
 }
 
@@ -475,6 +509,22 @@ static void cache_free(Cache *cache)
     g_free(cache);
 }
 
+/*
+ * caches_free(): free an array of Cache structs.
+ * @caches: caches to free
+ *
+ * Calling cache_free for each cache in @caches, and then NULL-ify them so that
+ * we mark them as freed, other async callbacks can check to see whether a cache
+ * is freed already or not by checking against NULL.
+ */
+static void caches_free(Cache **caches)
+{
+    for (int i = 0; i < cores; i++) {
+        cache_free(caches[i]);
+        caches[i] = NULL;
+    }
+}
+
 static int dcmp(gconstpointer a, gconstpointer b)
 {
     InsnData *insn_a = (InsnData *) a;
@@ -493,19 +543,38 @@ static int icmp(gconstpointer a, gconstpointer b)
 
 static void log_stats()
 {
-    g_autoptr(GString) rep = g_string_new("");
-    g_string_append_printf(rep,
-        "Data accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n",
-        dmem_accesses,
-        dmisses,
-        ((double) dmisses / (double) dmem_accesses) * 100.0);
-
-    g_string_append_printf(rep,
-        "Instruction accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n",
-        imem_accesses,
-        imisses,
-        ((double) imisses / (double) imem_accesses) * 100.0);
+    int i, iters;
+    CoreStats cs;
+    double dmiss_rate, imiss_rate;
+
+    g_autoptr(GString) rep = g_string_new("core #, data accesses, data misses,"
+                                          " dmiss rate, insn accesses,"
+                                          " insn misses, imiss rate\n");
+
+    /* Only iterate and print a sum row if cores > 1 */
+    iters = cores == 1 ? 1 : cores + 1;
+    for (i = 0; i < iters; i++) {
+        cs = stats[i];
+        dmiss_rate = ((double) cs.dmisses) / (cs.dmem_accesses) * 100.0;
+        imiss_rate = ((double) cs.imisses) / (cs.imem_accesses) * 100.0;
+
+        if (i == cores) {
+            g_string_append_printf(rep, "%-8s", "sum");
+        } else {
+            g_string_append_printf(rep, "%-8d", i);
+        }
+
+        g_string_append_printf(rep, "%-14lu %-12lu %9.4lf%%  %-14lu %-12lu"
+                               " %9.4lf%%\n",
+                               cs.dmem_accesses,
+                               cs.dmisses,
+                               cs.dmem_accesses ? dmiss_rate : 0.0,
+                               cs.imem_accesses,
+                               cs.imisses,
+                               cs.imem_accesses ? imiss_rate : 0.0);
+    }
 
+    g_string_append(rep, "\n");
     qemu_plugin_outs(rep->str);
 }
 
@@ -553,15 +622,14 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)
     log_stats();
     log_top_insns();
 
-    cache_free(dcache);
-    dcache = NULL;
-
-    cache_free(icache);
-    icache = NULL;
+    caches_free(dcaches);
+    caches_free(icaches);
 
     g_hash_table_destroy(miss_ht);
     miss_ht = NULL;
 
+    g_free(stats);
+
     g_mutex_unlock(&mtx);
 }
 
@@ -608,6 +676,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
 
     policy = LRU;
 
+    cores = sys ? qemu_plugin_n_vcpus() : 1;
+
     for (i = 0; i < argc; i++) {
         char *opt = argv[i];
         if (g_str_has_prefix(opt, "iblksize=")) {
@@ -624,6 +694,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
             dcachesize = g_ascii_strtoll(opt + 11, NULL, 10);
         } else if (g_str_has_prefix(opt, "limit=")) {
             limit = g_ascii_strtoll(opt + 6, NULL, 10);
+        } else if (g_str_has_prefix(opt, "cores=")) {
+            cores = g_ascii_strtoll(opt + 6, NULL, 10);
         } else if (g_str_has_prefix(opt, "evict=")) {
             gchar *p = opt + 6;
             if (g_strcmp0(p, "rand") == 0) {
@@ -644,22 +716,28 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
 
     policy_init();
 
-    dcache = cache_init(dblksize, dassoc, dcachesize);
-    if (!dcache) {
+    dcaches = caches_init(dblksize, dassoc, dcachesize);
+    if (!dcaches) {
         const char *err = cache_config_error(dblksize, dassoc, dcachesize);
         fprintf(stderr, "dcache cannot be constructed from given parameters\n");
         fprintf(stderr, "%s\n", err);
         return -1;
     }
 
-    icache = cache_init(iblksize, iassoc, icachesize);
-    if (!icache) {
+    icaches = caches_init(iblksize, iassoc, icachesize);
+    if (!icaches) {
         const char *err = cache_config_error(iblksize, iassoc, icachesize);
         fprintf(stderr, "icache cannot be constructed from given parameters\n");
         fprintf(stderr, "%s\n", err);
         return -1;
     }
 
+    /*
+     * plus one to save the sum in. If only one core is used then no need to
+     * get an auxiliary struct.
+     */
+    stats = g_new0(CoreStats, cores == 1 ? 1 : cores + 1);
+
     qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans);
     qemu_plugin_register_atexit_cb(id, plugin_exit, NULL);
 
Signature · 1 line
2.25.1
#12 lore raw

Re: [PATCH 4/6] plugins/cache: Supported multicore cache modelling

Quoted 104 lines · Mahmoud Mandour

Multicore L1 cache modelling is introduced and is supported for both full system emulation and linux-user.

For full-system emulation, L1 icache and dcache are maintained for each available core, since this information is exposed to the plugin through qemu_plugin_n_vcpus().

For linux-user, a static number of cores is assumed (default 1 core, and can be provided as a plugin argument cores=N). Every memory access goes through one of these caches, this approach is taken as it's somewhat akin to what happens on real setup, where a program that dispatches more threads than the available cores, they'll thrash each other

---
 contrib/plugins/cache.c | 156 ++++++++++++++++++++++++++++++----------
 1 file changed, 117 insertions(+), 39 deletions(-)
contrib/plugins/cache.c +29 -10
index a452aba01c..60f7be208b 100644
@@ -23,11 +23,6 @@ static GRand *rng;
 static int limit;
 static bool sys;
 
-static uint64_t dmem_accesses;
-static uint64_t dmisses;
-
-static uint64_t imem_accesses;
-static uint64_t imisses;
 
 enum EvictionPolicy {
     LRU,
@@ -90,13 +85,22 @@ typedef struct {
     uint64_t imisses;
 } InsnData;
 
+typedef struct {
+    uint64_t dmem_accesses;
+    uint64_t dmisses;
+    uint64_t imem_accesses;
+    uint64_t imisses;
+} CoreStats;
+
 void (*update_hit)(Cache *cache, int set, int blk);
 void (*update_miss)(Cache *cache, int set, int blk);
 
 void (*metadata_init)(Cache *cache);
 void (*metadata_destroy)(Cache *cache);
 
-Cache *dcache, *icache;
+static int cores;
+CoreStats *stats;
+Cache **dcaches, **icaches;
 
 static int pow_of_two(int num)
 {
@@ -233,10 +237,6 @@ static bool bad_cache_params(int blksize, int assoc, int cachesize)
 
 static Cache *cache_init(int blksize, int assoc, int cachesize)
 {
-    if (bad_cache_params(blksize, assoc, cachesize)) {
-        return NULL;
-    }
-
     Cache *cache;
     int i;
     uint64_t blk_mask;
@@ -263,6 +263,24 @@ static Cache *cache_init(int blksize, int assoc, int cachesize)
     return cache;
 }
 
+static Cache **caches_init(int blksize, int assoc, int cachesize)
+{
+    Cache **caches;
+    int i;
+
+    if (bad_cache_params(blksize, assoc, cachesize)) {
+        return NULL;
+    }
+
+    caches = g_new(Cache *, cores);
+
+    for (i = 0; i < cores; i++) {
+        caches[i] = cache_init(blksize, assoc, cachesize);
+    }
+
+    return caches;
+}
+
 static int get_invalid_block(Cache *cache, uint64_t set)
 {
     int i;
@@ -353,6 +371,7 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
 {
     uint64_t effective_addr;
     struct qemu_plugin_hwaddr *hwaddr;
+    int cache_idx;
     InsnData *insn;
 
     hwaddr = qemu_plugin_get_hwaddr(info, vaddr);

contrib/plugins/cache.c static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, · line 389

Quoted 9 lines
contrib/plugins/cache.c +2 -1
@@ -363,17 +382,24 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,
     effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) : vaddr;
 
     g_mutex_lock(&mtx);
-    if (dcache == NULL) {
+    cache_idx = vcpu_index % cores;
+    if (dcaches[cache_idx] == NULL) {
         g_mutex_unlock(&mtx);
         return;
     }

I was hoping we could avoid this with some careful atomic/memory barrier use given dcaches is setup on init. However it's the exit case that gets us which further increases my desire to fix the exit race in linux-user and make sure callbacks don't occur in this case.

Quoted 39 lines
contrib/plugins/cache.c +13 -5
-    if (!access_cache(dcache, effective_addr)) {
+    if (!access_cache(dcaches[cache_idx], effective_addr)) {
         insn = (InsnData *) userdata;
         insn->dmisses++;
-        dmisses++;
+        if (cores > 1) {
+            stats[cores].dmisses++;
+        }
+        stats[cache_idx].dmisses++;
+    }
+    if (cores > 1) {
+        stats[cores].dmem_accesses++;
     }
-    dmem_accesses++;
+    stats[cache_idx].dmem_accesses++;
     g_mutex_unlock(&mtx);
 }
 
@@ -381,21 +407,29 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)
 {
     uint64_t insn_addr;
     InsnData *insn;
+    uint64_t cache_idx;
 
     g_mutex_lock(&mtx);
     insn_addr = ((InsnData *) userdata)->addr;
 
-    if (icache == NULL) {
+    cache_idx = vcpu_index % cores;
+    if (icaches[cache_idx] == NULL) {
         g_mutex_unlock(&mtx);
         return;
     }
 
-    if (!access_cache(icache, insn_addr)) {
+    if (!access_cache(icaches[cache_idx], insn_addr)) {
         insn = (InsnData *) userdata;
         insn->imisses++;

contrib/plugins/cache.c static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata) · line 431

Quoted 10 lines
contrib/plugins/cache.c +7 -2
-        imisses++;
+        if (cores > 1) {
+            stats[cores].imisses++;
+        }
+        stats[cache_idx].imisses++;
     }
-    imem_accesses++;
+    if (cores > 1) {
+        stats[cores].imem_accesses++;
+    }

I'd much rather avoid summing the stats during execution than at the end when we can add things simply.

Quoted 136 lines
contrib/plugins/cache.c +60 -21
+    stats[cache_idx].imem_accesses++;
     g_mutex_unlock(&mtx);
 }
 
@@ -475,6 +509,22 @@ static void cache_free(Cache *cache)
     g_free(cache);
 }
 
+/*
+ * caches_free(): free an array of Cache structs.
+ * @caches: caches to free
+ *
+ * Calling cache_free for each cache in @caches, and then NULL-ify them so that
+ * we mark them as freed, other async callbacks can check to see whether a cache
+ * is freed already or not by checking against NULL.
+ */
+static void caches_free(Cache **caches)
+{
+    for (int i = 0; i < cores; i++) {
+        cache_free(caches[i]);
+        caches[i] = NULL;
+    }
+}
+
 static int dcmp(gconstpointer a, gconstpointer b)
 {
     InsnData *insn_a = (InsnData *) a;
@@ -493,19 +543,38 @@ static int icmp(gconstpointer a, gconstpointer b)
 
 static void log_stats()
 {
-    g_autoptr(GString) rep = g_string_new("");
-    g_string_append_printf(rep,
-        "Data accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n",
-        dmem_accesses,
-        dmisses,
-        ((double) dmisses / (double) dmem_accesses) * 100.0);
-
-    g_string_append_printf(rep,
-        "Instruction accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n",
-        imem_accesses,
-        imisses,
-        ((double) imisses / (double) imem_accesses) * 100.0);
+    int i, iters;
+    CoreStats cs;
+    double dmiss_rate, imiss_rate;
+
+    g_autoptr(GString) rep = g_string_new("core #, data accesses, data misses,"
+                                          " dmiss rate, insn accesses,"
+                                          " insn misses, imiss rate\n");
+
+    /* Only iterate and print a sum row if cores > 1 */
+    iters = cores == 1 ? 1 : cores + 1;
+    for (i = 0; i < iters; i++) {
+        cs = stats[i];
+        dmiss_rate = ((double) cs.dmisses) / (cs.dmem_accesses) * 100.0;
+        imiss_rate = ((double) cs.imisses) / (cs.imem_accesses) * 100.0;
+
+        if (i == cores) {
+            g_string_append_printf(rep, "%-8s", "sum");
+        } else {
+            g_string_append_printf(rep, "%-8d", i);
+        }
+
+        g_string_append_printf(rep, "%-14lu %-12lu %9.4lf%%  %-14lu %-12lu"
+                               " %9.4lf%%\n",
+                               cs.dmem_accesses,
+                               cs.dmisses,
+                               cs.dmem_accesses ? dmiss_rate : 0.0,
+                               cs.imem_accesses,
+                               cs.imisses,
+                               cs.imem_accesses ? imiss_rate : 0.0);
+    }
 
+    g_string_append(rep, "\n");
     qemu_plugin_outs(rep->str);
 }
 
@@ -553,15 +622,14 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)
     log_stats();
     log_top_insns();
 
-    cache_free(dcache);
-    dcache = NULL;
-
-    cache_free(icache);
-    icache = NULL;
+    caches_free(dcaches);
+    caches_free(icaches);
 
     g_hash_table_destroy(miss_ht);
     miss_ht = NULL;
 
+    g_free(stats);
+
     g_mutex_unlock(&mtx);
 }
 
@@ -608,6 +676,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
 
     policy = LRU;
 
+    cores = sys ? qemu_plugin_n_vcpus() : 1;
+
     for (i = 0; i < argc; i++) {
         char *opt = argv[i];
         if (g_str_has_prefix(opt, "iblksize=")) {
@@ -624,6 +694,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
             dcachesize = g_ascii_strtoll(opt + 11, NULL, 10);
         } else if (g_str_has_prefix(opt, "limit=")) {
             limit = g_ascii_strtoll(opt + 6, NULL, 10);
+        } else if (g_str_has_prefix(opt, "cores=")) {
+            cores = g_ascii_strtoll(opt + 6, NULL, 10);
         } else if (g_str_has_prefix(opt, "evict=")) {
             gchar *p = opt + 6;
             if (g_strcmp0(p, "rand") == 0) {
@@ -644,22 +716,28 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
 
     policy_init();
 
-    dcache = cache_init(dblksize, dassoc, dcachesize);
-    if (!dcache) {
+    dcaches = caches_init(dblksize, dassoc, dcachesize);
+    if (!dcaches) {
         const char *err = cache_config_error(dblksize, dassoc, dcachesize);
         fprintf(stderr, "dcache cannot be constructed from given parameters\n");
         fprintf(stderr, "%s\n", err);
         return -1;
     }
 
-    icache = cache_init(iblksize, iassoc, icachesize);
-    if (!icache) {
+    icaches = caches_init(iblksize, iassoc, icachesize);
+    if (!icaches) {
         const char *err = cache_config_error(iblksize, iassoc, icachesize);
         fprintf(stderr, "icache cannot be constructed from given parameters\n");

contrib/plugins/cache.c int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, · line 740

Quoted 10 lines
contrib/plugins/cache.c +6 -0
         fprintf(stderr, "%s\n", err);
         return -1;
     }
 
+    /*
+     * plus one to save the sum in. If only one core is used then no need to
+     * get an auxiliary struct.
+     */
+    stats = g_new0(CoreStats, cores == 1 ? 1 : cores + 1);
+

See above, lets keep the sum as a post processing step.

Quoted 2 lines
contrib/plugins/cache.c +0 -0
     qemu_plugin_register_vcpu_tb_trans_cb(id, vcpu_tb_trans);
     qemu_plugin_register_atexit_cb(id, plugin_exit, NULL);
Signature · 1 line
Alex Bennée
#13 lore raw

PATCH 5/6 docs/devel/tcg-plugins: added cores arg to cache plugin

In reply to Mahmoud Mandour
---
 docs/devel/tcg-plugins.rst | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
docs/devel/tcg-plugins.rst +8 -5
index 595b8e0ea4..370c11373f 100644
@@ -330,11 +330,8 @@ configuration when a given working set is run::
 
 will report the following::
 
-    Data accesses: 996479, Misses: 507
-    Miss rate: 0.050879%
-
-    Instruction accesses: 2641737, Misses: 18617
-    Miss rate: 0.704726%
+    core #, data accesses, data misses, dmiss rate, insn accesses, insn misses, imiss rate
+    0       996695         508             0.0510%  2642799        18617           0.7044%
 
     address, data misses, instruction
     0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx)
@@ -378,3 +375,9 @@ The plugin has a number of arguments, all of them are optional:
   Sets the eviction policy to POLICY. Available policies are: :code:`lru`,
   :code:`fifo`, and :code:`rand`. The plugin will use the specified policy for
   both instruction and data caches. (default: POLICY = :code:`lru`)
+
+  * arg="cores=N"
+
+  Sets the number of cores for which we maintain separate icache and dcache.
+  (default: for linux-user, N = 1, for full system emulation: N = cores
+  available to guest)
Signature · 1 line
2.25.1
#14 lore raw

PATCH 6/6 plugins/cache: Fixed "function decl. is not a prototype" warnings

In reply to Mahmoud Mandour
---
 contrib/plugins/cache.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
contrib/plugins/cache.c +3 -3
index 60f7be208b..f82a8310dc 100644
@@ -541,7 +541,7 @@ static int icmp(gconstpointer a, gconstpointer b)
     return insn_a->imisses < insn_b->imisses ? 1 : -1;
 }
 
-static void log_stats()
+static void log_stats(void)
 {
     int i, iters;
     CoreStats cs;
@@ -578,7 +578,7 @@ static void log_stats()
     qemu_plugin_outs(rep->str);
 }
 
-static void log_top_insns()
+static void log_top_insns(void)
 {
     int i;
     GList *curr, *miss_insns;
@@ -633,7 +633,7 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)
     g_mutex_unlock(&mtx);
 }
 
-static void policy_init()
+static void policy_init(void)
 {
     switch (policy) {
     case LRU:
Signature · 1 line
2.25.1
Applied
#16 lore raw
In reply to Mahmoud Mandour
Quoted 4 lines · Mahmoud Mandour

Hello,

This series introduce some minor improvements/bug fixes in the cache plugins and multicore cache modelling.

Queued patches 1,2 and 6 for-6.1/fixes-for-rc1 as they are bug fixes, thanks.

Signature · 1 line
Alex Bennée