plugins/cache: multicore cache emulation and minor
cover plugins/cache: multicore cache emulation and minor
Hello,
This series introduce some minor improvements/bug fixes in the cache plugins and multicore cache modelling.
One prominent bug is the use-after-free bug induced by linux-user multithreaded programs. Since plugin_exit is not guaranteed to be called after all callbacks, it may free data that subsequent callbacks may try to use.
Trying to uninstall the plugin after exiting does not solve the problem since it will unregister the callbacks but it won't prevent the already-fired callback instances from continuing.
To mitigate this issue, the data is NULLified on exitting so that callbacks can check whether plugin_exit has been called already or not.
Also, raising the levels of warnings (by the time this is sent, it's not yet upstreamed but it's in the process of getting merged) induced some warnings, this is fixed in its own patch.
Mahmoud Mandour (6):
plugins/cache: Fixed a bug with destroying FIFO metadata plugins/cache: limited the scope of a mutex lock plugins/cache: Fixed a use-after-free bug with multithreaded usermode plugins/cache: Supported multicore cache modelling docs/devel/tcg-plugins: added cores arg to cache plugin plugins/cache: Fixed "function decl. is not a prototype" warnings
contrib/plugins/cache.c | 188 +++++++++++++++++++++++++++++-------- docs/devel/tcg-plugins.rst | 13 ++- 2 files changed, 155 insertions(+), 46 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/6 plugins/cache: Fixed a bug with destroying FIFO metadata
This manifests itself when associativity degree is greater than the number of sets and FIFO is used, otherwise it's also a memory leak whenever FIFO was used.
--- contrib/plugins/cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
contrib/plugins/cache.c +1 -1
@@ -200,7 +200,7 @@ static void fifo_destroy(Cache *cache) {int i; -for (i =0 ; i < cache->assoc; i++) { +for (i =0 ; i < cache->num_sets; i++) {g_queue_free (cache->sets[i].fifo_queue); } }
Signature · 1 line
2.25.1
Re: [PATCH 1/6] plugins/cache: Fixed a bug with destroying FIFO metadata
Quoted 5 lines · Mahmoud Mandour
This manifests itself when associativity degree is greater than the number of sets and FIFO is used, otherwise it's also a memory leak whenever FIFO was used.
Signed-off-by Mahmoud Mandour <[email protected]>
Quoted 17 lines
--- contrib/plugins/cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)contrib/plugins/cache.c +1
-1@@ -200,7 +200,7 @@ static void fifo_destroy(Cache *cache) {int i; -for (i =0 ; i < cache->assoc; i++) { +for (i =0 ; i < cache->num_sets; i++) {g_queue_free (cache->sets[i].fifo_queue); } }
Signature · 1 line
Alex Bennée
PATCH 2/6 plugins/cache: limited the scope of a mutex lock
In reply to Mahmoud MandourIt's not necessary to lock the address translation portion of the vcpu_mem_access callback.
--- contrib/plugins/cache.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
contrib/plugins/cache.c +1 -2
@@ -355,15 +355,14 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,struct qemu_plugin_hwaddr *hwaddr;InsnData *insn; -g_mutex_lock (&mtx); hwaddr =qemu_plugin_get_hwaddr (info, vaddr);if (hwaddr &&qemu_plugin_hwaddr_is_io (hwaddr)) { -g_mutex_unlock (&mtx);return ; } effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr; +g_mutex_lock (&mtx);if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;
Signature · 1 line
2.25.1
Re: [PATCH 2/6] plugins/cache: limited the scope of a mutex lock
Quoted 19 lines · Mahmoud Mandour
It's not necessary to lock the address translation portion of the
vcpu_mem_accesscallback.Signed-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)contrib/plugins/cache.c +0
-1@@ -355,15 +355,14 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info,struct qemu_plugin_hwaddr *hwaddr;InsnData *insn; -g_mutex_lock (&mtx); hwaddr =qemu_plugin_get_hwaddr (info, vaddr);if (hwaddr &&qemu_plugin_hwaddr_is_io (hwaddr)) {
contrib/plugins/cache.c static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, · line 368
Quoted 10 lines
contrib/plugins/cache.c +1
-1-g_mutex_unlock (&mtx);return ; } effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr; +g_mutex_lock (&mtx);if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;
This is fine, but I see an exit leg creeps in later which I think we can eliminate. I'll comment there:
Signature · 1 line
Alex Bennée
PATCH 3/6 plugins/cache: Fixed a use-after-free bug with multithreaded usermode
In reply to Mahmoud MandourSince callbacks may be interleaved because of multithreaded execution, we should not make assumptions about plugin_exit either. The problem with plugin_exit is that it frees shared data structures (caches and miss_ht hash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.
This is mitigated in this commit by synchronizing the call to plugin_exit through locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.
It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since plugin_exit is already called once and reported the statistics.
--- contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
contrib/plugins/cache.c +30 -1
@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr;g_mutex_lock (&mtx); +if (dcache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;
@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; +if (icache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (icache, insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;
@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) effective_addr = (uint64_t )qemu_plugin_insn_vaddr (insn); } +g_mutex_lock (&mtx); + +/* +* is the plugin_exit callback called? If so, any further callback +* registration is useless as it won't get accounted for after calling +* plugin_exit once already, and also will use miss_ht after it's freed +*/ +if (miss_ht ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +/* * Instructions might get translated multiple times, we do not create * new entries for those instructions. Instead, we fetch the same * entry from the hash table and register it for the callback again. */ -g_mutex_lock (&mtx); + data =g_hash_table_lookup (miss_ht,GUINT_TO_POINTER (effective_addr));if (data ==NULL ) { data =g_new0 (InsnData,1 );
@@ -527,13 +549,20 @@ static void log_top_insns()static void plugin_exit (qemu_plugin_id_t id,void *p) { +g_mutex_lock (&mtx);log_stats ();log_top_insns ();cache_free (dcache); + dcache =NULL ; +cache_free (icache); + icache =NULL ;g_hash_table_destroy (miss_ht); + miss_ht =NULL ; + +g_mutex_unlock (&mtx); }static void policy_init ()
Signature · 1 line
2.25.1
Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode
Quoted 5 lines · Mahmoud Mandour
Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about
plugin_exiteither. The problem withplugin_exitis that it frees shared data structures (caches andmiss_hthash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.
What was your test case for this because I wonder if it would be worth coming up with one for check-tcg? From what I remember the race is in between preexit_cleanup and the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?
Quoted 90 lines
This is mitigated in this commit by synchronizing the call to
plugin_exitthrough locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since
plugin_exitis already called once and reported the statistics.Signed-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)contrib/plugins/cache.c +30
-1@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr;g_mutex_lock (&mtx); +if (dcache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; +if (icache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (icache, insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) effective_addr = (uint64_t )qemu_plugin_insn_vaddr (insn); } +g_mutex_lock (&mtx); + +/* +* is the plugin_exit callback called? If so, any further callback +* registration is useless as it won't get accounted for after calling +* plugin_exit once already, and also will use miss_ht after it's freed +*/ +if (miss_ht ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +/* * Instructions might get translated multiple times, we do not create * new entries for those instructions. Instead, we fetch the same * entry from the hash table and register it for the callback again. */ -g_mutex_lock (&mtx); + data =g_hash_table_lookup (miss_ht,GUINT_TO_POINTER (effective_addr));if (data ==NULL ) { data =g_new0 (InsnData,1 );@@ -527,13 +549,20 @@ static void log_top_insns()static void plugin_exit (qemu_plugin_id_t id,void *p) { +g_mutex_lock (&mtx);log_stats ();log_top_insns ();cache_free (dcache); + dcache =NULL ; +cache_free (icache); + icache =NULL ;g_hash_table_destroy (miss_ht); + miss_ht =NULL ; + +g_mutex_unlock (&mtx); }static void policy_init ()
Signature · 1 line
Alex Bennée
Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode
Quoted 11 lines · Alex Bennée
Quoted 5 lines · Mahmoud Mandour
Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about
plugin_exiteither. The problem withplugin_exitis that it frees shared data structures (caches andmiss_hthash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?
I think just any ad-hoc multithreaded execution will evoke the race pretty much
consistently.
Quoted 5 lines
From what I remember the race is
in betweenpreexit_cleanupand the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?
I think so. This remedies the bug for this particular plugin and I think there
would be a better solution of course. However, I just can't ever get plugin_exit
callback to be called more than once so I think that's probably not the problem?
The problem is that we *use* the data in translation/mem_access/exec callbacks
after a plugin_exit call is already called (this can be easily verified by having a
boolean set to true once plugin_exit is called and then g_assert this boolean is
false in the callbacks)
Quoted 106 lines
contrib/plugins/cache.c static void vcpu_mem_access(unsigned int
Quoted 20 lines
This is mitigated in this commit by synchronizing the call to
plugin_exitthrough locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since
plugin_exitis already called once and reported the statistics.Signed-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)contrib/plugins/cache.c +0
-0@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int
vcpu_index,qemu_plugin_meminfo_tinfo,contrib/plugins/cache.c static void vcpu_mem_access(unsigned int · line 363
Quoted 1 line
contrib/plugins/cache.c +0
-0effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) :vaddr;
contrib/plugins/cache.c static void vcpu_insn_exec(unsigned int vcpu_index,
Quoted 11 lines
contrib/plugins/cache.c +5
-0g_mutex_lock (&mtx); +if (dcache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index,void *userdata)
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id,
Quoted 12 lines
contrib/plugins/cache.c +5
-0g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; +if (icache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (icache, insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id,struct
qemu_plugin_tb*tb)contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
Quoted 7 lines
contrib/plugins/cache.c +4
-0effective_addr = (uint64_t )qemu_plugin_insn_vaddr (insn); } +g_mutex_lock (&mtx); + +/* +* is the plugin_exit callback called? If so, any further callback
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 423
Quoted 1 line
contrib/plugins/cache.c +1
-0+* registration is useless as it won't get accounted for after calling
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 424
Quoted 1 line
contrib/plugins/cache.c +1
-0+* plugin_exit once already, and also will use miss_ht after it's freed
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
Quoted 8 lines
contrib/plugins/cache.c +6
-0+*/ +if (miss_ht ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +/* * Instructions might get translated multiple times, we do not create
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 433
Quoted 1 line
contrib/plugins/cache.c +0
-0* new entries for those instructions. Instead, we fetch the same
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 434
Quoted 1 line
contrib/plugins/cache.c +0
-0* entry from the hash table and register it for the callback again.
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
Quoted 4 lines
contrib/plugins/cache.c +1
-1*/ -g_mutex_lock (&mtx); + data = g_hash_table_lookup(miss_ht,
GUINT_TO_POINTER(effective_addr));Quoted 13 lines
contrib/plugins/cache.c +3
-0if (data ==NULL ) { data =g_new0 (InsnData,1 );@@ -527,13 +549,20 @@ static void log_top_insns()static void plugin_exit (qemu_plugin_id_t id,void *p) { +g_mutex_lock (&mtx);log_stats ();log_top_insns ();cache_free (dcache); + dcache =NULL ; +contrib/plugins/cache.c static void log_top_insns() · line 568
Quoted 10 lines
contrib/plugins/cache.c +4
-0cache_free (icache); + icache =NULL ;g_hash_table_destroy (miss_ht); + miss_ht =NULL ; + +g_mutex_unlock (&mtx); }static void policy_init ()--
Alex Bennée
Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode
Quoted 10 lines · Mahmoud Mandour
On Mon, Jul 19, 2021 at 11:48 AM Alex Bennée <[email protected]> wrote:
Quoted 5 lines · Mahmoud Mandour
Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about
plugin_exiteither. The problem withplugin_exitis that it frees shared data structures (caches andmiss_hthash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.
Quoted 5 lines
What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?
I think just any ad-hoc multithreaded execution will evoke the race pretty much consistently.
I haven't managed to trigger it with testthread but of course my libcache is trying to to defend against it.
Quoted 6 lines
From what I remember the race is
in betweenpreexit_cleanupand the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?
Quoted 8 lines
I think so. This remedies the bug for this particular plugin and I think there would be a better solution of course. However, I just can't ever get
plugin_exitcallback to be called more than once so I think that's probably not the problem?The problem is that we *use* the data in translation/mem_access/exec callbacks after a
plugin_exitcall is already called (this can be easily verified by having a boolean set to true onceplugin_exitis called and theng_assertthis boolean is false in the callbacks)
We have mechanisms for safely unloading plugins during running so I think we should be able to do something cleanly here. I'll cook up an RFC.
Quoted 94 lines
Quoted 81 lines
This is mitigated in this commit by synchronizing the call to
plugin_exitthrough locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since
plugin_exitis already called once and reported the statistics.Signed-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)contrib/plugins/cache.c +26
-1@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr;g_mutex_lock (&mtx); +if (dcache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; +if (icache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (icache, insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) effective_addr = (uint64_t )qemu_plugin_insn_vaddr (insn); } +g_mutex_lock (&mtx); + +/* +* is the plugin_exit callback called? If so, any further callback +* registration is useless as it won't get accounted for after calling +* plugin_exit once already, and also will use miss_ht after it's freed +*/ +if (miss_ht ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +/* * Instructions might get translated multiple times, we do not create * new entries for those instructions. Instead, we fetch the same * entry from the hash table and register it for the callback again. */ -g_mutex_lock (&mtx); + data =g_hash_table_lookup (miss_ht,GUINT_TO_POINTER (effective_addr));if (data ==NULL ) { data =g_new0 (InsnData,1 );@@ -527,13 +549,20 @@ static void log_top_insns()static void plugin_exit (qemu_plugin_id_t id,void *p) { +g_mutex_lock (&mtx);log_stats ();log_top_insns ();cache_free (dcache); + dcache =NULL ; +contrib/plugins/cache.c static void log_top_insns() · line 568
Quoted 10 lines
contrib/plugins/cache.c +4
-0cache_free (icache); + icache =NULL ;g_hash_table_destroy (miss_ht); + miss_ht =NULL ; + +g_mutex_unlock (&mtx); }static void policy_init ()--
Alex Bennée
Signature · 1 line
Alex Bennée
Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode
Quoted 18 lines · Alex Bennée
Quoted 1 line · Mahmoud Mandour
On Mon, Jul 19, 2021 at 11:48 AM Alex Bennée <[email protected]>
Quoted 7 lines · wrote:
Quoted 4 lines · Mahmoud Mandour
Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about
plugin_exiteither. The problem withplugin_exitis that it frees shared data structures (caches andmiss_hthash table). It should not be assumed that callbacks willnot
Quoted 5 lines
Quoted 1 line
be called after it and hence use already-freed data structures.
What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?
Quoted 7 lines
Quoted 1 line
I think just any ad-hoc multithreaded execution will evoke the race
pretty much
Quoted 1 line
consistently.
I haven't managed to trigger it with testthread but of course my libcache is trying to to defend against it.
https://pastebin.com/X4Xazemh
That's a minimal program that reproduces the bug consistently for me (to my observation, a simple
program that creates a bunch of threads that immediately exit does not produce the bug as frequently)
You can also make hotblocks produce a similar crash (but make sure to add a g_hash_table_destroy(hotblocks)
at the end of plugin_exit.)
Quoted 17 lines
Quoted 8 lines
From what I remember the race is
in betweenpreexit_cleanupand the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?I think so. This remedies the bug for this particular plugin and I think
there
Quoted 1 line
would be a better solution of course. However, I just can't ever get
plugin_exitQuoted 1 line
callback to be called more than once so I think that's probably not the
problem?
Quoted 2 lines
The problem is that we *use* the data in translation/mem_access/exec
callbacks
Quoted 1 line
after a
plugin_exitcall is already called (this can be easily verified
Quoted 9 lines
by having a
Quoted 1 line
boolean set to true once
plugin_exitis called and theng_assertthisboolean is
Quoted 1 line
false in the callbacks)
We have mechanisms for safely unloading plugins during running so I think we should be able to do something cleanly here. I'll cook up an RFC.
I'll be waiting for it. Note that as I think I mentioned in the cover letter, it's that plugin_uninstall
is probably problematic since it unregisters callbacks but already-fired callbacks will continue executing.
So calling plugin_uninstall at the end of plugin_exit does not relieve the bug...
Quoted 112 lines
Quoted 10 lines
Quoted 10 lines
This is mitigated in this commit by synchronizing the call to
plugin_exitthrough locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since
plugin_exitisalready
Quoted 7 lines
Quoted 7 lines
called once and reported the statistics.
Signed-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)Quoted 5 lines
contrib/plugins/cache.c static void vcpu_mem_access(unsigned int
Quoted 5 lines
contrib/plugins/cache.c +0
-0@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int
vcpu_index,qemu_plugin_meminfo_tinfo,Quoted 1 line
contrib/plugins/cache.c static void vcpu_mem_access(unsigned int · line 363
Quoted 1 line
contrib/plugins/cache.c +0
-0effective_addr = hwaddr ? qemu_plugin_hwaddr_phys_addr(hwaddr) :vaddr;
Quoted 11 lines
contrib/plugins/cache.c static void vcpu_insn_exec(unsigned int
Quoted 11 lines
contrib/plugins/cache.c +5
-0g_mutex_lock (&mtx); +if (dcache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int
vcpu_index, void *userdata)Quoted 12 lines
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id,
Quoted 12 lines
contrib/plugins/cache.c +5
-0g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; +if (icache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (icache, insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id,struct
qemu_plugin_tb*tb)Quoted 7 lines
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
Quoted 7 lines
contrib/plugins/cache.c +4
-0effective_addr = (uint64_t )qemu_plugin_insn_vaddr (insn); } +g_mutex_lock (&mtx); + +/* +* is the plugin_exit callback called? If so, any further callback
Quoted 1 line
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 423
Quoted 1 line
contrib/plugins/cache.c +1
-0+* registration is useless as it won't get accounted for after calling
Quoted 1 line
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 424
Quoted 1 line
contrib/plugins/cache.c +1
-0+* plugin_exit once already, and also will use miss_ht after it's freed
Quoted 8 lines
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
Quoted 8 lines
contrib/plugins/cache.c +6
-0+*/ +if (miss_ht ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +/* * Instructions might get translated multiple times, we do not create
Quoted 1 line
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 433
Quoted 1 line
contrib/plugins/cache.c +0
-0* new entries for those instructions. Instead, we fetch the same
Quoted 1 line
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, · line 434
Quoted 1 line
contrib/plugins/cache.c +0
-0* entry from the hash table and register it for the callback again.
Quoted 4 lines
contrib/plugins/cache.c static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb)
Quoted 4 lines
contrib/plugins/cache.c +1
-1*/ -g_mutex_lock (&mtx); + data = g_hash_table_lookup(miss_ht,
GUINT_TO_POINTER(effective_addr));Quoted 22 lines
Quoted 22 lines
contrib/plugins/cache.c +7
-0if (data ==NULL ) { data =g_new0 (InsnData,1 );@@ -527,13 +549,20 @@ static void log_top_insns()static void plugin_exit (qemu_plugin_id_t id,void *p) { +g_mutex_lock (&mtx);log_stats ();log_top_insns ();cache_free (dcache); + dcache =NULL ; +cache_free (icache); + icache =NULL ;g_hash_table_destroy (miss_ht); + miss_ht =NULL ; + +g_mutex_unlock (&mtx); }Quoted 4 lines
contrib/plugins/cache.c static void log_top_insns() · line 568
Quoted 1 line
contrib/plugins/cache.c +0
-0static void policy_init ()--
Alex Bennée--
Alex Bennée
Re: [PATCH 3/6] plugins/cache: Fixed a use-after-free bug with multithreaded usermode
Quoted 23 lines · Mahmoud Mandour
On Mon, Jul 19, 2021 at 1:08 PM Alex Bennée <[email protected]> wrote:
Quoted 10 lines · Mahmoud Mandour
On Mon, Jul 19, 2021 at 11:48 AM Alex Bennée <[email protected]> wrote:
Quoted 5 lines · Mahmoud Mandour
Since callbacks may be interleaved because of multithreaded execution, we should not make assumptions about
plugin_exiteither. The problem withplugin_exitis that it frees shared data structures (caches andmiss_hthash table). It should not be assumed that callbacks will not be called after it and hence use already-freed data structures.Quoted 5 lines
What was your test case for this because I wonder if it would be worth coming up with one for check-tcg?
I think just any ad-hoc multithreaded execution will evoke the race pretty much consistently.
I haven't managed to trigger it with testthread but of course my libcache is trying to to defend against it.
Quoted 7 lines
https://pastebin.com/X4Xazemh
That's a minimal program that reproduces the bug consistently for me (to my observation, a simple program that creates a bunch of threads that immediately exit does not produce the bug as frequently)You can also make hotblocks produce a similar crash (but make sure to add a
g_hash_table_destroy(hotblocks)at the end ofplugin_exit.)
Thanks, I'll give that a try.
Quoted 16 lines
Quoted 7 lines
From what I remember the race is
in betweenpreexit_cleanupand the eventual _exit/exit_group which nixes all other child threads. Maybe we should be triggering a more graceful unload here?Quoted 8 lines
I think so. This remedies the bug for this particular plugin and I think there would be a better solution of course. However, I just can't ever get
plugin_exitcallback to be called more than once so I think that's probably not the problem?The problem is that we *use* the data in translation/mem_access/exec callbacks after a
plugin_exitcall is already called (this can be easily verified by having a boolean set to true onceplugin_exitis called and theng_assertthis boolean is false in the callbacks)
Quoted 8 lines
We have mechanisms for safely unloading plugins during running so I think we should be able to do something cleanly here. I'll cook up an RFC.
I'll be waiting for it. Note that as I think I mentioned in the cover letter, it's that
plugin_uninstallis probably problematic since it unregisters callbacks but already-fired callbacks will continue executing. So callingplugin_uninstallat the end ofplugin_exitdoes not relieve
the bug...
That's OK - the plugin_uninstall contract explicitly says that callbacks may occur until the callback is called.
Quoted 99 lines
Quoted 92 lines
Quoted 92 lines
This is mitigated in this commit by synchronizing the call to
plugin_exitthrough locking to ensure execlusive access to data structures, and NULL-ifying those data structures so that subsequent callbacks can check whether the data strucutres are freed, and if so, immediately exit.It's okay to immediately exit and don't account for those callbacks since they won't be accounted for anyway since
plugin_exitis already called once and reported the statistics.Signed-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)contrib/plugins/cache.c +30
-1@@ -363,6 +363,11 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr;g_mutex_lock (&mtx); +if (dcache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (dcache, effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++;@@ -380,6 +385,11 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata)g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; +if (icache ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +if (!access_cache (icache, insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;@@ -406,12 +416,24 @@ static void vcpu_tb_trans(qemu_plugin_id_t id, struct qemu_plugin_tb *tb) effective_addr = (uint64_t )qemu_plugin_insn_vaddr (insn); } +g_mutex_lock (&mtx); + +/* +* is the plugin_exit callback called? If so, any further callback +* registration is useless as it won't get accounted for after calling +* plugin_exit once already, and also will use miss_ht after it's freed +*/ +if (miss_ht ==NULL ) { +g_mutex_unlock (&mtx); +return ; + } +/* * Instructions might get translated multiple times, we do not create * new entries for those instructions. Instead, we fetch the same * entry from the hash table and register it for the callback again. */ -g_mutex_lock (&mtx); + data =g_hash_table_lookup (miss_ht,GUINT_TO_POINTER (effective_addr));if (data ==NULL ) { data =g_new0 (InsnData,1 );@@ -527,13 +549,20 @@ static void log_top_insns()static void plugin_exit (qemu_plugin_id_t id,void *p) { +g_mutex_lock (&mtx);log_stats ();log_top_insns ();cache_free (dcache); + dcache =NULL ; +cache_free (icache); + icache =NULL ;g_hash_table_destroy (miss_ht); + miss_ht =NULL ; + +g_mutex_unlock (&mtx); }Quoted 4 lines
contrib/plugins/cache.c static void log_top_insns() · line 568
Quoted 1 line
contrib/plugins/cache.c +0
-0static void policy_init ()--
Alex Bennée--
Alex Bennée
Signature · 1 line
Alex Bennée
PATCH 4/6 plugins/cache: Supported multicore cache modelling
In reply to Mahmoud MandourMulticore L1 cache modelling is introduced and is supported for both full system emulation and linux-user.
For full-system emulation, L1 icache and dcache are maintained for each available core, since this information is exposed to the plugin through qemu_plugin_n_vcpus().
For linux-user, a static number of cores is assumed (default 1 core, and can be provided as a plugin argument cores=N). Every memory access goes through one of these caches, this approach is taken as it's somewhat akin to what happens on real setup, where a program that dispatches more threads than the available cores, they'll thrash each other
--- contrib/plugins/cache.c | 156 ++++++++++++++++++++++++++++++---------- 1 file changed, 117 insertions(+), 39 deletions(-)
contrib/plugins/cache.c +117 -39
@@ -23,11 +23,6 @@ static GRand *rng;static int limit;static bool sys; -static uint64_t dmem_accesses; -static uint64_t dmisses; - -static uint64_t imem_accesses; -static uint64_t imisses;enum EvictionPolicy {LRU ,
@@ -90,13 +85,22 @@ typedef struct {uint64_t imisses; } InsnData; +typedef struct { +uint64_t dmem_accesses; +uint64_t dmisses; +uint64_t imem_accesses; +uint64_t imisses; +}CoreStats ; +void (*update_hit)(Cache *cache,int set,int blk);void (*update_miss)(Cache *cache,int set,int blk);void (*metadata_init)(Cache *cache);void (*metadata_destroy)(Cache *cache); -Cache *dcache, *icache; +static int cores; +CoreStats *stats; +Cache **dcaches, **icaches;static int pow_of_two (int num) {
@@ -233,10 +237,6 @@ static bool bad_cache_params(int blksize, int assoc, int cachesize)static Cache *cache_init (int blksize,int assoc,int cachesize) { -if (bad_cache_params (blksize, assoc, cachesize)) { -return NULL ; - } -Cache *cache;int i;uint64_t blk_mask;
@@ -263,6 +263,24 @@ static Cache *cache_init(int blksize, int assoc, int cachesize)return cache; } +static Cache **caches_init (int blksize,int assoc,int cachesize) +{ +Cache **caches; +int i; + +if (bad_cache_params (blksize, assoc, cachesize)) { +return NULL ; + } + + caches = g_new(Cache *, cores); + +for (i =0 ; i < cores; i++) { + caches[i] =cache_init (blksize, assoc, cachesize); + } + +return caches; +} +static int get_invalid_block (Cache *cache,uint64_t set) {int i;
@@ -353,6 +371,7 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, {uint64_t effective_addr;struct qemu_plugin_hwaddr *hwaddr; +int cache_idx;InsnData *insn; hwaddr =qemu_plugin_get_hwaddr (info, vaddr);
@@ -363,17 +382,24 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr;g_mutex_lock (&mtx); -if (dcache ==NULL ) { + cache_idx = vcpu_index % cores; +if (dcaches[cache_idx] ==NULL ) {g_mutex_unlock (&mtx);return ; } -if (!access_cache (dcache, effective_addr)) { +if (!access_cache (dcaches[cache_idx], effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++; - dmisses++; +if (cores >1 ) { + stats[cores].dmisses++; + } + stats[cache_idx].dmisses++; + } +if (cores >1 ) { + stats[cores].dmem_accesses++; } - dmem_accesses++; + stats[cache_idx].dmem_accesses++;g_mutex_unlock (&mtx); }
@@ -381,21 +407,29 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata) {uint64_t insn_addr;InsnData *insn; +uint64_t cache_idx;g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; -if (icache ==NULL ) { + cache_idx = vcpu_index % cores; +if (icaches[cache_idx] ==NULL ) {g_mutex_unlock (&mtx);return ; } -if (!access_cache (icache, insn_addr)) { +if (!access_cache (icaches[cache_idx], insn_addr)) { insn = (InsnData *) userdata; insn->imisses++; - imisses++; +if (cores >1 ) { + stats[cores].imisses++; + } + stats[cache_idx].imisses++; } - imem_accesses++; +if (cores >1 ) { + stats[cores].imem_accesses++; + } + stats[cache_idx].imem_accesses++;g_mutex_unlock (&mtx); }
@@ -475,6 +509,22 @@ static void cache_free(Cache *cache)g_free (cache); } +/* +* caches_free(): free an array of Cache structs. +* @caches: caches to free +* +* Calling cache_free for each cache in @caches, and then NULL-ify them so that +* we mark them as freed, other async callbacks can check to see whether a cache +* is freed already or not by checking against NULL. +*/ +static void caches_free (Cache **caches) +{ +for (int i =0 ; i < cores; i++) { +cache_free (caches[i]); + caches[i] =NULL ; + } +} +static int dcmp (gconstpointer a,gconstpointer b) {InsnData *insn_a = (InsnData *) a;
@@ -493,19 +543,38 @@ static int icmp(gconstpointer a, gconstpointer b)static void log_stats () { -g_autoptr (GString) rep =g_string_new ("" ); -g_string_append_printf (rep, -"Data accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n" , - dmem_accesses, - dmisses, - ((double ) dmisses / (double ) dmem_accesses) *100.0 ); - -g_string_append_printf (rep, -"Instruction accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n" , - imem_accesses, - imisses, - ((double ) imisses / (double ) imem_accesses) *100.0 ); +int i, iters; +CoreStats cs; +double dmiss_rate, imiss_rate; + +g_autoptr (GString) rep =g_string_new ("core #, data accesses, data misses," +" dmiss rate, insn accesses," +" insn misses, imiss rate\n" ); + +/* Only iterate and print a sum row if cores > 1 */ + iters = cores ==1 ?1 : cores +1 ; +for (i =0 ; i < iters; i++) { + cs = stats[i]; + dmiss_rate = ((double ) cs.dmisses) / (cs.dmem_accesses) *100.0 ; + imiss_rate = ((double ) cs.imisses) / (cs.imem_accesses) *100.0 ; + +if (i == cores) { +g_string_append_printf (rep,"%-8s" ,"sum" ); + }else { +g_string_append_printf (rep,"%-8d" , i); + } + +g_string_append_printf (rep,"%-14lu %-12lu %9.4lf%% %-14lu %-12lu" +" %9.4lf%%\n" , + cs.dmem_accesses, + cs.dmisses, + cs.dmem_accesses ? dmiss_rate :0.0 , + cs.imem_accesses, + cs.imisses, + cs.imem_accesses ? imiss_rate :0.0 ); + } +g_string_append (rep,"\n" );qemu_plugin_outs (rep->str); }
@@ -553,15 +622,14 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)log_stats ();log_top_insns (); -cache_free (dcache); - dcache =NULL ; - -cache_free (icache); - icache =NULL ; +caches_free (dcaches); +caches_free (icaches);g_hash_table_destroy (miss_ht); miss_ht =NULL ; +g_free (stats); +g_mutex_unlock (&mtx); }
@@ -608,6 +676,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, policy =LRU ; + cores = sys ?qemu_plugin_n_vcpus () :1 ; +for (i =0 ; i < argc; i++) {char *opt = argv[i];if (g_str_has_prefix (opt,"iblksize=" )) {
@@ -624,6 +694,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, dcachesize =g_ascii_strtoll (opt +11 ,NULL ,10 ); }else if (g_str_has_prefix (opt,"limit=" )) { limit =g_ascii_strtoll (opt +6 ,NULL ,10 ); + }else if (g_str_has_prefix (opt,"cores=" )) { + cores =g_ascii_strtoll (opt +6 ,NULL ,10 ); }else if (g_str_has_prefix (opt,"evict=" )) {gchar *p = opt +6 ;if (g_strcmp0 (p,"rand" ) ==0 ) {
@@ -644,22 +716,28 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,policy_init (); - dcache =cache_init (dblksize, dassoc, dcachesize); -if (!dcache) { + dcaches =caches_init (dblksize, dassoc, dcachesize); +if (!dcaches) {const char *err =cache_config_error (dblksize, dassoc, dcachesize);fprintf (stderr,"dcache cannot be constructed from given parameters\n" );fprintf (stderr,"%s\n" , err);return -1 ; } - icache =cache_init (iblksize, iassoc, icachesize); -if (!icache) { + icaches =caches_init (iblksize, iassoc, icachesize); +if (!icaches) {const char *err =cache_config_error (iblksize, iassoc, icachesize);fprintf (stderr,"icache cannot be constructed from given parameters\n" );fprintf (stderr,"%s\n" , err);return -1 ; } +/* +* plus one to save the sum in. If only one core is used then no need to +* get an auxiliary struct. +*/ + stats =g_new0 (CoreStats, cores ==1 ?1 : cores +1 ); +qemu_plugin_register_vcpu_tb_trans_cb (id, vcpu_tb_trans);qemu_plugin_register_atexit_cb (id, plugin_exit,NULL );
Signature · 1 line
2.25.1
Re: [PATCH 4/6] plugins/cache: Supported multicore cache modelling
Quoted 104 lines · Mahmoud Mandour
Multicore L1 cache modelling is introduced and is supported for both full system emulation and linux-user.
For full-system emulation, L1 icache and dcache are maintained for each available core, since this information is exposed to the plugin through
qemu_plugin_n_vcpus().For linux-user, a static number of cores is assumed (default 1 core, and can be provided as a plugin argument
cores=N). Every memory access goes through one of these caches, this approach is taken as it's somewhat akin to what happens on real setup, where a program that dispatches more threads than the available cores, they'll thrash each otherSigned-off-by Mahmoud Mandour <[email protected]>--- contrib/plugins/cache.c | 156 ++++++++++++++++++++++++++++++---------- 1 file changed, 117 insertions(+), 39 deletions(-)contrib/plugins/cache.c +29
-10@@ -23,11 +23,6 @@ static GRand *rng;static int limit;static bool sys; -static uint64_t dmem_accesses; -static uint64_t dmisses; - -static uint64_t imem_accesses; -static uint64_t imisses;enum EvictionPolicy {LRU ,@@ -90,13 +85,22 @@ typedef struct {uint64_t imisses; } InsnData; +typedef struct { +uint64_t dmem_accesses; +uint64_t dmisses; +uint64_t imem_accesses; +uint64_t imisses; +}CoreStats ; +void (*update_hit)(Cache *cache,int set,int blk);void (*update_miss)(Cache *cache,int set,int blk);void (*metadata_init)(Cache *cache);void (*metadata_destroy)(Cache *cache); -Cache *dcache, *icache; +static int cores; +CoreStats *stats; +Cache **dcaches, **icaches;static int pow_of_two (int num) {@@ -233,10 +237,6 @@ static bool bad_cache_params(int blksize, int assoc, int cachesize)static Cache *cache_init (int blksize,int assoc,int cachesize) { -if (bad_cache_params (blksize, assoc, cachesize)) { -return NULL ; - } -Cache *cache;int i;uint64_t blk_mask;@@ -263,6 +263,24 @@ static Cache *cache_init(int blksize, int assoc, int cachesize)return cache; } +static Cache **caches_init (int blksize,int assoc,int cachesize) +{ +Cache **caches; +int i; + +if (bad_cache_params (blksize, assoc, cachesize)) { +return NULL ; + } + + caches = g_new(Cache *, cores); + +for (i =0 ; i < cores; i++) { + caches[i] =cache_init (blksize, assoc, cachesize); + } + +return caches; +} +static int get_invalid_block (Cache *cache,uint64_t set) {int i;@@ -353,6 +371,7 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, {uint64_t effective_addr;struct qemu_plugin_hwaddr *hwaddr; +int cache_idx;InsnData *insn; hwaddr =qemu_plugin_get_hwaddr (info, vaddr);
contrib/plugins/cache.c static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, · line 389
Quoted 9 lines
contrib/plugins/cache.c +2
-1@@ -363,17 +382,24 @@ static void vcpu_mem_access(unsigned int vcpu_index, qemu_plugin_meminfo_t info, effective_addr = hwaddr ?qemu_plugin_hwaddr_phys_addr (hwaddr) : vaddr;g_mutex_lock (&mtx); -if (dcache ==NULL ) { + cache_idx = vcpu_index % cores; +if (dcaches[cache_idx] ==NULL ) {g_mutex_unlock (&mtx);return ; }
I was hoping we could avoid this with some careful atomic/memory barrier use given dcaches is setup on init. However it's the exit case that gets us which further increases my desire to fix the exit race in linux-user and make sure callbacks don't occur in this case.
Quoted 39 lines
contrib/plugins/cache.c +13
-5-if (!access_cache (dcache, effective_addr)) { +if (!access_cache (dcaches[cache_idx], effective_addr)) { insn = (InsnData *) userdata; insn->dmisses++; - dmisses++; +if (cores >1 ) { + stats[cores].dmisses++; + } + stats[cache_idx].dmisses++; + } +if (cores >1 ) { + stats[cores].dmem_accesses++; } - dmem_accesses++; + stats[cache_idx].dmem_accesses++;g_mutex_unlock (&mtx); }@@ -381,21 +407,29 @@ static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata) {uint64_t insn_addr;InsnData *insn; +uint64_t cache_idx;g_mutex_lock (&mtx); insn_addr = ((InsnData *) userdata)->addr; -if (icache ==NULL ) { + cache_idx = vcpu_index % cores; +if (icaches[cache_idx] ==NULL ) {g_mutex_unlock (&mtx);return ; } -if (!access_cache (icache, insn_addr)) { +if (!access_cache (icaches[cache_idx], insn_addr)) { insn = (InsnData *) userdata; insn->imisses++;
contrib/plugins/cache.c static void vcpu_insn_exec(unsigned int vcpu_index, void *userdata) · line 431
Quoted 10 lines
contrib/plugins/cache.c +7
-2- imisses++; +if (cores >1 ) { + stats[cores].imisses++; + } + stats[cache_idx].imisses++; } - imem_accesses++; +if (cores >1 ) { + stats[cores].imem_accesses++; + }
I'd much rather avoid summing the stats during execution than at the end when we can add things simply.
Quoted 136 lines
contrib/plugins/cache.c +60
-21+ stats[cache_idx].imem_accesses++;g_mutex_unlock (&mtx); }@@ -475,6 +509,22 @@ static void cache_free(Cache *cache)g_free (cache); } +/* +* caches_free(): free an array of Cache structs. +* @caches: caches to free +* +* Calling cache_free for each cache in @caches, and then NULL-ify them so that +* we mark them as freed, other async callbacks can check to see whether a cache +* is freed already or not by checking against NULL. +*/ +static void caches_free (Cache **caches) +{ +for (int i =0 ; i < cores; i++) { +cache_free (caches[i]); + caches[i] =NULL ; + } +} +static int dcmp (gconstpointer a,gconstpointer b) {InsnData *insn_a = (InsnData *) a;@@ -493,19 +543,38 @@ static int icmp(gconstpointer a, gconstpointer b)static void log_stats () { -g_autoptr (GString) rep =g_string_new ("" ); -g_string_append_printf (rep, -"Data accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n" , - dmem_accesses, - dmisses, - ((double ) dmisses / (double ) dmem_accesses) *100.0 ); - -g_string_append_printf (rep, -"Instruction accesses: %lu, Misses: %lu\nMiss rate: %lf%%\n\n" , - imem_accesses, - imisses, - ((double ) imisses / (double ) imem_accesses) *100.0 ); +int i, iters; +CoreStats cs; +double dmiss_rate, imiss_rate; + +g_autoptr (GString) rep =g_string_new ("core #, data accesses, data misses," +" dmiss rate, insn accesses," +" insn misses, imiss rate\n" ); + +/* Only iterate and print a sum row if cores > 1 */ + iters = cores ==1 ?1 : cores +1 ; +for (i =0 ; i < iters; i++) { + cs = stats[i]; + dmiss_rate = ((double ) cs.dmisses) / (cs.dmem_accesses) *100.0 ; + imiss_rate = ((double ) cs.imisses) / (cs.imem_accesses) *100.0 ; + +if (i == cores) { +g_string_append_printf (rep,"%-8s" ,"sum" ); + }else { +g_string_append_printf (rep,"%-8d" , i); + } + +g_string_append_printf (rep,"%-14lu %-12lu %9.4lf%% %-14lu %-12lu" +" %9.4lf%%\n" , + cs.dmem_accesses, + cs.dmisses, + cs.dmem_accesses ? dmiss_rate :0.0 , + cs.imem_accesses, + cs.imisses, + cs.imem_accesses ? imiss_rate :0.0 ); + } +g_string_append (rep,"\n" );qemu_plugin_outs (rep->str); }@@ -553,15 +622,14 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)log_stats ();log_top_insns (); -cache_free (dcache); - dcache =NULL ; - -cache_free (icache); - icache =NULL ; +caches_free (dcaches); +caches_free (icaches);g_hash_table_destroy (miss_ht); miss_ht =NULL ; +g_free (stats); +g_mutex_unlock (&mtx); }@@ -608,6 +676,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, policy =LRU ; + cores = sys ?qemu_plugin_n_vcpus () :1 ; +for (i =0 ; i < argc; i++) {char *opt = argv[i];if (g_str_has_prefix (opt,"iblksize=" )) {@@ -624,6 +694,8 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, dcachesize =g_ascii_strtoll (opt +11 ,NULL ,10 ); }else if (g_str_has_prefix (opt,"limit=" )) { limit =g_ascii_strtoll (opt +6 ,NULL ,10 ); + }else if (g_str_has_prefix (opt,"cores=" )) { + cores =g_ascii_strtoll (opt +6 ,NULL ,10 ); }else if (g_str_has_prefix (opt,"evict=" )) {gchar *p = opt +6 ;if (g_strcmp0 (p,"rand" ) ==0 ) {@@ -644,22 +716,28 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,policy_init (); - dcache =cache_init (dblksize, dassoc, dcachesize); -if (!dcache) { + dcaches =caches_init (dblksize, dassoc, dcachesize); +if (!dcaches) {const char *err =cache_config_error (dblksize, dassoc, dcachesize);fprintf (stderr,"dcache cannot be constructed from given parameters\n" );fprintf (stderr,"%s\n" , err);return -1 ; } - icache =cache_init (iblksize, iassoc, icachesize); -if (!icache) { + icaches =caches_init (iblksize, iassoc, icachesize); +if (!icaches) {const char *err =cache_config_error (iblksize, iassoc, icachesize);fprintf (stderr,"icache cannot be constructed from given parameters\n" );
contrib/plugins/cache.c int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info, · line 740
Quoted 10 lines
contrib/plugins/cache.c +6
-0fprintf (stderr,"%s\n" , err);return -1 ; } +/* +* plus one to save the sum in. If only one core is used then no need to +* get an auxiliary struct. +*/ + stats =g_new0 (CoreStats, cores ==1 ?1 : cores +1 ); +
See above, lets keep the sum as a post processing step.
Quoted 2 lines
contrib/plugins/cache.c +0
-0qemu_plugin_register_vcpu_tb_trans_cb (id, vcpu_tb_trans);qemu_plugin_register_atexit_cb (id, plugin_exit,NULL );
Signature · 1 line
Alex Bennée
PATCH 5/6 docs/devel/tcg-plugins: added cores arg to cache plugin
In reply to Mahmoud Mandour--- docs/devel/tcg-plugins.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
docs/devel/tcg-plugins.rst +8 -5
@@ -330,11 +330,8 @@ configuration when a given working set is run:: will report the following:: - Data accesses: 996479, Misses: 507 - Miss rate: 0.050879% - - Instruction accesses: 2641737, Misses: 18617 - Miss rate: 0.704726% + core #, data accesses, data misses, dmiss rate, insn accesses, insn misses, imiss rate + 0 996695 508 0.0510% 2642799 18617 0.7044% address, data misses, instruction 0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx)
@@ -378,3 +375,9 @@ The plugin has a number of arguments, all of them are optional: Sets the eviction policy to POLICY. Available policies are: :code:`lru`, :code:`fifo`, and :code:`rand`. The plugin will use the specified policy for both instruction and data caches. (default: POLICY = :code:`lru`) + + * arg="cores=N" + + Sets the number of cores for which we maintain separate icache and dcache. + (default: for linux-user, N = 1, for full system emulation: N = cores + available to guest)
Signature · 1 line
2.25.1
PATCH 6/6 plugins/cache: Fixed "function decl. is not a prototype" warnings
In reply to Mahmoud Mandour--- contrib/plugins/cache.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
contrib/plugins/cache.c +3 -3
@@ -541,7 +541,7 @@ static int icmp(gconstpointer a, gconstpointer b)return insn_a->imisses < insn_b->imisses ?1 :-1 ; } -static void log_stats () +static void log_stats (void ) {int i, iters;CoreStats cs;
@@ -578,7 +578,7 @@ static void log_stats()qemu_plugin_outs (rep->str); } -static void log_top_insns () +static void log_top_insns (void ) {int i;GList *curr, *miss_insns;
@@ -633,7 +633,7 @@ static void plugin_exit(qemu_plugin_id_t id, void *p)g_mutex_unlock (&mtx); } -static void policy_init () +static void policy_init (void ) {switch (policy) {case LRU :
Signature · 1 line
2.25.1
Re: [PATCH 6/6] plugins/cache: Fixed "function decl. is not a prototype" warnings
Quoted 1 line · Mahmoud Mandour
Signed-off-by Mahmoud Mandour <[email protected]>
Signature · 1 line
Alex Bennée
Alex Bennée
AppliedQuoted 4 lines · Mahmoud Mandour
Hello,
This series introduce some minor improvements/bug fixes in the cache plugins and multicore cache modelling.
Queued patches 1,2 and 6 for-6.1/fixes-for-rc1 as they are bug fixes, thanks.
Signature · 1 line
Alex Bennée