gdb command: qemu iohandlers
PATCH 1/1 gdb command: qemu iohandlers
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Useful for debugging why an incoming migration has stalled, e.g.
----
{fd_read = 0x7fd4c8e40d00 <fd_coroutine_enter>, fd_write = 0x0, opaque = 0x7fd4b8bfeb00, next = {le_next = 0x7fd4cac81b00, le_prev = 0x7fd4c93d2bd0 <io_handlers>}, fd = 22, pollfds_idx = 0, deleted = false}
#0 qemu_coroutine_switch (from_=from_@entry=0x7fd4cbb33c00, to_=to_@entry=0x7fd4c8b13a90,
action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
#1 0x00007fd4c8e40507 in qemu_coroutine_yield () at qemu-coroutine.c:145
#2 0x00007fd4c8e40e75 in yield_until_fd_readable (fd=22) at qemu-coroutine-io.c:90
#3 0x00007fd4c8df347f in qemu_rdma_block_for_wrid (rdma=rdma@entry=0x7fd4b8c7e010,
wrid_requested=wrid_requested@entry=2000,
byte_len=byte_len@entry=0x0) at migration/rdma.c:1510
#4 0x00007fd4c8df388f in qemu_rdma_post_send_control (rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,
head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608
#5 0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010,
head=0x7fd4b8bfed50, expecting=3)
at migration/rdma.c:1814
#6 0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950,
buf=0x7fd4cc24fd20 "TR\022",
pos=<optimized out>, size=32768) at migration/rdma.c:2611
#7 0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at
migration/qemu-file.c:214
#8 0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0,
offset=offset@entry=0)
at migration/qemu-file.c:447
#9 0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0)
at /root/qemu-world3/migration/savevm.c:1128
#10 0x00007fd4c8ded895 in process_incoming_migration_co
(opaque=0x7fd4cc24fcf0) at migration/migration.c:145
#11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>) at coroutine-ucontext.c:80
#12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6
#13 0x00007ffee263e870 in ?? ()
#14 0x0000000000000000 in ?? ()
----
{fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5,
next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx =
1, deleted = false}--- scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
scripts/qemu-gdb.py +25 -0
@@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command): coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer()) bt_jmpbuf(coroutine_pointer['env']['__jmpbuf']) +class IOhandlersCommand(gdb.Command): + '''Display a backtrace for io handlers''' + def __init__(self): + gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA, + gdb.COMPLETE_NONE) + + def invoke(self, arg, from_tty): + cur = gdb.parse_and_eval('io_handlers.lh_first') + sym_fd_coroutine_enter = gdb.parse_and_eval('fd_coroutine_enter') + + while not isnull(cur): + entry = cur.dereference() + gdb.write('----\n%s\n' % entry) + + # For functions we know, extract the coroutine pointer and backtrace + if cur['fd_read'] == sym_fd_coroutine_enter: + # yield_until_fd_readable + coptr = (cur['opaque'].cast(gdb.lookup_type('FDYieldUntilData').pointer()))['co'] + coptr = coptr.cast(gdb.lookup_type('CoroutineUContext').pointer()) + bt_jmpbuf(coptr['env']['__jmpbuf']) + + cur = cur['next']['le_next'] + gdb.write('----\n') + class MtreeCommand(gdb.Command): '''Display the memory tree hierarchy''' def __init__(self):
@@ -161,4 +185,5 @@ class MtreeCommand(gdb.Command): QemuCommand() CoroutineCommand() +IOhandlersCommand() MtreeCommand()
Signature · 1 line
2.4.3
Quoted 6 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Useful for debugging why an incoming migration has stalled, e.g.
I'd rather we did the split of qemu-gdb.py into one subfile per command before adding new ones: see patches 1 and 2 in this series:
https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02961.html
thanks
-- PMM
Quoted 7 lines · * Peter Maydell ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Quoted 6 lines
Quoted 1 line
Useful for debugging why an incoming migration has stalled, e.g.
I'd rather we did the split of
qemu-gdb.pyinto one subfile per command before adding new ones: see patches 1 and 2 in this series:
https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02961.html
No problem - I'm happy to rework after that split goes in.
Note I'm using the bt_jmpbuf function which that patchset puts in coroutine.py; maybe there would be a better way to share it.
Dave
Quoted 3 lines
thanks
-- PMM
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 62 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Useful for debugging why an incoming migration has stalled, e.g.
----
{fd_read=0x7fd4c8e40d00<fd_coroutine_enter>,fd_write=0x0, opaque =0x7fd4b8bfeb00, next = {le_next=0x7fd4cac81b00,le_prev=0x7fd4c93d2bd0<io_handlers>}, fd = 22,pollfds_idx= 0, deleted = false}
#0qemu_coroutine_switch(from_=from_@entry=0x7fd4cbb33c00, to_=to_@entry=0x7fd4c8b13a90,
action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
#10x00007fd4c8e40507inqemu_coroutine_yield() at qemu-coroutine.c:145
#20x00007fd4c8e40e75inyield_until_fd_readable(fd=22) at qemu-coroutine-io.c:90
#30x00007fd4c8df347finqemu_rdma_block_for_wrid(rdma=rdma@entry=0x7fd4b8c7e010,
wrid_requested=wrid_requested@entry=2000,
byte_len=byte_len@entry=0x0) atmigration/rdma.c:1510
#40x00007fd4c8df388finqemu_rdma_post_send_control(rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608 #5 0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010, head=0x7fd4b8bfed50, expecting=3) at migration/rdma.c:1814 #6 0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950, buf=0x7fd4cc24fd20 "TR\022", pos=<optimized out>, size=32768) at migration/rdma.c:2611 #7 0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at migration/qemu-file.c:214 #8 0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0, offset=offset@entry=0) at migration/qemu-file.c:447 #9 0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0) at /root/qemu-world3/migration/savevm.c:1128 #10 0x00007fd4c8ded895 in process_incoming_migration_co (opaque=0x7fd4cc24fcf0) at migration/migration.c:145 #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:80 #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6 #13 0x00007ffee263e870 in ?? () #14 0x0000000000000000 in ?? () ---- {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5, next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx = 1, deleted = false}Signed-off-by Dr. David Alan Gilbert <[email protected]>--- scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)scripts/qemu-gdb.py +0
-0@@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command): coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())
scripts/qemu-gdb.py class CoroutineCommand(gdb.Command): · line 112
Quoted 10 lines
scripts/qemu-gdb.py +8
-0bt_jmpbuf(coroutine_pointer['env']['__jmpbuf']) +class IOhandlersCommand(gdb.Command): + '''Display a backtrace for io handlers''' + def __init__(self): + gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA, + gdb.COMPLETE_NONE) + + def invoke(self, arg, from_tty): + cur = gdb.parse_and_eval('io_handlers.lh_first')
Fam is getting rid of io_handlers and replacing it with an AioContext.
I have CCed him.
Quoted 65 lines · Stefan Hajnoczi
Quoted 64 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Useful for debugging why an incoming migration has stalled, e.g.
----
{fd_read=0x7fd4c8e40d00<fd_coroutine_enter>,fd_write=0x0, opaque =0x7fd4b8bfeb00, next = {le_next=0x7fd4cac81b00,le_prev=0x7fd4c93d2bd0<io_handlers>}, fd = 22,pollfds_idx= 0, deleted = false}
#0qemu_coroutine_switch(from_=from_@entry=0x7fd4cbb33c00, to_=to_@entry=0x7fd4c8b13a90,
action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
#10x00007fd4c8e40507inqemu_coroutine_yield() at qemu-coroutine.c:145
#20x00007fd4c8e40e75inyield_until_fd_readable(fd=22) at qemu-coroutine-io.c:90
#30x00007fd4c8df347finqemu_rdma_block_for_wrid(rdma=rdma@entry=0x7fd4b8c7e010,
wrid_requested=wrid_requested@entry=2000,
byte_len=byte_len@entry=0x0) atmigration/rdma.c:1510
#40x00007fd4c8df388finqemu_rdma_post_send_control(rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608 #5 0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010, head=0x7fd4b8bfed50, expecting=3) at migration/rdma.c:1814 #6 0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950, buf=0x7fd4cc24fd20 "TR\022", pos=<optimized out>, size=32768) at migration/rdma.c:2611 #7 0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at migration/qemu-file.c:214 #8 0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0, offset=offset@entry=0) at migration/qemu-file.c:447 #9 0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0) at /root/qemu-world3/migration/savevm.c:1128 #10 0x00007fd4c8ded895 in process_incoming_migration_co (opaque=0x7fd4cc24fcf0) at migration/migration.c:145 #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:80 #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6 #13 0x00007ffee263e870 in ?? () #14 0x0000000000000000 in ?? () ---- {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5, next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx = 1, deleted = false}Signed-off-by Dr. David Alan Gilbert <[email protected]>--- scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)scripts/qemu-gdb.py +0
-0@@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command): coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer()) bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])
Quoted 11 lines
scripts/qemu-gdb.py class CoroutineCommand(gdb.Command): · line 112
Quoted 8 lines
scripts/qemu-gdb.py +8
-0+class IOhandlersCommand(gdb.Command): + '''Display a backtrace for io handlers''' + def __init__(self): + gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA, + gdb.COMPLETE_NONE) + + def invoke(self, arg, from_tty): + cur = gdb.parse_and_eval('io_handlers.lh_first')Fam is getting rid of
io_handlersand replacing it with an AioContext.
Yes, the series is
[PATCH 0/9] slirp: iohandler: Rebase onto aio
Fam
Quoted 78 lines · * Fam Zheng ([email protected]) wrote:
Quoted 65 lines · Stefan Hajnoczi
Quoted 64 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Useful for debugging why an incoming migration has stalled, e.g.
----
{fd_read=0x7fd4c8e40d00<fd_coroutine_enter>,fd_write=0x0, opaque =0x7fd4b8bfeb00, next = {le_next=0x7fd4cac81b00,le_prev=0x7fd4c93d2bd0<io_handlers>}, fd = 22,pollfds_idx= 0, deleted = false}
#0qemu_coroutine_switch(from_=from_@entry=0x7fd4cbb33c00, to_=to_@entry=0x7fd4c8b13a90,
action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
#10x00007fd4c8e40507inqemu_coroutine_yield() at qemu-coroutine.c:145
#20x00007fd4c8e40e75inyield_until_fd_readable(fd=22) at qemu-coroutine-io.c:90
#30x00007fd4c8df347finqemu_rdma_block_for_wrid(rdma=rdma@entry=0x7fd4b8c7e010,
wrid_requested=wrid_requested@entry=2000,
byte_len=byte_len@entry=0x0) atmigration/rdma.c:1510
#40x00007fd4c8df388finqemu_rdma_post_send_control(rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608 #5 0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010, head=0x7fd4b8bfed50, expecting=3) at migration/rdma.c:1814 #6 0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950, buf=0x7fd4cc24fd20 "TR\022", pos=<optimized out>, size=32768) at migration/rdma.c:2611 #7 0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at migration/qemu-file.c:214 #8 0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0, offset=offset@entry=0) at migration/qemu-file.c:447 #9 0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0) at /root/qemu-world3/migration/savevm.c:1128 #10 0x00007fd4c8ded895 in process_incoming_migration_co (opaque=0x7fd4cc24fcf0) at migration/migration.c:145 #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:80 #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6 #13 0x00007ffee263e870 in ?? () #14 0x0000000000000000 in ?? () ---- {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5, next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx = 1, deleted = false}Signed-off-by Dr. David Alan Gilbert <[email protected]>--- scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)scripts/qemu-gdb.py +0
-0@@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command): coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer()) bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])Quoted 11 lines
scripts/qemu-gdb.py class CoroutineCommand(gdb.Command): · line 112
Quoted 8 lines
scripts/qemu-gdb.py +8
-0+class IOhandlersCommand(gdb.Command): + '''Display a backtrace for io handlers''' + def __init__(self): + gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA, + gdb.COMPLETE_NONE) + + def invoke(self, arg, from_tty): + cur = gdb.parse_and_eval('io_handlers.lh_first')Fam is getting rid of
io_handlersand replacing it with an AioContext.
Quoted 3 lines
Yes, the series is
[PATCH 0/9] slirp: iohandler: Rebase onto aio
OK, I can rework after that goes in.
I guess the equivalent there is to dump the iohandler_ctx->aio_handlers list?
Dave
Quoted 3 lines
Fam
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 79 lines · Dr. David Alan Gilbert
Quoted 78 lines · * Fam Zheng ([email protected]) wrote:
Quoted 76 lines · Stefan Hajnoczi
Quoted 62 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add a gdb command to print the current set of IOHandlers and if one of them is a thread yielding for data print the backtrace.
Useful for debugging why an incoming migration has stalled, e.g.
----
{fd_read=0x7fd4c8e40d00<fd_coroutine_enter>,fd_write=0x0, opaque =0x7fd4b8bfeb00, next = {le_next=0x7fd4cac81b00,le_prev=0x7fd4c93d2bd0<io_handlers>}, fd = 22,pollfds_idx= 0, deleted = false}
#0qemu_coroutine_switch(from_=from_@entry=0x7fd4cbb33c00, to_=to_@entry=0x7fd4c8b13a90,
action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
#10x00007fd4c8e40507inqemu_coroutine_yield() at qemu-coroutine.c:145
#20x00007fd4c8e40e75inyield_until_fd_readable(fd=22) at qemu-coroutine-io.c:90
#30x00007fd4c8df347finqemu_rdma_block_for_wrid(rdma=rdma@entry=0x7fd4b8c7e010,
wrid_requested=wrid_requested@entry=2000,
byte_len=byte_len@entry=0x0) atmigration/rdma.c:1510
#40x00007fd4c8df388finqemu_rdma_post_send_control(rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608 #5 0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010, head=0x7fd4b8bfed50, expecting=3) at migration/rdma.c:1814 #6 0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950, buf=0x7fd4cc24fd20 "TR\022", pos=<optimized out>, size=32768) at migration/rdma.c:2611 #7 0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at migration/qemu-file.c:214 #8 0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0, offset=offset@entry=0) at migration/qemu-file.c:447 #9 0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0) at /root/qemu-world3/migration/savevm.c:1128 #10 0x00007fd4c8ded895 in process_incoming_migration_co (opaque=0x7fd4cc24fcf0) at migration/migration.c:145 #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:80 #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6 #13 0x00007ffee263e870 in ?? () #14 0x0000000000000000 in ?? () ---- {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5, next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx = 1, deleted = false}Signed-off-by Dr. David Alan Gilbert <[email protected]>--- scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)scripts/qemu-gdb.py +0
-0@@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command): coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())scripts/qemu-gdb.py class CoroutineCommand(gdb.Command): · line 112
Quoted 10 lines
scripts/qemu-gdb.py +8
-0bt_jmpbuf(coroutine_pointer['env']['__jmpbuf']) +class IOhandlersCommand(gdb.Command): + '''Display a backtrace for io handlers''' + def __init__(self): + gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA, + gdb.COMPLETE_NONE) + + def invoke(self, arg, from_tty): + cur = gdb.parse_and_eval('io_handlers.lh_first')Fam is getting rid of
io_handlersand replacing it with an AioContext.
Quoted 6 lines
Quoted 3 lines
Yes, the series is
[PATCH 0/9] slirp: iohandler: Rebase onto aio
OK, I can rework after that goes in.
I guess the equivalent there is to dump theiohandler_ctx->aio_handlerslist?
Yes, I think so.
Fam