sheepdog driver patch: fixs the problem of qemu process become crashed when the sheepdog gateway break the IO and then recover
PATCH 1/1 sheepdog driver patch: fixs the problem of qemu process become crashed when the sheepdog gateway break the IO and then recover
this patch fixs the problem of qemu process become crashed when the sheepdog gateway break the IO for a few seconds and then recover.
problem reproduce:
1.start a fio process in qemu to produce IOs to sheepdog gateway, whatever IO type you like. 2.kill the sheepdog gateway.
3.wait for a few seconds.
4.restart the sheepdog gateway.
5.qemu process crashed with segfault error 6.
problem cause:
the last io coroutine will be destroyed after reconnect to sheepdog gateway, but the coroutine still be scheduled and the s->co_recv is still the last io coroutine pointer which had been destroyed, so when this coroutine go to coroutine context switch, it will make qemu process crashed.
problem fix:
just make s->co_recv = NULL when the last io coroutine reconnect to sheepdog gateway.
--- block/sheepdog.c | 1 + 1 file changed, 1 insertion(+)
block/sheepdog.c +1 -0
@@ -727,6 +727,7 @@ static coroutine_fn void reconnect_to_sdog(void *opaque) NULL, NULL, NULL);close (s->fd); s->fd =-1 ; + s->co_recv =NULL ;/* Wait for outstanding write requests to be completed. */ while (s->co_send !=NULL ) {
Signature · 1 line
2.25.1
Am 01.10.2020 um 04:21 hat mingwei geschrieben:
Quoted 8 lines
this patch fixs the problem of qemu process become crashed when the sheepdog gateway break the IO for a few seconds and then recover.
problem reproduce:
1.start a fio process in qemu to produce IOs to sheepdog gateway, whatever IO type you like. 2.kill the sheepdog gateway.
3.wait for a few seconds.
4.restart the sheepdog gateway.
5.qemu process crashed with segfault error 6.
Can you post a stack trace?
Signal 6 is not a segfault, but SIGABRT.
Quoted 16 lines
problem cause:
the last io coroutine will be destroyed after reconnect to sheepdog gateway, but the coroutine still be scheduled and thes->co_recvis still the last io coroutine pointer which had been destroyed, so when this coroutine go to coroutine context switch, it will make qemu process crashed.problem fix:
just makes->co_recv= NULL when the last io coroutine reconnect to sheepdog gateway.Signed-off-by mingwei <[email protected]>--- block/sheepdog.c | 1 + 1 file changed, 1 insertion(+)block/sheepdog.c +0
-0
block/sheepdog.c static coroutine_fn void reconnect_to_sdog(void *opaque) · line 730
Quoted 4 lines
block/sheepdog.c +1
-0@@ -727,6 +727,7 @@ static coroutine_fn void reconnect_to_sdog(void *opaque) NULL, NULL, NULL);close (s->fd); s->fd =-1 ; + s->co_recv =NULL ;
If s->co_revc != NULL before this, there is still a coroutine running that hasn't terminated yet. Don't we need to make sure that the coroutine actually terminates instead of just overwriting the pointer to it?
Otherwise, we either leak the coroutine and the memory used for its stack, or the coroutine continues to run at some point and might interfer with the operation of the new instance.
block/sheepdog.c static coroutine_fn void reconnect_to_sdog(void *opaque) · line 733
Quoted 2 lines
block/sheepdog.c +0
-0/* Wait for outstanding write requests to be completed. */ while (s->co_send !=NULL ) {
co_write_request(opaque);
}This existing code after your change is wrong, too, by the way. It potentially calls aio_co_wake() multiple times in a row, which will crash if it ends up only scheduling the coroutine instead of directly entering it.
Kevin