Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

DISCUSSION Peter Lieven 3 participants 15 messages Oct 29, 2012 – Nov 22, 2012 KVM

Share & tools

Peter Lieven

author

2012-10-29 14:09 UTC #0 lore raw

Hi,

If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI qemu-kvm crashes reliably with the following error:

Bad ram pointer 0x3039303620008000

This happens directly after the confirmation of the Timezone before the Disk is partitioned.

If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.

Here is a stack trace:

Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1> 0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2> 0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
addr1 = 93825009559312
#3 <https://github.com/sahlberg/libiscsi/issues/3> 0x000055555580a9ca in virtqueue_fill (vq=0x5555565da710, elem=0x555556722238, len=1, idx=0)
at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:240
size = 1
offset = 0
i = 0
#4 <https://github.com/sahlberg/libiscsi/issues/4> 0x000055555580abf0 in virtqueue_push (vq=0x5555565da710, elem=0x555556722238, len=1) at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:276
No locals.
#5 <https://github.com/sahlberg/libiscsi/issues/5> 0x0000555555800952 in virtio_blk_req_complete (req=0x555556722230, status=0) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:62
s = 0x5555565da640
#6 <https://github.com/sahlberg/libiscsi/issues/6> 0x00005555558010bf in virtio_blk_handle_scsi (req=0x555556722230) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:261
ret = 0
i = 1
status = 0
hdr = {interface_id = 83, dxfer_direction = -3, cmd_len = 6 '\006',
mx_sb_len = 96 '`', iovec_count = 1, dxfer_len = 56,
dxferp = 0x555556726248, cmdp = 0x2aab24b6c838 "\022\001\200",
sbp = 0x2aab1d677c30 "", timeout = 0, flags = 0, pack_id = 0,
usr_ptr = 0x0, status = 0 '\000', masked_status = 0 '\000',
msg_status = 0 '\000', sb_len_wr = 0 '\000', host_status = 0,
driver_status = 0, resid = 0, duration = 0, info = 0}
#7 <https://github.com/sahlberg/libiscsi/issues/7> 0x0000555555801724 in virtio_blk_handle_request (req=0x555556722230, mrb=0x7fffffffd9f0) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:393
type = 2
#8 <https://github.com/sahlberg/libiscsi/issues/8> 0x00005555558018c3 in virtio_blk_handle_output (vdev=0x5555565da640, vq=0x5555565da710) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:426
s = 0x5555565da640
req = 0x555556722230
mrb = {blkreq = {{sector = 0, nb_sectors = 0, qiov = 0x0, cb = 0,
opaque = 0x0, error = 0} }, num_writes = 0}
#9 <https://github.com/sahlberg/libiscsi/issues/9> 0x000055555580bd81 in virtio_queue_notify_vq (vq=0x5555565da710) at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:648
vdev = 0x5555565da640
#10 <https://github.com/sahlberg/libiscsi/issues/10> 0x000055555580d2ff in virtio_queue_host_notifier_read (n=0x5555565da75c) at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:1020
vq = 0x5555565da710
#11 <https://github.com/sahlberg/libiscsi/issues/11> 0x000055555565a47e in qemu_iohandler_poll (readfds=0x555556073160, writefds=0x5555560731e0, xfds=0x555556073260, ret=1) at iohandler.c:122
pioh = 0x555556541290
ioh = 0x7ffff0000e70
#12 <https://github.com/sahlberg/libiscsi/issues/12> 0x000055555572b742 in main_loop_wait (nonblocking=0) at main-loop.c:497 ret = 1
timeout = 4294967295
#13 <https://github.com/sahlberg/libiscsi/issues/13> 0x00005555557235e2 in main_loop () at /usr/src/qemu-kvm-1.2.0/vl.c:1643 nonblocking = false
last_io = 1
#14 <https://github.com/sahlberg/libiscsi/issues/14> 0x000055555572a21c in main (argc=42, argv=0x7fffffffe548, envp=0x7fffffffe6a0) at /usr/src/qemu-kvm-1.2.0/vl.c:3790
i = 64
snapshot = 0
linux_boot = 0
icount_option = 0x0
initrd_filename = 0x0
kernel_filename = 0x0
kernel_cmdline = 0x5555558d442a ""
boot_devices = "dc", '\000'
ds = 0x5555565465a0
dcl = 0x0
cyls = 0
heads = 0
secs = 0
translation = 0
hda_opts = 0x0
opts = 0x55555650f4b0
machine_opts = 0x55555650fcb0
olist = 0x5780f638f2e0
optind = 42
optarg = 0x7fffffffebd9 "cirrus"
loadvm = 0x0
machine = 0x555555c66780
cpu_model = 0x7fffffffeb5b "host,+x2apic,model_id=Intel(R) Xeon(R) CPU", ' ' , "L5640 @ 2.27GHz,-tsc" vga_model = 0x7fffffffebd9 "cirrus"
pid_file = 0x7fffffffeb1a "/var/run/qemu/vm-279.pid"
incoming = 0x0
show_vnc_port = 0
defconfig = true
userconfig = true
log_mask = 0x0
log_file = 0x0
mem_trace = {malloc = 0x55555572683e ,
realloc = 0x555555726896 ,
free = 0x5555557268fd , calloc = 0, try_malloc = 0,
try_realloc = 0}
trace_events = 0x0
trace_file = 0x0

Is this a regression in qemu-kvm. I remember there where some modifications regarding SCSI passthru lately. Maybe there was a problem introduced with this.

BR,
Peter

Stefan Hajnoczi

2012-10-30 08:32 UTC #1 lore raw

Quoted 1 line · Peter Lieven

Hi,

Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.

Sounds pedantic but I want to make sure this gets chalked up against the right device :).

Quoted 3 lines

If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:

Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?

Have you tried qemu-kvm.git/master?

Have you tried a local raw disk image to check whether libiscsi is involved?

Quoted 10 lines

Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:

Quoted 11 lines

Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645

buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.

It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.

Quoted 8 lines

addr1 = 93825009559312
#3 <https://github.com/sahlberg/libiscsi/issues/3>
0x000055555580a9ca in virtqueue_fill (vq=0x5555565da710, elem=0x555556722238, len=1, idx=0)
at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:240
size = 1
offset = 0
i = 0

Stefan

Peter Lieven

author

2012-10-30 09:43 UTC #2 lore raw

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-BLK -> Bad ram pointer

Quoted 5 lines · Stefan Hajnoczi

Quoted 1 line · Peter Lieven
Hi,
Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.

you are right, sorry for that.

Quoted 8 lines

Sounds pedantic but I want to make sure this gets chalked up against the right device :).
Quoted 3 lines
If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:
Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?

I use vanilla qemu-kvm 1.2.0 except for one virtio-blk related patch (CVE-2011-4127):
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=1ba1f2e319afdcb485963cd3f426fdffd1b725f2
that for some reason did not made it into qemu-kvm 1.2.0 and two aio related patchs:
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=00f78533326c5ba2e62fafada16655aa558a5520
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=2db2bfc0ccac5fd68dbf0ceb70fbc372c5d8a8c7

this is why I can circumvent the issue with scsi=off i guess.

Quoted 2 lines

Have you tried qemu-kvm.git/master?

not yet.

Quoted 3 lines

Have you tried a local raw disk image to check whether libiscsi is involved?

I have, here it does not happen. For a raw device scsi is scsi=off, isn't it?

Quoted 22 lines

Quoted 11 lines
Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:
Quoted 11 lines
Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645

Quoted 5 lines

buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.
It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.

I will collect that info for you.

Peter

Peter Lieven

author

2012-10-30 15:56 UTC #3 lore raw

In reply to Stefan Hajnoczi

Quoted 40 lines · Stefan Hajnoczi

Quoted 1 line · Peter Lieven
Hi,
Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.
Sounds pedantic but I want to make sure this gets chalked up against the right device :).
Quoted 3 lines
If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:
Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
Have you tried qemu-kvm.git/master?
Have you tried a local raw disk image to check whether libiscsi is involved?
Quoted 10 lines
Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:
Quoted 11 lines
Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645

Quoted 5 lines

buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.
It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.

(gdb) print *elem
$6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,

    2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
    140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
    2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
    12385, 93825009696000, 140737328183160, 0 <repeats 501 times>}, 
in_sg = {{
      iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
      iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
      iov_base = 0x2aab32443039, iov_len = 16}, {iov_base = 0x2aab2365c628,
      iov_len = 1}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0,
      iov_len = 0}, {iov_base = 0x2041, iov_len = 93825010788016}, {
      iov_base = 0x7ffff673f778, iov_len = 0}, {iov_base = 0x0,
      iov_len = 0} <repeats 256 times>, {iov_base = 0x1021,
      iov_len = 93825010788016}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 0} <repeats 255 times>, {iov_base = 0x0,
      iov_len = 24768}, {iov_base = 0x1020, iov_len = 93824999311936}, {
      iov_base = 0x0, iov_len = 2}, {iov_base = 0x0,
      iov_len = 0} <repeats 256 times>, {iov_base = 0x1021,
      iov_len = 93825009696000}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 0} <repeats 242 times>}, out_sg = {{
      iov_base = 0x2aab2365c608, iov_len = 16}, {iov_base = 0x2aab243fbae8,
      iov_len = 6}, {iov_base = 0x0, iov_len = 0} <repeats 11 times>, {
      iov_base = 0x0, iov_len = 33024}, {iov_base = 0x30,
      iov_len = 93825010821424}, {iov_base = 0x55555670d7a0, iov_len = 
0}, {
      iov_base = 0x55555670cbb0, iov_len = 0}, {iov_base = 0x71,
      iov_len = 93825008729792}, {iov_base = 0x55555670e960, iov_len = 
0}, {
      iov_base = 0x31, iov_len = 140737328183192}, {iov_base = 
0x7ffff673f798,
      iov_len = 0}, {iov_base = 0x555556711e20, iov_len = 80}, {
      iov_base = 0x20, iov_len = 93825010821584}, {iov_base = 0x0,
      iov_len = 33184}, {iov_base = 0x30, iov_len = 93825010821536}, {
      iov_base = 0x55555670e840, iov_len = 0}, {iov_base = 0x55555670e1b0,
      iov_len = 0}, {iov_base = 0x41, iov_len = 93825010821584}, {
      iov_base = 0x55555670eb20, iov_len = 32}, {iov_base = 0x20,
      iov_len = 93825010821920}, {iov_base = 0x0, iov_len = 33296}, {
      iov_base = 0x30, iov_len = 93825010821872}, {iov_base = 
0x55555670e8b0,
      iov_len = 0}, {iov_base = 0x55555670dc68, iov_len = 0}, {
      iov_base = 0x191, iov_len = 93825009696736}, {iov_base = 
0x55555670eb20,
      iov_len = 0}, {iov_base = 0x21, iov_len = 93825010826352}, {
      iov_base = 0x55555670e880, iov_len = 64}, {iov_base = 0x30,
      iov_len = 93825010821200}, {iov_base = 0x55555670e920, iov_len = 
0}, {
      iov_base = 0x55555670e5c8, iov_len = 0}, {iov_base = 0x41,
      iov_len = 93825008729792}, {iov_base = 0x55555670e9d0, iov_len = 
32}, {
      iov_base = 0x20, iov_len = 93825010821696}, {iov_base = 0x0,
      iov_len = 176}, {iov_base = 0x30, iov_len = 93825010821648}, {
      iov_base = 0x55555670e990, iov_len = 0}, {iov_base = 0x55555670e080,
      iov_len = 0}, {iov_base = 0x41, iov_len = 93825008729792}, {
      iov_base = 0x55555670eb20, iov_len = 32}, {iov_base = 0x20,
      iov_len = 93825010822032}, {iov_base = 0x0, iov_len = 288}, {
      iov_base = 0x30, iov_len = 93825010821984}, {iov_base = 
0x55555670ea00,
      iov_len = 0}, {iov_base = 0x55555670e000, iov_len = 0}, {
      iov_base = 0x41, iov_len = 93825010821808}, {iov_base = 
0x555556602590,
      iov_len = 32}, {iov_base = 0x20, iov_len = 93825009373648}, {
      iov_base = 0x0, iov_len = 33744}, {iov_base = 0x30,
      iov_len = 93825009373680}, {iov_base = 0x55555670ea70, iov_len = 
0}, {
      iov_base = 0x55555670da18, iov_len = 0}, {iov_base = 0x17271,
      iov_len = 93825009696736}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 93825010821920}, {iov_base = 
0x55555670ec00,
      iov_len = 64}, {iov_base = 0x30, iov_len = 93825010822096}, {
      iov_base = 0x55555670eae0, iov_len = 0}, {iov_base = 0x55555670da48,
      iov_len = 0}, {iov_base = 0xb1, iov_len = 93825009696736}, {
      iov_base = 0x5555567066b0, iov_len = 0}, {iov_base = 0x21,
      iov_len = 93825010822032}, {iov_base = 0x55555670e8f0, iov_len = 
64}, {
      iov_base = 0x30, iov_len = 93825010821312}, {iov_base = 
0x55555670eb50,
      iov_len = 0}, {iov_base = 0x55555670df90, iov_len = 0}, {
      iov_base = 0x41, iov_len = 93825008729792}, {iov_base = 
0x55555670eab0,
      iov_len = 32}, {iov_base = 0x20, iov_len = 93825010821808}, {
      iov_base = 0x0, iov_len = 288}, {iov_base = 0x30,
      iov_len = 93825010821760}, {iov_base = 0x55555670ebc0, iov_len = 
0}, {
      iov_base = 0x55555670da40, iov_len = 0}, {iov_base = 0x17121,
      iov_len = 93825009372400}, {iov_base = 0x5555565fbfe0, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 140737328184728}, {iov_base = 
0x5555565f9ea0,
      iov_len = 0}, {iov_base = 0x0, iov_len = 0} <repeats 255 times>, {
      iov_base = 0x0, iov_len = 4160}, {iov_base = 0x30,
      iov_len = 93825009721504}, {iov_base = 0x0, iov_len = 
93825010826368}, {
      iov_base = 0x3, iov_len = 3}, {iov_base = 0x160b1,
      iov_len = 93825009372400}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 93825010826304}, {iov_base = 
0x55555670fc50,
      iov_len = 0}, {iov_base = 0x55555670edd0, iov_len = 0}, {
      iov_base = 0x16061, iov_len = 93825009372400}, {
      iov_base = 0x7ffff673f778, iov_len = 0}, {iov_base = 0x0,
      iov_len = 93825009688224}, {iov_base = 0x7ffff673fd98,
      iov_len = 93825010826464}, {iov_base = 0x55555670fce0, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 0} <repeats 254 times>, {iov_base = 0x0,
      iov_len = 4160}, {iov_base = 0x30, iov_len = 93825010830976}, {
      iov_base = 0x55555670fcf0, iov_len = 0}, {iov_base = 0x55555670f060,
      iov_len = 0}, {iov_base = 0x71, iov_len = 93825008729792}, {
      iov_base = 0x555556710eb0, iov_len = 0}, {iov_base = 0x31,
      iov_len = 93825010821120}, {iov_base = 0x7ffff673f798,
      iov_len = 93825010821136}, {iov_base = 0x5555565fbbe8, iov_len = 
80}, {
      iov_base = 0x20, iov_len = 93825010831136}, {iov_base = 0x0,
      iov_len = 4320}, {iov_base = 0x30, iov_len = 93825010831088}, {
      iov_base = 0x555556710d90, iov_len = 0}, {iov_base = 0x555556710700,
      iov_len = 0}, {iov_base = 0x41, iov_len = 93825010831136}, {
      iov_base = 0x555556711070, iov_len = 32}, {iov_base = 0x20,
      iov_len = 93825010831472}, {iov_base = 0x0, iov_len = 4432}, {
      iov_base = 0x30, iov_len = 93825010831424}, {iov_base = 
0x555556710e00,
      iov_len = 0}, {iov_base = 0x5555567101b8, iov_len = 0}, {
      iov_base = 0x191, iov_len = 93825009372400}, {iov_base = 
0x555556711070,
      iov_len = 0}, {iov_base = 0x21, iov_len = 93825009374992}, {
      iov_base = 0x555556710dd0, iov_len = 64}, {iov_base = 0x30,
      iov_len = 93825010830752}, {iov_base = 0x555556710e70, iov_len = 
0}, {
      iov_base = 0x555556710b18, iov_len = 0}, {iov_base = 0x41,
      iov_len = 93825008729792}, {iov_base = 0x555556710f20, iov_len = 
32}, {
      iov_base = 0x20, iov_len = 93825010831248}, {iov_base = 0x0,
      iov_len = 176}, {iov_base = 0x30, iov_len = 93825010831200}, {
      iov_base = 0x555556710ee0, iov_len = 0}, {iov_base = 0x5555567105d0,
      iov_len = 0}, {iov_base = 0x41, iov_len = 93825008729792}, {
      iov_base = 0x555556711070, iov_len = 32}, {iov_base = 0x20,
      iov_len = 93825010831584}, {iov_base = 0x0, iov_len = 288}, {
      iov_base = 0x30, iov_len = 93825010831536}, {iov_base = 
0x555556710f50,
      iov_len = 0}, {iov_base = 0x555556710550, iov_len = 0}, {
      iov_base = 0x41, iov_len = 93825010831360}, {iov_base = 
0x55555670fc70,
      iov_len = 32}, {iov_base = 0x20, iov_len = 93825010826432}, {
      iov_base = 0x0, iov_len = 4880}, {iov_base = 0x30,
      iov_len = 93825010826384}, {iov_base = 0x555556710fc0, iov_len = 
0}, {
      iov_base = 0x55555670ff68, iov_len = 0}, {iov_base = 0x14d21,
      iov_len = 93825009372400}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 93825010831472}, {iov_base = 
0x555556711150,
      iov_len = 64}, {iov_base = 0x30, iov_len = 93825010831648}, {
      iov_base = 0x555556711030, iov_len = 0}, {iov_base = 0x55555670ff98,
      iov_len = 0}, {iov_base = 0xb1, iov_len = 93825009372400}, {
      iov_base = 0x55555670fcc0, iov_len = 0}, {iov_base = 0x21,
      iov_len = 93825010831584}, {iov_base = 0x555556710e40, iov_len = 
64}, {
      iov_base = 0x30, iov_len = 93825010830864}, {iov_base = 
0x5555567110a0,
      iov_len = 0}, {iov_base = 0x5555567104e0, iov_len = 0}, {
      iov_base = 0x41, iov_len = 93825010831696}, {iov_base = 
0x555556711000,
      iov_len = 32}, {iov_base = 0x20, iov_len = 93825010831360}, {
      iov_base = 0x0, iov_len = 288}, {iov_base = 0x30,
      iov_len = 93825010831312}, {iov_base = 0x555556711110, iov_len = 
0}, {
      iov_base = 0x55555670ff90, iov_len = 0}, {iov_base = 0x14bd1,
      iov_len = 93825008729792}, {iov_base = 0x55555670fc70, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 140737328185080}, {iov_base = 
0x7ffff673fef8,
      iov_len = 93825010831728}, {iov_base = 0x555556711170, iov_len = 
0}, {
      iov_base = 0x0, iov_len = 0} <repeats 255 times>...}}

Stefan Hajnoczi

2012-10-30 18:27 UTC #4 lore raw

Quoted 46 lines · Peter Lieven

Quoted 45 lines · Stefan Hajnoczi
Quoted 2 lines · Peter Lieven
Hi,
Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.
Sounds pedantic but I want to make sure this gets chalked up against the right device :).
Quoted 3 lines
If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:
Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
Have you tried qemu-kvm.git/master?
Have you tried a local raw disk image to check whether libiscsi is involved?
Quoted 21 lines
Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:
Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645

Quoted 8 lines

Quoted 5 lines
buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.
It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.
(gdb) print *elem

Great, thanks for providing this info:

Quoted 5 lines

$6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,

    2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
    140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
    2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
    12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},

Up to here everything is fine.

Quoted 4 lines

in_sg =

{{
      iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
      iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {

The fields are bogus, in_sg has been overwritten with ASCII data. Unfortunately I don't see any hint of where this ASCII data came from yet.

The hdr fields you provided in stack frame #6 show that in_sg was overwritten during or after the bdrv_ioctl() call. We pulled valid data out of the vring and mapped buffers correctly. But something is overwriting in_sg and when we complete the request we blow up due to the bogus values.

Please post your full qemu-kvm command-line.

Please also post the exact qemu-kvm version you are using. I can see it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g. distro packages may carry patches so the full package version information would be useful)?

Thanks,
Stefan

Peter Lieven

author

2012-10-30 19:37 UTC #5 lore raw

Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:

Quoted 66 lines · Peter Lieven

Quoted 41 lines · Stefan Hajnoczi
Quoted 40 lines · Peter Lieven
Quoted 1 line
Hi,
Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.
Sounds pedantic but I want to make sure this gets chalked up against the right device :).
Quoted 3 lines
If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:
Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
Have you tried qemu-kvm.git/master?
Have you tried a local raw disk image to check whether libiscsi is involved?
Quoted 21 lines
Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:
Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
Quoted 7 lines
Quoted 5 lines
buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.
It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.
(gdb) print *elem
Great, thanks for providing this info:
Quoted 5 lines
$6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
     2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
     140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
     2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
     12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},
Up to here everything is fine.
Quoted 4 lines
in_sg =
{{
       iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
       iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
The fields are bogus, in_sg has been overwritten with ASCII data. Unfortunately I don't see any hint of where this ASCII data came from yet.

Quoted 5 lines

The hdr fields you provided in stack frame #6 show that in_sg was overwritten during or after the bdrv_ioctl() call. We pulled valid data out of the vring and mapped buffers correctly. But something is overwriting in_sg and when we complete the request we blow up due to the bogus values.

Ok. What I have to mention. I've been testing with qemu-kvm 1.2.0 and libiscsi for a few weeks now. Its been very stable. The only thing it blows up is during the debian/ubuntu installer. Ubuntu itself for instance is running flawlessly. My guess is that the installer is probing for something. The installer itself also runs flawlessly when I disable scsi passthru with scsi=off.

Quoted 2 lines

Please post your full qemu-kvm command-line.

/usr/bin/qemu-kvm-1.2.0 -net tap,vlan=164,script=no,downscript=no,ifname=tap0 -net nic,vlan=164,model=e1000,macaddr=52:54:00:ff:01:35 -iscsi initiator-name=iqn.2005-03.org.virtual-core:0025b51f001c -drive format=iscsi,file=iscsi://172.21.200.56/iqn.2001-05.com.equallogic:0-8a0906-335f4e007-d29001a3355508e8-libiscsi-test-hd0/0,if=virtio,cache=none,aio=native -m 2048 -smp 2,sockets=1,cores=2,threads=1 -monitor tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name 'libiscsi-debug' -boot order=dc,menu=off -k de -pidfile /var/run/qemu/vm-280.pid -mem-path /hugepages -mem-prealloc -cpu host,+x2apic,model_id='Intel(R) Xeon(R) CPU L5640 @ 2.27GHz',-tsc -rtc base=utc -usb -usbdevice tablet -no-hpet -vga cirrus

Quoted 5 lines

Please also post the exact qemu-kvm version you are using. I can see it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g. distro packages may carry patches so the full package version information would be useful)?

I use vanilly qemu-kvm 1.2.0 with some cherry picked patches. I will retry with untouched qemu-kvm 1.2.0 and latest git tomorrow at latest.

Quoted 3 lines

Thanks,
Stefan

Thank you, too
Peter

ronnie sahlberg

2012-10-30 21:09 UTC #6 lore raw

But older distros/kernels work fine?

Can you take a network trace?

About half a year there was an issue where recent kernels had added support to start using new scsi opcodes, but the qemu functions that determine "which transfer direction is used for this opcode" had not yet been updated, so that the opcode was sent with the wrong transfer direction.

That caused the guests memory to be overwritten and crash.

I dont have (easy) access to the git tree right now, but it was a patch for the ATA_PASSTHROUGH command that fixed that.

Could you take a network trace and check maybe this is something similar ? I.e. does the guest send an "unusual" scsi opcode just before the crash ?

regards
ronnie sahlberg

Quoted 123 lines · Peter Lieven

Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:
Quoted 78 lines · Peter Lieven
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Peter Lieven
Quoted 2 lines
Hi,
Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.
Sounds pedantic but I want to make sure this gets chalked up against the right device :).
Quoted 3 lines
If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:
Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
Have you tried qemu-kvm.git/master?
Have you tried a local raw disk image to check whether libiscsi is involved?
Quoted 21 lines
Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:
Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
Quoted 8 lines
Quoted 5 lines
buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.
It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.
(gdb) print *elem
Great, thanks for providing this info:
Quoted 7 lines
$6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
     2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
     140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
     2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512
times>,
     12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},
Up to here everything is fine.
Quoted 4 lines
in_sg =
{{
       iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
       iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
The fields are bogus, in_sg has been overwritten with ASCII data. Unfortunately I don't see any hint of where this ASCII data came from yet.
Quoted 5 lines
The hdr fields you provided in stack frame #6 show that in_sg was overwritten during or after the bdrv_ioctl() call. We pulled valid data out of the vring and mapped buffers correctly. But something is overwriting in_sg and when we complete the request we blow up due to the bogus values.
Ok. What I have to mention. I've been testing with qemu-kvm 1.2.0 and libiscsi for a few weeks now. Its been very stable. The only thing it blows up is during the debian/ubuntu installer. Ubuntu itself for instance is running flawlessly. My guess is that the installer is probing for something. The installer itself also runs flawlessly when I disable scsi passthru with scsi=off.
Quoted 2 lines
Please post your full qemu-kvm command-line.
/usr/bin/qemu-kvm-1.2.0 -net
tap,vlan=164,script=no,downscript=no,ifname=tap0 -net
nic,vlan=164,model=e1000,macaddr=52:54:00:ff:01:35 -iscsi
initiator-name=iqn.2005-03.org.virtual-core:0025b51f001c -drive format=iscsi,file=iscsi://172.21.200.56/iqn.2001-05.com.equallogic:0-8a0906-335f4e007-d29001a3355508e8-libiscsi-test-hd0/0,if=virtio,cache=none,aio=native -m 2048 -smp 2,sockets=1,cores=2,threads=1 -monitor
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
'libiscsi-debug' -boot order=dc,menu=off -k de -pidfile
/var/run/qemu/vm-280.pid -mem-path /hugepages -mem-prealloc -cpu
host,+x2apic,model_id='Intel(R) Xeon(R) CPU L5640 @ 2.27GHz',-tsc
-rtc base=utc -usb -usbdevice tablet -no-hpet -vga cirrus
Quoted 5 lines
Please also post the exact qemu-kvm version you are using. I can see it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g. distro packages may carry patches so the full package version information would be useful)?
I use vanilly qemu-kvm 1.2.0 with some cherry picked patches. I will retry with untouched qemu-kvm 1.2.0 and latest git tomorrow at latest.
Quoted 4 lines
Thanks,
Stefan
Thank you, too
Peter

Stefan Hajnoczi

2012-10-31 05:48 UTC #7 lore raw

Quoted 10 lines · ronnie sahlberg

About half a year there was an issue where recent kernels had added support to start using new scsi opcodes, but the qemu functions that determine "which transfer direction is used for this opcode" had not yet been updated, so that the opcode was sent with the wrong transfer direction.
That caused the guests memory to be overwritten and crash.
I dont have (easy) access to the git tree right now, but it was a patch for the ATA_PASSTHROUGH command that fixed that.

This patch?

http://patchwork.ozlabs.org/patch/174946/

Stefan

ronnie sahlberg

2012-10-31 14:08 UTC #8 lore raw

Quoted 10 lines · Stefan Hajnoczi

Quoted 8 lines · ronnie sahlberg
About half a year there was an issue where recent kernels had added support to start using new scsi opcodes, but the qemu functions that determine "which transfer direction is used for this opcode" had not yet been updated, so that the opcode was sent with the wrong transfer direction.
That caused the guests memory to be overwritten and crash.

Quoted 8 lines

Quoted 2 lines
I dont have (easy) access to the git tree right now, but it was a patch for the ATA_PASSTHROUGH command that fixed that.
This patch?
http://patchwork.ozlabs.org/patch/174946/
Stefan

This is the one I was thinking about :
381b634c275ca1a2806e97392527bbfc01bcb333

But that also crashed when using local /dev/sg* devices.

Peter Lieven

author

2012-11-05 15:19 UTC #9 lore raw

Am 31.10.2012 um 15:08 schrieb ronnie sahlberg:

Quoted 16 lines · Stefan Hajnoczi

Quoted 15 lines
Quoted 10 lines · ronnie sahlberg
About half a year there was an issue where recent kernels had added support to start using new scsi opcodes, but the qemu functions that determine "which transfer direction is used for this opcode" had not yet been updated, so that the opcode was sent with the wrong transfer direction.
That caused the guests memory to be overwritten and crash.
I dont have (easy) access to the git tree right now, but it was a patch for the ATA_PASSTHROUGH command that fixed that.
This patch?

Quoted 8 lines

Quoted 3 lines
http://patchwork.ozlabs.org/patch/174946/
Stefan
This is the one I was thinking about :
381b634c275ca1a2806e97392527bbfc01bcb333
But that also crashed when using local /dev/sg* devices.

I was using a local LVM Volume not an iSCSI disk.

I added debugging output and breakpoints to scsi_cmd_xfer_mode(). The function is not called.

Peter

Peter Lieven

author

2012-11-08 15:26 UTC #10 lore raw

In reply to ronnie sahlberg

Has anyone any other idea what the cause could be or where to start?

Peter

Am 31.10.2012 um 15:08 schrieb ronnie sahlberg:

Quoted 24 lines · Stefan Hajnoczi

Quoted 10 lines
Quoted 8 lines · ronnie sahlberg
About half a year there was an issue where recent kernels had added support to start using new scsi opcodes, but the qemu functions that determine "which transfer direction is used for this opcode" had not yet been updated, so that the opcode was sent with the wrong transfer direction.
That caused the guests memory to be overwritten and crash.
Quoted 8 lines
Quoted 2 lines
I dont have (easy) access to the git tree right now, but it was a patch for the ATA_PASSTHROUGH command that fixed that.
This patch?
http://patchwork.ozlabs.org/patch/174946/
Stefan
This is the one I was thinking about :
381b634c275ca1a2806e97392527bbfc01bcb333
But that also crashed when using local /dev/sg* devices.

Stefan Hajnoczi

2012-11-19 17:20 UTC #11 lore raw

Quoted 1 line · Peter Lieven

Has anyone any other idea what the cause could be or where to start?

Hi Peter,
I suggested posting the source tree you are building. Since you have applied patches yourself no one else is able to follow along with the gdb output or reproduce the issue accurately.

Stefan

Peter Lieven

author

2012-11-22 14:10 UTC #12 lore raw

Quoted 7 lines · Stefan Hajnoczi

Quoted 1 line · Peter Lieven
Has anyone any other idea what the cause could be or where to start?
Hi Peter,
I suggested posting the source tree you are building. Since you have applied patches yourself no one else is able to follow along with the gdb output or reproduce the issue accurately.

Sorry for the late reply, I used qemu git at e24dc9feb0d68142d54dc3c097f57588836d1338 and libiscsi git at 3b3036b9dae55f0c3eef9d75db89c7b78f637a12.

The cmdline:
qemu-system-x86_64 -enable-kvm -m 1024 -drive if=virtio,file=iscsi://172.21.200.56/iqn.2001-05.com.equallogic:0-8a0906-62ff4e007-e4a3c8908af50839-test-3000g/0 -cdrom ubuntu-12.04.1-server-amd64.iso -vnc :1

The vm crashes with:

Bad ram pointer 0x7fd220008000

after the user settings and timezone config when loading the module libdmraid1.0.0.rc16-udeb

I hope this helps to reproduce.

Peter

Quoted 3 lines

Stefan

Stefan Hajnoczi

2012-10-31 05:53 UTC #13 lore raw

In reply to Peter Lieven

Quoted 8 lines · Peter Lieven

Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:
Quoted 4 lines
Please also post the exact qemu-kvm version you are using. I can see it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g. distro packages may carry patches so the full package version information would be useful)?
I use vanilly qemu-kvm 1.2.0 with some cherry picked patches. I will retry with untouched qemu-kvm 1.2.0 and latest git tomorrow at latest.

Please post the source tree including applied patches. For example, you could push to a public git repo. This is necessary so that we can dig into the source tree to figure out where the bug lies.

There are some other interesting questions that Ronnie has raised, so perhaps they will lead to the answer if you cannot publish the tree at the moment.

My next question is: have you tested qemu.git/master?

Stefan

Peter Lieven

author

2012-10-30 20:24 UTC #14 lore raw

In reply to Stefan Hajnoczi

Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:

Quoted 72 lines · Peter Lieven

Quoted 41 lines · Stefan Hajnoczi
Quoted 40 lines · Peter Lieven
Quoted 1 line
Hi,
Bug subject should be virtio-blk, not virtio-scsi. virtio-scsi is a different virtio device type from virtoi-blk and is not present in the backtrace you posted.
Sounds pedantic but I want to make sure this gets chalked up against the right device :).
Quoted 3 lines
If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage backend that supports iSCSI
qemu-kvm crashes reliably with the following error:
Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
Have you tried qemu-kvm.git/master?
Have you tried a local raw disk image to check whether libiscsi is involved?
Quoted 21 lines
Bad ram pointer 0x3039303620008000
This happens directly after the confirmation of the Timezone before the Disk is partitioned.
If I specify -global virtio-blk-pci.scsi=off in the cmdline this does not happen.
Here is a stack trace:
Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1>
0x00005555557b751d in qemu_ram_addr_from_host_nofail ( ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835 ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2>
0x00005555557b9177 in cpu_physical_memory_unmap ( buffer=0x3039303620008000, len=4986663671065686081, is_write=1, access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
Quoted 7 lines
Quoted 5 lines
buffer and len are ASCII junk. It appears to be hex digits and it's not clear where they come from.
It would be interesting to print *elem one stack frame up in #3 virtqueue_fill() to show the iovecs and in/out counts.
(gdb) print *elem
Great, thanks for providing this info:
Quoted 5 lines
$6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
     2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
     140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
     2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
     12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},
Up to here everything is fine.
Quoted 4 lines
in_sg =
{{
       iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
       iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
The fields are bogus, in_sg has been overwritten with ASCII data. Unfortunately I don't see any hint of where this ASCII data came from yet.
The hdr fields you provided in stack frame #6 show that in_sg was overwritten during or after the bdrv_ioctl() call. We pulled valid data out of the vring and mapped buffers correctly. But something is overwriting in_sg and when we complete the request we blow up due to the bogus values.

Quoted 6 lines

Please post your full qemu-kvm command-line.
Please also post the exact qemu-kvm version you are using. I can see it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g. distro packages may carry patches so the full package version information would be useful)?

Stefan, Ronnie, if I do remove the following patch from my cherry-picked patches its working again:

iSCSI: We need to support SG_IO also from iscsi_ioctl()

Peter