boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

report wanghaibin 3 participants 6 messages Jul 18, 2017 KVM arm64

lore

Share & tools

lore

wanghaibin

author

2017-07-18 04:07 UTC #0 lore raw

Hi, all:

I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt sections 2.3)

Here is part of qemu cmd:
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4

A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it. Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge. Boot the vm, it's failed.

I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed. Of course, I think this commit must be correct surely.

(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration). This commit brings in that the four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
(Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count), So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.

Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.

Thanks.

Marc Zyngier

2017-07-18 09:15 UTC #1 lore raw

Quoted 16 lines · wanghaibin

Hi, all:
I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt sections 2.3)
Here is part of qemu cmd:
(quoted diff) +0 -3
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it. Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge. Boot the vm, it's failed.
I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed. Of course, I think this commit must be correct surely.

Quoted 6 lines

(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration). This commit brings in that the four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
(Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count), So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.

+Robin, who is the author of that patch.

Regarding (2), the number of MSIs should be the total number of devices that are going to generate the same DevID. Since the bridge is non-transparent, everything behind it aliases with it. So you should probably see all the virtio devices and the bridges themselves being counted. If that's not the case, then "we have a bug"(tm).

Can you please post your full qemu cmd line so that we can reproduce it and investigate the issue?

Thanks,

M.

Signature · 1 line

Jazz is not dead. It just smells funny...

Robin Murphy

2017-07-18 10:02 UTC #2 lore raw

Quoted 5 lines · Marc Zyngier

Quoted 4 lines · wanghaibin
Hi, all:
I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt sections 2.3)

Quoted 7 lines

Quoted 7 lines
Here is part of qemu cmd:
(quoted diff) +0 -3
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it. Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge. Boot the vm, it's failed.

What's the nature of the failure? Does it hit some actual error case in the GIC code, or does it simply hang up probing the virtio devices because interrupts never arrive?

Quoted 8 lines

Quoted 8 lines
I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed. Of course, I think this commit must be correct surely.
(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration). This commit brings in that the four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
(Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count), So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.

Now, at worst that patch *should* result in the same number of vectors being reserved as before - never fewer. Does anything change with it reverted?

Quoted 1 line

Quoted 1 line
Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.

Note that there are further issues with PCI RID to DevID mappings in the face of aliases[1], but I think the current code does happen to work out OK for the PCI-PCIe bridge case already.

Quoted 10 lines

+Robin, who is the author of that patch.
Regarding (2), the number of MSIs should be the total number of devices that are going to generate the same DevID. Since the bridge is non-transparent, everything behind it aliases with it. So you should probably see all the virtio devices and the bridges themselves being counted. If that's not the case, then "we have a bug"(tm).
Can you please post your full qemu cmd line so that we can reproduce it and investigate the issue?

Yes, that would be good.

Robin.

[1] https://patchwork.ozlabs.org/patch/769303/

Quoted 4 lines

Thanks,
	M.

wanghaibin

author

2017-07-18 11:07 UTC #3 lore raw

Quoted 11 lines · Robin Murphy

Quoted 10 lines · Marc Zyngier
Quoted 9 lines · wanghaibin
Hi, all:
I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt sections 2.3)
Here is part of qemu cmd:
(quoted diff) +0 -3
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4

Quoted 6 lines

Quoted 2 lines
Quoted 2 lines
A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it. Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge. Boot the vm, it's failed.
What's the nature of the failure? Does it hit some actual error case in the GIC code, or does it simply hang up probing the virtio devices because interrupts never arrive?

Qemu cmdline, xml info, qemu version info, guest kernel version info at the bottom of this mail.

Guest hang log:

[  242.740171] INFO: task kworker/u16:4:446 blocked for more than 120 seconds.
[  242.741102]       Not tainted 4.12.0+ #18
[  242.741619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.742610] kworker/u16:4   D    0   446      2 0x00000000
[  242.743339] Workqueue: scsi_tmf_0 scmd_eh_abort_handler
[  242.744014] Call trace:
[  242.744375] [<ffff0000080856bc>] __switch_to+0x94/0xa8
[  242.745042] [<ffff00000892a3f8>] __schedule+0x1a0/0x5e0
[  242.745716] [<ffff00000892a870>] schedule+0x38/0xa0
[  242.746346] [<ffff00000892da84>] schedule_timeout+0x194/0x2b8
[  242.747092] [<ffff00000892b438>] wait_for_common+0xa0/0x148
[  242.747810] [<ffff00000892b4f4>] wait_for_completion+0x14/0x20
[  242.748595] [<ffff0000085e78d8>] virtscsi_tmf.constprop.15+0x88/0xf0
[  242.749408] [<ffff0000085e79dc>] virtscsi_abort+0x9c/0xb8
[  242.750099] [<ffff0000085db4dc>] scmd_eh_abort_handler+0x5c/0x108
[  242.750887] [<ffff0000080d8594>] process_one_work+0x124/0x2a8
[  242.751618] [<ffff0000080d8774>] worker_thread+0x5c/0x3d8
[  242.752330] [<ffff0000080de7b4>] kthread+0xfc/0x128
[  242.752960] [<ffff000008082ec0>] ret_from_fork+0x10/0x50

But I still doubt the total vector count takes this problem in, I add some log in guest:
In guest boot, pci_dev: 02:04:00(virtio-gpu-pci) first load, and only alloc 4 ITT entries, the log as follow:

[    0.986233] ~~its_pci_msi_prepare:pci dev: 2,32, nvec:3~~
[    0.998952] ~~its_pci_msi_prepare:devid:8,alias count:4~~
[    1.000028] **its_msi_prepare:devid:8, nves:4**
[    1.001001] ##its_create_device:devid: 8, ITT 4 entries, 2 bits, lpi base:8192, nr:32##
[    1.002585] **its_msi_prepare:ITT 4 entries, 2 bits**
[    1.003593] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.004880] ID:0 pID:8192 vID:52
[    1.005529] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.006777] ID:1 pID:8193 vID:53
[    1.007437] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.008718] ID:2 pID:8194 vID:54
[    1.009366] !!msi_domain_alloc_irqs: to active!!
[    1.010281] ^^^SEND mapti: hwirq:8192,event:0^^
[    1.011224] !!msi_domain_alloc_irqs: to active!!
[    1.012161] ^^^SEND mapti: hwirq:8193,event:1^^
[    1.013095] !!msi_domain_alloc_irqs: to active!!
[    1.014013] ^^^SEND mapti: hwirq:8194,event:2^^

and the guest booted continue, when load the pci_dev: 02:03:00 (virtio-scsi), the log shows it shared the same devid with virtio-gpu-pci, shared ite_dev, reusing ITT. So that, the virtio-gpu-pci dev only alloc 4 ITT, and the virtio-scsi send mapti with eventid 5/6, this will be captured by Eric's commit:
guest log:

[    1.057978] !!msi_domain_alloc_irqs: to prepare: nvec:4!!
[    1.072773] ~~its_pci_msi_prepare:devid:8,alias count:5~~
[    1.073943] **its_msi_prepare:devid:8, nves:5**
[    1.074850] **its_msi_prepare:Reusing ITT for devID:8**
[    1.075873] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.077154] ID:3 pID:8195 vID:55
[    1.077813] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.079044] ID:4 pID:8196 vID:56
[    1.079683] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.080947] ID:5 pID:8197 vID:57
[    1.081592] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.082825] ID:6 pID:8198 vID:58

part of Eric's commit:

@@ -784,6 +788,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,

        if (!device)
                return E_ITS_MAPTI_UNMAPPED_DEVICE;

+ if (event_id >= BIT_ULL(device->num_eventid_bits))
+ return E_ITS_MAPTI_ID_OOR;

Thanks!

Quoted 28 lines

Quoted 9 lines
Quoted 9 lines
I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed. Of course, I think this commit must be correct surely.
(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration). This commit brings in that the four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
(Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count), So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
Now, at worst that patch *should* result in the same number of vectors being reserved as before - never fewer. Does anything change with it reverted?
Quoted 1 line
Quoted 1 line
Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.
Note that there are further issues with PCI RID to DevID mappings in the face of aliases[1], but I think the current code does happen to work out OK for the PCI-PCIe bridge case already.
Quoted 8 lines
+Robin, who is the author of that patch.
Regarding (2), the number of MSIs should be the total number of devices that are going to generate the same DevID. Since the bridge is non-transparent, everything behind it aliases with it. So you should probably see all the virtio devices and the bridges themselves being counted. If that's not the case, then "we have a bug"(tm).

Quoted 4 lines

Quoted 2 lines
Can you please post your full qemu cmd line so that we can reproduce it and investigate the issue?
Yes, that would be good.

I used the qemu version: 2.9.50,
guest linux version: Linux 4.12

xml :

<domain type='kvm' id='5'>
<name>abu</name> <uuid>76365c65-7ee7-43ff-bb57-f0f80b75323a</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>8</vcpu> <resource>
<partition>/machine</partition>
</resource>
<os>

    <type arch='aarch64' machine='virt-2.9'>hvm</type>
    <kernel>/mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image</kernel>
    <cmdline>console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='hpet' present='no'/>
    <timer name='pit' tickpolicy='delay'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/wanghaibin/vm-res/euler_b500.raw'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ehci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:60:6b:1d'/>
      <source bridge='br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='c216af51-9088-4bf8-ba04-cf6172cb3753'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </video>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>

</seclabel>
</domain>

qemu cmdline :

/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=abu,debug-threads=on -S
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abu/master-key.aes -machine virt-2.9,accel=kvm,usb=off,gic-version=3 -cpu host -m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 76365c65-7ee7-43ff-bb57-f0f80b75323a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -kernel /mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image -append console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -drive file=/mnt/wanghaibin/vm-res/euler_b500.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1 -serial pty -device usb-tablet,id=input0 -device usb-kbd,id=input1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4 -msg timestamp=on

Thanks

Quoted 13 lines

Robin.
[1] https://patchwork.ozlabs.org/patch/769303/
Quoted 4 lines
Thanks,
	M.
.

Robin Murphy

2017-07-18 11:22 UTC #4 lore raw

Quoted 94 lines · wanghaibin

Quoted 11 lines · Robin Murphy

Quoted 10 lines · Marc Zyngier
Quoted 9 lines · wanghaibin
Hi, all:
I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt sections 2.3)
Here is part of qemu cmd:
(quoted diff) +0 -3
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4

Quoted 6 lines

Quoted 2 lines
Quoted 2 lines
A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it. Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge. Boot the vm, it's failed.
What's the nature of the failure? Does it hit some actual error case in the GIC code, or does it simply hang up probing the virtio devices because interrupts never arrive?

Qemu cmdline, xml info, qemu version info, guest kernel version info at the bottom of this mail.

Guest hang log:

[  242.740171] INFO: task kworker/u16:4:446 blocked for more than 120 seconds.
[  242.741102]       Not tainted 4.12.0+ #18
[  242.741619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.742610] kworker/u16:4   D    0   446      2 0x00000000
[  242.743339] Workqueue: scsi_tmf_0 scmd_eh_abort_handler
[  242.744014] Call trace:
[  242.744375] [<ffff0000080856bc>] __switch_to+0x94/0xa8
[  242.745042] [<ffff00000892a3f8>] __schedule+0x1a0/0x5e0
[  242.745716] [<ffff00000892a870>] schedule+0x38/0xa0
[  242.746346] [<ffff00000892da84>] schedule_timeout+0x194/0x2b8
[  242.747092] [<ffff00000892b438>] wait_for_common+0xa0/0x148
[  242.747810] [<ffff00000892b4f4>] wait_for_completion+0x14/0x20
[  242.748595] [<ffff0000085e78d8>] virtscsi_tmf.constprop.15+0x88/0xf0
[  242.749408] [<ffff0000085e79dc>] virtscsi_abort+0x9c/0xb8
[  242.750099] [<ffff0000085db4dc>] scmd_eh_abort_handler+0x5c/0x108
[  242.750887] [<ffff0000080d8594>] process_one_work+0x124/0x2a8
[  242.751618] [<ffff0000080d8774>] worker_thread+0x5c/0x3d8
[  242.752330] [<ffff0000080de7b4>] kthread+0xfc/0x128
[  242.752960] [<ffff000008082ec0>] ret_from_fork+0x10/0x50

[    0.986233] ~~its_pci_msi_prepare:pci dev: 2,32, nvec:3~~
[    0.998952] ~~its_pci_msi_prepare:devid:8,alias count:4~~
[    1.000028] **its_msi_prepare:devid:8, nves:4**
[    1.001001] ##its_create_device:devid: 8, ITT 4 entries, 2 bits, lpi base:8192, nr:32##
[    1.002585] **its_msi_prepare:ITT 4 entries, 2 bits**
[    1.003593] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.004880] ID:0 pID:8192 vID:52
[    1.005529] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.006777] ID:1 pID:8193 vID:53
[    1.007437] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.008718] ID:2 pID:8194 vID:54
[    1.009366] !!msi_domain_alloc_irqs: to active!!
[    1.010281] ^^^SEND mapti: hwirq:8192,event:0^^
[    1.011224] !!msi_domain_alloc_irqs: to active!!
[    1.012161] ^^^SEND mapti: hwirq:8193,event:1^^
[    1.013095] !!msi_domain_alloc_irqs: to active!!
[    1.014013] ^^^SEND mapti: hwirq:8194,event:2^^

[    1.057978] !!msi_domain_alloc_irqs: to prepare: nvec:4!!
[    1.072773] ~~its_pci_msi_prepare:devid:8,alias count:5~~
[    1.073943] **its_msi_prepare:devid:8, nves:5**
[    1.074850] **its_msi_prepare:Reusing ITT for devID:8**
[    1.075873] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.077154] ID:3 pID:8195 vID:55
[    1.077813] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.079044] ID:4 pID:8196 vID:56
[    1.079683] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.080947] ID:5 pID:8197 vID:57
[    1.081592] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.082825] ID:6 pID:8198 vID:58

part of Eric's commit:

(quoted diff) +2 -0

@@ -784,6 +788,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
        if (!device)
                return E_ITS_MAPTI_UNMAPPED_DEVICE;
 
+       if (event_id >= BIT_ULL(device->num_eventid_bits))
+               return E_ITS_MAPTI_ID_OOR;

Quoted 23 lines

Thanks!
Quoted 21 lines
Quoted 9 lines
Quoted 9 lines
I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed. Of course, I think this commit must be correct surely.
(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration). This commit brings in that the four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
(Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count), So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
Now, at worst that patch *should* result in the same number of vectors being reserved as before - never fewer. Does anything change with it reverted?
Quoted 1 line
Quoted 1 line
Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.
Note that there are further issues with PCI RID to DevID mappings in the face of aliases[1], but I think the current code does happen to work out OK for the PCI-PCIe bridge case already.
Quoted 1 line
+Robin, who is the author of that patch.

Oops, only now do I actually notice - e8137f4f5088 is the *original* commit trying to do the right thing, but incorrectly. My patch fixing it is actually 3403b0259d15 ("irqchip/gic-v3-its: Fix MSI alias accounting"), and isn't in 4.12, which would explain things here. I expect that cherry-picking 3403b0259d15 should lead to success, since it looks like this is a case of the exact problem it was intended to solve.

Thanks,
Robin.

Quoted 154 lines

Quoted 10 lines

Quoted 8 lines
Regarding (2), the number of MSIs should be the total number of devices that are going to generate the same DevID. Since the bridge is non-transparent, everything behind it aliases with it. So you should probably see all the virtio devices and the bridges themselves being counted. If that's not the case, then "we have a bug"(tm).
Can you please post your full qemu cmd line so that we can reproduce it and investigate the issue?
Yes, that would be good.

I used the qemu version: 2.9.50,
guest linux version: Linux 4.12

xml :

    <type arch='aarch64' machine='virt-2.9'>hvm</type>
    <kernel>/mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image</kernel>
    <cmdline>console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='hpet' present='no'/>
    <timer name='pit' tickpolicy='delay'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/wanghaibin/vm-res/euler_b500.raw'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ehci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:60:6b:1d'/>
      <source bridge='br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='c216af51-9088-4bf8-ba04-cf6172cb3753'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </video>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>

</seclabel>
</domain>

qemu cmdline :

/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=abu,debug-threads=on -S

(quoted diff) +0 ~~-10~~

-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abu/master-key.aes -machine virt-2.9,accel=kvm,usb=off,gic-version=3
-cpu host -m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 76365c65-7ee7-43ff-bb57-f0f80b75323a -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
-rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -kernel /mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image
-append console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3
-drive file=/mnt/wanghaibin/vm-res/euler_b500.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-serial pty -device usb-tablet,id=input0 -device usb-kbd,id=input1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4 -msg timestamp=on

Thanks

Quoted 13 lines

Robin.
[1] https://patchwork.ozlabs.org/patch/769303/
Quoted 4 lines
Thanks,
	M.
.

wanghaibin

author

2017-07-18 11:48 UTC #5 lore raw

Quoted 126 lines · Robin Murphy

Quoted 94 lines · wanghaibin
Quoted 11 lines · Robin Murphy
Quoted 10 lines · Marc Zyngier
Quoted 9 lines · wanghaibin
Hi, all:
I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt sections 2.3)
Here is part of qemu cmd:
(quoted diff) +0 -3
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it. Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge. Boot the vm, it's failed.
What's the nature of the failure? Does it hit some actual error case in the GIC code, or does it simply hang up probing the virtio devices because interrupts never arrive?
Qemu cmdline, xml info, qemu version info, guest kernel version info at the bottom of this mail.
Guest hang log:
[  242.740171] INFO: task kworker/u16:4:446 blocked for more than 120 seconds.
[  242.741102]       Not tainted 4.12.0+ #18
[  242.741619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.742610] kworker/u16:4   D    0   446      2 0x00000000
[  242.743339] Workqueue: scsi_tmf_0 scmd_eh_abort_handler
[  242.744014] Call trace:
[  242.744375] [<ffff0000080856bc>] __switch_to+0x94/0xa8
[  242.745042] [<ffff00000892a3f8>] __schedule+0x1a0/0x5e0
[  242.745716] [<ffff00000892a870>] schedule+0x38/0xa0
[  242.746346] [<ffff00000892da84>] schedule_timeout+0x194/0x2b8
[  242.747092] [<ffff00000892b438>] wait_for_common+0xa0/0x148
[  242.747810] [<ffff00000892b4f4>] wait_for_completion+0x14/0x20
[  242.748595] [<ffff0000085e78d8>] virtscsi_tmf.constprop.15+0x88/0xf0
[  242.749408] [<ffff0000085e79dc>] virtscsi_abort+0x9c/0xb8
[  242.750099] [<ffff0000085db4dc>] scmd_eh_abort_handler+0x5c/0x108
[  242.750887] [<ffff0000080d8594>] process_one_work+0x124/0x2a8
[  242.751618] [<ffff0000080d8774>] worker_thread+0x5c/0x3d8
[  242.752330] [<ffff0000080de7b4>] kthread+0xfc/0x128
[  242.752960] [<ffff000008082ec0>] ret_from_fork+0x10/0x50
But I still doubt the total vector count takes this problem in, I add some log in guest:
In guest boot, pci_dev: 02:04:00(virtio-gpu-pci) first load, and only alloc 4 ITT entries, the log as follow:
[    0.986233] ~~its_pci_msi_prepare:pci dev: 2,32, nvec:3~~
[    0.998952] ~~its_pci_msi_prepare:devid:8,alias count:4~~
[    1.000028] **its_msi_prepare:devid:8, nves:4**
[    1.001001] ##its_create_device:devid: 8, ITT 4 entries, 2 bits, lpi base:8192, nr:32##
[    1.002585] **its_msi_prepare:ITT 4 entries, 2 bits**
[    1.003593] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.004880] ID:0 pID:8192 vID:52
[    1.005529] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.006777] ID:1 pID:8193 vID:53
[    1.007437] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.008718] ID:2 pID:8194 vID:54
[    1.009366] !!msi_domain_alloc_irqs: to active!!
[    1.010281] ^^^SEND mapti: hwirq:8192,event:0^^
[    1.011224] !!msi_domain_alloc_irqs: to active!!
[    1.012161] ^^^SEND mapti: hwirq:8193,event:1^^
[    1.013095] !!msi_domain_alloc_irqs: to active!!
[    1.014013] ^^^SEND mapti: hwirq:8194,event:2^^
and the guest booted continue, when load the pci_dev: 02:03:00 (virtio-scsi), the log shows it shared the same devid with virtio-gpu-pci, shared ite_dev, reusing ITT. So that, the virtio-gpu-pci dev only alloc 4 ITT, and the virtio-scsi send mapti with eventid 5/6, this will be captured by Eric's commit:
guest log:
[    1.057978] !!msi_domain_alloc_irqs: to prepare: nvec:4!!
[    1.072773] ~~its_pci_msi_prepare:devid:8,alias count:5~~
[    1.073943] **its_msi_prepare:devid:8, nves:5**
[    1.074850] **its_msi_prepare:Reusing ITT for devID:8**
[    1.075873] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.077154] ID:3 pID:8195 vID:55
[    1.077813] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.079044] ID:4 pID:8196 vID:56
[    1.079683] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.080947] ID:5 pID:8197 vID:57
[    1.081592] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.082825] ID:6 pID:8198 vID:58
part of Eric's commit:
(quoted diff) +2 -0
@@ -784,6 +788,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
        if (!device)
                return E_ITS_MAPTI_UNMAPPED_DEVICE;
 
+       if (event_id >= BIT_ULL(device->num_eventid_bits))
+               return E_ITS_MAPTI_ID_OOR;
 
Quoted 23 lines
Thanks!
Quoted 21 lines
Quoted 9 lines
Quoted 9 lines
I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed. Of course, I think this commit must be correct surely.
(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration). This commit brings in that the four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
(Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count), So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
Now, at worst that patch *should* result in the same number of vectors being reserved as before - never fewer. Does anything change with it reverted?
Quoted 1 line
Quoted 1 line
Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.
Note that there are further issues with PCI RID to DevID mappings in the face of aliases[1], but I think the current code does happen to work out OK for the PCI-PCIe bridge case already.
Quoted 1 line
+Robin, who is the author of that patch.
Oops, only now do I actually notice - e8137f4f5088 is the *original* commit trying to do the right thing, but incorrectly. My patch fixing it is actually 3403b0259d15 ("irqchip/gic-v3-its: Fix MSI alias accounting"), and isn't in 4.12, which would explain things here. I expect that cherry-picking 3403b0259d15 should lead to success, since it looks like this is a case of the exact problem it was intended to solve.

Quoted 2 lines

Thanks,
Robin.

Already tested, everything is OK.

Thanks very much!

Quoted 159 lines

Quoted 137 lines

Quoted 11 lines

Quoted 9 lines
Regarding (2), the number of MSIs should be the total number of devices that are going to generate the same DevID. Since the bridge is non-transparent, everything behind it aliases with it. So you should probably see all the virtio devices and the bridges themselves being counted. If that's not the case, then "we have a bug"(tm).
Can you please post your full qemu cmd line so that we can reproduce it and investigate the issue?
Yes, that would be good.

I used the qemu version: 2.9.50,
guest linux version: Linux 4.12

xml :

    <type arch='aarch64' machine='virt-2.9'>hvm</type>
    <kernel>/mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image</kernel>
    <cmdline>console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='hpet' present='no'/>
    <timer name='pit' tickpolicy='delay'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/wanghaibin/vm-res/euler_b500.raw'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ehci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:60:6b:1d'/>
      <source bridge='br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='c216af51-9088-4bf8-ba04-cf6172cb3753'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </video>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>

</seclabel>
</domain>

qemu cmdline :

/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=abu,debug-threads=on -S

(quoted diff) +0 ~~-10~~

-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abu/master-key.aes -machine virt-2.9,accel=kvm,usb=off,gic-version=3
-cpu host -m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 76365c65-7ee7-43ff-bb57-f0f80b75323a -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
-rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -kernel /mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image
-append console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3
-drive file=/mnt/wanghaibin/vm-res/euler_b500.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-serial pty -device usb-tablet,id=input0 -device usb-kbd,id=input1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4 -msg timestamp=on

Quoted 18 lines

Thanks
Quoted 13 lines
Robin.
[1] https://patchwork.ozlabs.org/patch/769303/
Quoted 4 lines
Thanks,
	M.
.