New: vcpu1, guest rIP offset ignored wrmsr or rdmsr

Bug 208091 [email protected] 1 participant 12 messages Jun 6, 2020 – Jun 19, 2020 KVM

Share & tools

[email protected]

author

2020-06-06 17:15 UTC #0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=208091

            Bug ID: 208091
           Summary: vcpu1, guest rIP offset ignored wrmsr or rdmsr
           Product: Virtualization
           Version: unspecified
    Kernel Version: Linux pvx 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15
                    May 2020 15:06:08 +0200) x86_64 GNU/Linux
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: low
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

On launch of VM these messages are generated. Filing this report since hinted at a reason for improvement such as for patch 7670771. First ever report, sorry if violating any good habits.

pvx kernel: [ 77.102686] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x3a
pvx kernel: [ 77.102712] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0xd90
pvx kernel: [ 77.102736] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x570
pvx kernel: [ 77.102755] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x571
pvx kernel: [ 77.102773] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x572
pvx kernel: [ 77.102792] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x560
pvx kernel: [ 77.102811] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x561
pvx kernel: [ 77.102829] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x580
pvx kernel: [ 77.102847] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x581
pvx kernel: [ 77.102866] kvm [3464]: vcpu1, guest rIP: 0xffffffff8666b9e4
ignored rdmsr: 0x582

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-06 17:15 UTC #1 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Joris L. ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

           Hardware|All                         |x86-64
           Severity|low                         |normal

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-06 17:16 UTC #2 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Joris L. ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                URL|                            |https://patchwork.kernel.or
                   |                            |g/patch/7670771/

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-06 17:40 UTC #3 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #1 from Joris L. ([email protected]) --- iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99eeb0]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99ef10]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99ef50]
iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=DD:00.0 address=0x7fb99ef70]
vfio-pci 0000:28:00.0: timed out waiting for pending transaction; performing function level reset anyway

device=DD < anonimised

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 15:18 UTC #4 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #2 from Joris L. ([email protected]) --- More of the same

[  507.188509] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[  507.188524] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[  513.661635] kvm [19941]: vcpu0, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.661644] kvm [19941]: vcpu0, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100
[  513.703575] kvm [19941]: vcpu1, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.703588] kvm [19941]: vcpu1, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100
[  513.704769] kvm [19941]: vcpu2, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.704782] kvm [19941]: vcpu2, guest rIP: 0xfffff8000afda0bf ignored wrmsr:
0xc0011023 data 0x100
[  513.705948] kvm [19941]: vcpu3, guest rIP: 0xfffff8000afda0aa ignored rdmsr:
0xc0011023
[  513.705961] kvm [19941]: vcpu3, guest rIP: 0xfffff8000afda0bf ignored wrmsr:

0xc0011023 data 0x100

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 15:50 UTC #5 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Sean Christopherson ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |sean.j.christopherson@intel
                   |                            |.com

--- Comment #3 from Sean Christopherson ([email protected]) --- The "ignored {rd,wr}msr" output is expected when kvm.ignore_msrs=1, which is not the default. Is that enabled for any particular reason? Or maybe a better question is, what is it that you're trying to do and what, if any, errors are occurring?

E.g. in the first trace, it looks like you're advertising an Intel vCPU to the guest on top of AMD hardware. That can be made to work, but it's not recommended. Ignoring MSRs in that scenario is almost guaranteed to cause weirdness.

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 15:55 UTC #6 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #4 from Joris L. ([email protected]) --- These message are observed when starting a Proxmox VE 6.2 VM guest with GPU passthrough. Current configuration is at the point there are not other error messages visible related to VM activity.

Available to respond to and provide for actions to improve documenting these messages etc.

The vcpu0, guest rIP messages persist while a vm is running. Stopping and staring a VM results in no screen. messages however now show (repeat kvm ... msr messages, repeeat block of kvm, vcpu... rIP messages) followed by ecap messages.

[ 2762.744734] kvm_get_msr_common: 275 callbacks suppressed
[ 2762.744738] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010064
[ 2762.744775] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010065
[ 2762.744853] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010066
[ 2762.744885] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010067
[ 2762.744917] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010068
[ 2762.744948] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010069
[ 2762.744983] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc001006a
[ 2762.745014] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc001006b
[ 2762.745044] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xc0010293
[ 2762.750132] kvm [19941]: vcpu2, guest rIP: 0xfffff8001123174b ignored rdmsr:
0xe8
[ 2844.034627] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
[ 2844.034641] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
[ 2845.308346] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.324311] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.566461] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.569777] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.599693] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.602566] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.625243] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.627436] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.659282] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.665352] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.673673] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.673924] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.673948] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.674104] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.674870] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.675040] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.675063] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.675210] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.708785] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.710674] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.727404] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.727598] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.728296] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.728484] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.764636] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.765256] vfio-pci 0000:17:00.1: vfio_bar_restore: reset recovery -
restoring BARs
[ 2845.933638] vfio-pci 0000:17:00.0: vfio_bar_restore: reset recovery -

restoring BARs

now the VM no longer displays the desktop.

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 16:06 UTC #7 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #5 from Joris L. ([email protected]) --- (In reply to Sean Christopherson from comment #3)

Quoted 9 lines

The "ignored {rd,wr}msr" output is expected when kvm.ignore_msrs=1, which is not the default. Is that enabled for any particular reason? Or maybe a better question is, what is it that you're trying to do and what, if any, errors are occurring?
E.g. in the first trace, it looks like you're advertising an Intel vCPU to the guest on top of AMD hardware. That can be made to work, but it's not recommended. Ignoring MSRs in that scenario is almost guaranteed to cause weirdness.

Thanks Sean. I am cleaning up this config. Will check.

This is indeed a Ryzen CPU with Qemu/Proxmox on top of it.
For GPU Passthrough to a VM the kvm.ignore_msrs=1 is often recommended. I brief test shows the VM is not working properly without this parameter, even when booting the VM with CPU:host.

br,

Joris

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 16:15 UTC #8 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #6 from Joris L. ([email protected]) --- These i get with kvm ignore.msrs=1 enabled and CPU=host

[   46.512153] kvm [7509]: vcpu0, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.512161] kvm [7509]: vcpu0, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100
[   46.555401] kvm [7509]: vcpu1, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.555406] kvm [7509]: vcpu1, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100
[   46.556809] kvm [7509]: vcpu2, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.556823] kvm [7509]: vcpu2, guest rIP: 0xfffff8039588c0bf ignored wrmsr:
0xc0011023 data 0x100
[   46.558065] kvm [7509]: vcpu3, guest rIP: 0xfffff8039588c0aa ignored rdmsr:
0xc0011023
[   46.558078] kvm [7509]: vcpu3, guest rIP: 0xfffff8039588c0bf ignored wrmsr:

0xc0011023 data 0x100

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 19:15 UTC #9 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #7 from Joris L. ([email protected]) --- With the kvm ignor.msrs parameter removed i do indeed see different results. Now i am back at these messages. Sorry for the ignorance, learning as i read the feedback.

[10427.220982] vfio-pci 0000:17:00.0: timed out waiting for pending
transaction; performing function level reset anyway
[10428.469006] vfio-pci 0000:17:00.0: not ready 1023ms after FLR; waiting
[10429.524940] vfio-pci 0000:17:00.0: not ready 2047ms after FLR; waiting
[10431.668982] vfio-pci 0000:17:00.0: not ready 4095ms after FLR; waiting
[10436.020672] vfio-pci 0000:17:00.0: not ready 8191ms after FLR; waiting
[10444.469003] vfio-pci 0000:17:00.0: not ready 16383ms after FLR; waiting
[10462.133026] vfio-pci 0000:17:00.0: not ready 32767ms after FLR; waiting
[10496.948661] vfio-pci 0000:17:00.0: not ready 65535ms after FLR; giving up

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-08 19:23 UTC #10 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

--- Comment #8 from Joris L. ([email protected]) --- followed by

[10956.384880] vfio-pci 0000:17:00.0: Refused to change power state, currently
in D3
[10957.109026] vfio-pci 0000:17:00.0: timed out waiting for pending
transaction; performing function level reset anyway
[10958.356694] vfio-pci 0000:17:00.0: not ready 1023ms after FLR; waiting
[10959.412727] vfio-pci 0000:17:00.0: not ready 2047ms after FLR; waiting
[10961.588924] vfio-pci 0000:17:00.0: not ready 4095ms after FLR; waiting
[10965.940829] vfio-pci 0000:17:00.0: not ready 8191ms after FLR; waiting

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.

[email protected]

author

2020-06-19 22:24 UTC #11 lore raw

[Bug 208091] vcpu1, guest rIP offset ignored wrmsr or rdmsr

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=208091

Joris L. ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |DOCUMENTED

--- Comment #9 from Joris L. ([email protected]) --- eventually, boot time parameters resolved all of these issues, sharing in case they matter.

iommu=pt amd_iommu=forced_isolation video=vesafb:off,efifb:off rcu_nocbs=0-15 amdgpu.runpm=0 pci=nobar,noats,big_root_window pcie_acs_override=downstream,multifunction pcie_pme=nomsi pcie_aspm=force pcie_ports=native

Signature · 2 lines

You are receiving this mail because:
You are watching the assignee of the bug.