Graphical glitches (not refreshing?) with Linux's xe driver + Xen 4.19
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
The result is rather sad:
https://openqa.qubes-os.org/tests/166691#step/simple_gui_apps/30
or
https://openqa.qubes-os.org/tests/166700#step/desktop_linux_manager_config/42
The black windows are mapped from a domU (using grant tables). But if you watch the video (in the "logs&assets" tab), login window of dom0 is affected too (background is rendered significantly later than its text and widgets).
I didn't found any relevant errors in kernel (or Xen) messages. Just in case, full dumps:
https://openqa.qubes-os.org/tests/166700/file/desktop_linux_manager_config-journalctl.log
https://openqa.qubes-os.org/tests/166700/file/desktop_linux_manager_config-xl-dmesg.log
Some user reported similar issue with Intel Arc Pro B50, which also uses "xe" driver:
https://github.com/QubesOS/qubes-issues/issues/10661
I tested also Linux 6.19-rc8, no improvements.
My previous tests of the xe driver on older systems (Meteor Lake, Raptor Lake) resulted in similar behavior, while i915 works fine there. But there I needed to use force_probe parameter, and blamed the issue on xe not officially supporting those platforms. I wanted to compare with i915 on LNL, but i915 doesn't support that model at all (and if I force it, I get kernel panic in i915_pci_probe...).
Interestingly, it appears to work just fine on Qubes OS 4.2 (Xen 4.17.6, same Linux dom0 version). I did not tested with Xen 4.21.
In both cases, our Xen has the patch to use Linux's PAT layout applied. That's a workaround for
https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7648, which is fixed now, but apparently similar bug is still present in a few other drivers (including binary nvidia one...).
The same Linux kernel started without Xen works just fine.
Any ideas?
Having good and a bad version, I can try to bisect this one, but given it requires rebuilding Xen toolstack each time too (due to bisection across releases), I'd rather avoid it if possible...
I'm CC-ing also intel-xe list, maybe somebody there have some ideas how to debug this.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 45 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
The result is rather sad:
https://openqa.qubes-os.org/tests/166691#step/simple_gui_apps/30
or
https://openqa.qubes-os.org/tests/166700#step/desktop_linux_manager_config/42The black windows are mapped from a domU (using grant tables). But if you watch the video (in the "logs&assets" tab), login window of dom0 is affected too (background is rendered significantly later than its text and widgets).
I didn't found any relevant errors in kernel (or Xen) messages. Just in case, full dumps:
https://openqa.qubes-os.org/tests/166700/file/desktop_linux_manager_config-journalctl.log
https://openqa.qubes-os.org/tests/166700/file/desktop_linux_manager_config-xl-dmesg.logSome user reported similar issue with Intel Arc Pro B50, which also uses "xe" driver:
https://github.com/QubesOS/qubes-issues/issues/10661I tested also Linux 6.19-rc8, no improvements.
My previous tests of the xe driver on older systems (Meteor Lake, Raptor Lake) resulted in similar behavior, while i915 works fine there. But there I needed to use
force_probeparameter, and blamed the issue on xe not officially supporting those platforms. I wanted to compare with i915 on LNL, but i915 doesn't support that model at all (and if I force it, I get kernel panic ini915_pci_probe...).Interestingly, it appears to work just fine on Qubes OS 4.2 (Xen 4.17.6, same Linux dom0 version). I did not tested with Xen 4.21.
In both cases, our Xen has the patch to use Linux's PAT layout applied. That's a workaround for
https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/7648, which is fixed now, but apparently similar bug is still present in a few other drivers (including binary nvidia one...).The same Linux kernel started without Xen works just fine.
Quoted 8 lines
Any ideas?
Having good and a bad version, I can try to bisect this one, but given it requires rebuilding Xen toolstack each time too (due to bisection across releases), I'd rather avoid it if possible...
I'm CC-ing also intel-xe list, maybe somebody there have some ideas how to debug this.
Unfortunately I'm afraid we don't have any good advice here.
First I thought about trying to isolate Mesa version in the Qubes OS version, but the fact that it works in bare metal makes this a silly option.
It looks more a cache management indeed, and I'm afraid we will only find the definite spot after the bisect :(
Quoted 5 lines
Signature · 4 lines
-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Thanks, Roger.
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Yes, it makes a difference. I end up with black screen as soon as dom0 starts... And due to unrelated bug (in xhci console code?), I don't have kernel messages :/
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 9 lines · Marek Marczykowski-Górecki
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Quoted 6 lines
Quoted 2 lines
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Yes, it makes a difference. I end up with black screen as soon as dom0 starts... And due to unrelated bug (in xhci console code?), I don't have kernel messages :/
Hm, that's ... very helpful. We should fix the xhci stuff so we know what's going on there. Is the xhci bug also PVH only, or PV is equally affected?
Thanks, Roger.
Quoted 13 lines · Roger Pau Monné
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Quoted 3 lines
Yes, it makes a difference. I end up with black screen as soon as dom0 starts... And due to unrelated bug (in xhci console code?), I don't have kernel messages :/
Hm, that's ... very helpful. We should fix the xhci stuff so we know what's going on there. Is the xhci bug also PVH only, or PV is equally affected?
XHCI console works just fine with PV dom0. But I just noticed I had older Xen binary there today, will re-test on the same 4.19.4 again. And then will submit separate bug report (even if that's likely be me fixing it anyway).
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 18 lines · Marek Marczykowski-Górecki
Quoted 17 lines · Roger Pau Monné
Quoted 15 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Yes, it makes a difference. I end up with black screen as soon as dom0 starts... And due to unrelated bug (in xhci console code?), I don't have kernel messages :/
Quoted 8 lines
Quoted 3 lines
Hm, that's ... very helpful. We should fix the xhci stuff so we know what's going on there. Is the xhci bug also PVH only, or PV is equally affected?
XHCI console works just fine with PV dom0. But I just noticed I had older Xen binary there today, will re-test on the same 4.19.4 again. And then will submit separate bug report (even if that's likely be me fixing it anyway).
OK, could you also post whatever output you get when using a PVH dom0 with sync_console added to the Xen command line?
Thanks, Roger.
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
This reminds me of a problem that I helped debug a while back, and which also affected Qubes OS. That one was due to some part of Linux core MM clobbering the high bit of the PAT part of PTEs. As a result, a PTE that should have referenced entry X in the PAT wound up referencing entry (X & 3).
On native Linux, the bug was mostly harmless, which is why nobody found it earlier. However, on Xen PV, the different PAT caused memory to be cached that should not be. The resulting symptoms were very similar to what Marek described, though a bit less dramatic.
Can you try removing the Qubes downstream patch that changes the PAT?
Signature · 2 lines
Sincerely, Demi Marie Obenour (she/her/hers)
Quoted 27 lines · Demi Marie Obenour
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
This reminds me of a problem that I helped debug a while back, and which also affected Qubes OS. That one was due to some part of Linux core MM clobbering the high bit of the PAT part of PTEs. As a result, a PTE that should have referenced entry X in the PAT wound up referencing entry (X & 3).
Quoted 6 lines
On native Linux, the bug was mostly harmless, which is why nobody found it earlier. However, on Xen PV, the different PAT caused memory to be cached that should not be. The resulting symptoms were very similar to what Marek described, though a bit less dramatic.
Can you try removing the Qubes downstream patch that changes the PAT?
I'd assume removing such patch would make things only worse (it either shouldn't matter, or if the driver assumes Linux version, changing it would actually break stuff).
Anyway, removing the patch didn't change anything.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
Some further observations:
1. My initial impression that Xen 4.17.6 is not affected is false. Apparently I got lucky and didn't waited long enough for glitches to appear. Unfortunately this means I have no way to bisect this...
1a. Updated test procedure - either:
- start Qubes OS in full (including default system domUs) and try to open an app in one of them (for example file manager or pdf viewer)
- start Linux up to lightdm login page, log in, log out, click on a few lightdm menus (session type selector, poewroff menu etc)
The second version works even if toolstack version in dom0 doesn't match Xen version. If no glitches are observed after doing either of those procedures, assume it's good.
2. Xen staging is affected too. As well as Xen staging-4.19 without any qubes patches.
3. After enabling CONFIG_DEBUG in Xen, the xe.ko fails to load firmware:
xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0
xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status = 0x40000056, time = 0ms, freq = 1850MHz (req 1850MHz), done = -1
xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01
xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: firmware production part check failure
xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-EPROTO)
xe 0000:00:02.0: probe with driver xe failed with error -71CONFIG_DEBUG is the only change between "xe.ko loads fine but there are glitches later on" and "xe.ko fails to load at all". Full console logs:
https://gist.github.com/marmarek/47b5e62a2cdbae6678c2aecc5283cd3f, there are 3 files:
- CONFIG_DEBUG=n
- CONFIG_DEBUG=y
- CONFIG_DEBUG=y + iommu=debug
4. Updating to Linux 7.0-rc1 doesn't help, for example:
https://openqa.qubes-os.org/tests/168119#step/desktop_linux_manager_create_qube/11
Generally, it does feel like a bug in xe.ko, but I can't exclude some issue on Xen side too (especially given point 3 above).
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 57 lines · Marek Marczykowski-Górecki
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
Some further observations:
1. My initial impression that Xen 4.17.6 is not affected is false. Apparently I got lucky and didn't waited long enough for glitches to appear. Unfortunately this means I have no way to bisect this...
1a. Updated test procedure - either:
- start Qubes OS in full (including default system domUs) and try to open an app in one of them (for example file manager or pdf viewer)
- start Linux up to lightdm login page, log in, log out, click on a few lightdm menus (session type selector, poewroff menu etc)The second version works even if toolstack version in dom0 doesn't match Xen version. If no glitches are observed after doing either of those procedures, assume it's good.
2. Xen staging is affected too. As well as Xen staging-4.19 without any qubes patches.
3. After enabling
CONFIG_DEBUGin Xen, the xe.ko fails to load firmware:xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status = 0x40000056, time = 0ms, freq = 1850MHz (req 1850MHz), done = -1 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: firmware production part check failure xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-EPROTO) xe 0000:00:02.0: probe with driver xe failed with error -71
CONFIG_DEBUGis the only change between "xe.ko loads fine but there are glitches later on" and "xe.ko fails to load at all". Full console logs:
https://gist.github.com/marmarek/47b5e62a2cdbae6678c2aecc5283cd3f, there are 3 files:
-CONFIG_DEBUG=n
-CONFIG_DEBUG=y
-CONFIG_DEBUG=y+ iommu=debug
Quoted 5 lines
4. Updating to Linux 7.0-rc1 doesn't help, for example:
https://openqa.qubes-os.org/tests/168119#step/desktop_linux_manager_create_qube/11Generally, it does feel like a bug in xe.ko, but I can't exclude some issue on Xen side too (especially given point 3 above).
After waiting some time (Linux 6.19.5 this time), Xen CONFIG_DEBUG=n, I get some timeout messages:
[ 8.122120] xe 0000:00:02.0: [drm] [ENCODER:204:DDI A/PHY A] failed to retrieve link info, disabling eDP
[ 8.148476] xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0
[ 8.803845] xe 0000:00:02.0: [drm] Tile0: GT0: ccs1 fused off
[ 8.804208] xe 0000:00:02.0: [drm] Tile0: GT0: ccs2 fused off
[ 8.804556] xe 0000:00:02.0: [drm] Tile0: GT0: ccs3 fused off
[ 8.822426] xe 0000:00:02.0: [drm] Tile0: GT1: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0
[ 8.827140] xe 0000:00:02.0: [drm] Tile0: GT1: Using HuC firmware from xe/lnl_huc.bin version 9.4.13
[ 8.829478] xe 0000:00:02.0: [drm] Tile0: GT1: Using GSC firmware from xe/lnl_gsc_1.bin version 104.0.5.1429
[ 8.852923] xe 0000:00:02.0: [drm] Tile0: GT1: vcs1 fused off
[ 8.853513] xe 0000:00:02.0: [drm] Tile0: GT1: vcs2 fused off
[ 8.854090] xe 0000:00:02.0: [drm] Tile0: GT1: vcs3 fused off
[ 8.854706] xe 0000:00:02.0: [drm] Tile0: GT1: vcs4 fused off
[ 8.855310] xe 0000:00:02.0: [drm] Tile0: GT1: vcs5 fused off
[ 8.855904] xe 0000:00:02.0: [drm] Tile0: GT1: vcs6 fused off
[ 8.856495] xe 0000:00:02.0: [drm] Tile0: GT1: vcs7 fused off
[ 8.857079] xe 0000:00:02.0: [drm] Tile0: GT1: vecs1 fused off
[ 8.857675] xe 0000:00:02.0: [drm] Tile0: GT1: vecs2 fused off
[ 8.858272] xe 0000:00:02.0: [drm] Tile0: GT1: vecs3 fused off
[ 8.975881] xe 0000:00:02.0: [drm] Registered 3 planes with drm panic
[ 8.976586] [drm] Initialized xe 1.1.0 for 0000:00:02.0 on minor 0
[ 8.980882] ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no)
[ 9.033754] xe 0000:00:02.0: [drm] Tile0: GT1: found GSC cv104.1.0
...
[ 1218.319232] xe 0000:00:02.0: [drm] Tile0: GT0: Engine reset: engine_class=rcs, logical_mask: 0x1, guc_id=3
[ 1218.319890] xe 0000:00:02.0: [drm] Tile0: GT0: Timedout job: seqno=9883, lrc_seqno=9883, guc_id=3, flags=0x0 in Xorg [3245]
[ 1218.320736] xe 0000:00:02.0: [drm] Xe device coredump has been created
[ 1218.321140] xe 0000:00:02.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data
[ 1222.285626] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out
[ 1232.525685] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out
[ 1232.526280] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out
[ 1242.765717] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out
[ 1253.005696] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out
[ 1253.006248] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out
[ 1263.245599] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed outThe glitches appear much earlier, though.
Would content of /sys/class/drm/card0/device/devcoredump/data be useful for debugging this?
Full log at https://openqa.qubes-os.org/tests/168813/file/serial0.txt
(warning, almost 200MB of those errors...)
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 101 lines · Marek Marczykowski-Górecki
Quoted 57 lines · Marek Marczykowski-Górecki
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
Some further observations:
1. My initial impression that Xen 4.17.6 is not affected is false. Apparently I got lucky and didn't waited long enough for glitches to appear. Unfortunately this means I have no way to bisect this...
1a. Updated test procedure - either:
- start Qubes OS in full (including default system domUs) and try to open an app in one of them (for example file manager or pdf viewer)
- start Linux up to lightdm login page, log in, log out, click on a few lightdm menus (session type selector, poewroff menu etc)The second version works even if toolstack version in dom0 doesn't match Xen version. If no glitches are observed after doing either of those procedures, assume it's good.
2. Xen staging is affected too. As well as Xen staging-4.19 without any qubes patches.
3. After enabling
CONFIG_DEBUGin Xen, the xe.ko fails to load firmware:xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status = 0x40000056, time = 0ms, freq = 1850MHz (req 1850MHz), done = -1 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: firmware production part check failure xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-EPROTO) xe 0000:00:02.0: probe with driver xe failed with error -71
CONFIG_DEBUGis the only change between "xe.ko loads fine but there are glitches later on" and "xe.ko fails to load at all". Full console logs:
https://gist.github.com/marmarek/47b5e62a2cdbae6678c2aecc5283cd3f, there are 3 files:
-CONFIG_DEBUG=n
-CONFIG_DEBUG=y
-CONFIG_DEBUG=y+ iommu=debugQuoted 5 lines
4. Updating to Linux 7.0-rc1 doesn't help, for example:
https://openqa.qubes-os.org/tests/168119#step/desktop_linux_manager_create_qube/11Generally, it does feel like a bug in xe.ko, but I can't exclude some issue on Xen side too (especially given point 3 above).
After waiting some time (Linux 6.19.5 this time), Xen
CONFIG_DEBUG=n, I get some timeout messages:[ 8.122120] xe 0000:00:02.0: [drm] [ENCODER:204:DDI A/PHY A] failed to retrieve link info, disabling eDP [ 8.148476] xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 [ 8.803845] xe 0000:00:02.0: [drm] Tile0: GT0: ccs1 fused off [ 8.804208] xe 0000:00:02.0: [drm] Tile0: GT0: ccs2 fused off [ 8.804556] xe 0000:00:02.0: [drm] Tile0: GT0: ccs3 fused off [ 8.822426] xe 0000:00:02.0: [drm] Tile0: GT1: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 [ 8.827140] xe 0000:00:02.0: [drm] Tile0: GT1: Using HuC firmware from xe/lnl_huc.bin version 9.4.13 [ 8.829478] xe 0000:00:02.0: [drm] Tile0: GT1: Using GSC firmware from xe/lnl_gsc_1.bin version 104.0.5.1429 [ 8.852923] xe 0000:00:02.0: [drm] Tile0: GT1: vcs1 fused off [ 8.853513] xe 0000:00:02.0: [drm] Tile0: GT1: vcs2 fused off [ 8.854090] xe 0000:00:02.0: [drm] Tile0: GT1: vcs3 fused off [ 8.854706] xe 0000:00:02.0: [drm] Tile0: GT1: vcs4 fused off [ 8.855310] xe 0000:00:02.0: [drm] Tile0: GT1: vcs5 fused off [ 8.855904] xe 0000:00:02.0: [drm] Tile0: GT1: vcs6 fused off [ 8.856495] xe 0000:00:02.0: [drm] Tile0: GT1: vcs7 fused off [ 8.857079] xe 0000:00:02.0: [drm] Tile0: GT1: vecs1 fused off [ 8.857675] xe 0000:00:02.0: [drm] Tile0: GT1: vecs2 fused off [ 8.858272] xe 0000:00:02.0: [drm] Tile0: GT1: vecs3 fused off [ 8.975881] xe 0000:00:02.0: [drm] Registered 3 planes with drm panic [ 8.976586] [drm] Initialized xe 1.1.0 for 0000:00:02.0 on minor 0 [ 8.980882] ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 9.033754] xe 0000:00:02.0: [drm] Tile0: GT1: found GSC cv104.1.0 ... [ 1218.319232] xe 0000:00:02.0: [drm] Tile0: GT0: Engine reset: engine_class=rcs, logical_mask: 0x1, guc_id=3 [ 1218.319890] xe 0000:00:02.0: [drm] Tile0: GT0: Timedout job: seqno=9883, lrc_seqno=9883, guc_id=3, flags=0x0 in Xorg [3245] [ 1218.320736] xe 0000:00:02.0: [drm] Xe device coredump has been created [ 1218.321140] xe 0000:00:02.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data [ 1222.285626] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out [ 1232.525685] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out [ 1232.526280] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out [ 1242.765717] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out [ 1253.005696] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out [ 1253.006248] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out [ 1263.245599] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out
Quoted 6 lines
The glitches appear much earlier, though.
Would content of/sys/class/drm/card0/device/devcoredump/databe useful for debugging this?Full log at https://openqa.qubes-os.org/tests/168813/file/serial0.txt
(warning, almost 200MB of those errors...)
The issue still happens with Linux 7.0.12. Current log (quite similar to the previous one):
https://openqa.qubes-os.org/tests/184602/logfile?filename=serial0.txt
Not long after GPU errors, nvme driver fails due to full swiotlb.
Any ideas?
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 102 lines · Marek Marczykowski-Górecki
Quoted 101 lines · Marek Marczykowski-Górecki
Quoted 57 lines · Marek Marczykowski-Górecki
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
Some further observations:
1. My initial impression that Xen 4.17.6 is not affected is false. Apparently I got lucky and didn't waited long enough for glitches to appear. Unfortunately this means I have no way to bisect this...
1a. Updated test procedure - either:
- start Qubes OS in full (including default system domUs) and try to open an app in one of them (for example file manager or pdf viewer)
- start Linux up to lightdm login page, log in, log out, click on a few lightdm menus (session type selector, poewroff menu etc)The second version works even if toolstack version in dom0 doesn't match Xen version. If no glitches are observed after doing either of those procedures, assume it's good.
2. Xen staging is affected too. As well as Xen staging-4.19 without any qubes patches.
3. After enabling
CONFIG_DEBUGin Xen, the xe.ko fails to load firmware:xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status = 0x40000056, time = 0ms, freq = 1850MHz (req 1850MHz), done = -1 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: firmware production part check failure xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-EPROTO) xe 0000:00:02.0: probe with driver xe failed with error -71
CONFIG_DEBUGis the only change between "xe.ko loads fine but there are glitches later on" and "xe.ko fails to load at all". Full console logs:
https://gist.github.com/marmarek/47b5e62a2cdbae6678c2aecc5283cd3f, there are 3 files:
-CONFIG_DEBUG=n
-CONFIG_DEBUG=y
-CONFIG_DEBUG=y+ iommu=debugQuoted 5 lines
4. Updating to Linux 7.0-rc1 doesn't help, for example:
https://openqa.qubes-os.org/tests/168119#step/desktop_linux_manager_create_qube/11Generally, it does feel like a bug in xe.ko, but I can't exclude some issue on Xen side too (especially given point 3 above).
After waiting some time (Linux 6.19.5 this time), Xen
CONFIG_DEBUG=n, I get some timeout messages:[ 8.122120] xe 0000:00:02.0: [drm] [ENCODER:204:DDI A/PHY A] failed to retrieve link info, disabling eDP [ 8.148476] xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 [ 8.803845] xe 0000:00:02.0: [drm] Tile0: GT0: ccs1 fused off [ 8.804208] xe 0000:00:02.0: [drm] Tile0: GT0: ccs2 fused off [ 8.804556] xe 0000:00:02.0: [drm] Tile0: GT0: ccs3 fused off [ 8.822426] xe 0000:00:02.0: [drm] Tile0: GT1: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 [ 8.827140] xe 0000:00:02.0: [drm] Tile0: GT1: Using HuC firmware from xe/lnl_huc.bin version 9.4.13 [ 8.829478] xe 0000:00:02.0: [drm] Tile0: GT1: Using GSC firmware from xe/lnl_gsc_1.bin version 104.0.5.1429 [ 8.852923] xe 0000:00:02.0: [drm] Tile0: GT1: vcs1 fused off [ 8.853513] xe 0000:00:02.0: [drm] Tile0: GT1: vcs2 fused off [ 8.854090] xe 0000:00:02.0: [drm] Tile0: GT1: vcs3 fused off [ 8.854706] xe 0000:00:02.0: [drm] Tile0: GT1: vcs4 fused off [ 8.855310] xe 0000:00:02.0: [drm] Tile0: GT1: vcs5 fused off [ 8.855904] xe 0000:00:02.0: [drm] Tile0: GT1: vcs6 fused off [ 8.856495] xe 0000:00:02.0: [drm] Tile0: GT1: vcs7 fused off [ 8.857079] xe 0000:00:02.0: [drm] Tile0: GT1: vecs1 fused off [ 8.857675] xe 0000:00:02.0: [drm] Tile0: GT1: vecs2 fused off [ 8.858272] xe 0000:00:02.0: [drm] Tile0: GT1: vecs3 fused off [ 8.975881] xe 0000:00:02.0: [drm] Registered 3 planes with drm panic [ 8.976586] [drm] Initialized xe 1.1.0 for 0000:00:02.0 on minor 0 [ 8.980882] ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 9.033754] xe 0000:00:02.0: [drm] Tile0: GT1: found GSC cv104.1.0 ... [ 1218.319232] xe 0000:00:02.0: [drm] Tile0: GT0: Engine reset: engine_class=rcs, logical_mask: 0x1, guc_id=3 [ 1218.319890] xe 0000:00:02.0: [drm] Tile0: GT0: Timedout job: seqno=9883, lrc_seqno=9883, guc_id=3, flags=0x0 in Xorg [3245] [ 1218.320736] xe 0000:00:02.0: [drm] Xe device coredump has been created [ 1218.321140] xe 0000:00:02.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data [ 1222.285626] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out [ 1232.525685] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out [ 1232.526280] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out [ 1242.765717] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out [ 1253.005696] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out [ 1253.006248] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out [ 1263.245599] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out
Quoted 3 lines
Quoted 3 lines
The glitches appear much earlier, though.
Would content of/sys/class/drm/card0/device/devcoredump/databe useful for debugging this?
Yes, it would. Jobs hanging can be a bug anywhere in the stack (e.g., Hardware bug, KMD bug, UMD bug, application bug, etc...) but the devcoredump would give us some hints.
Quoted 7 lines
Quoted 3 lines
Full log at https://openqa.qubes-os.org/tests/168813/file/serial0.txt
(warning, almost 200MB of those errors...)The issue still happens with Linux 7.0.12. Current log (quite similar to the previous one):
https://openqa.qubes-os.org/tests/184602/logfile?filename=serial0.txt
Hmm, the 'not started' messages in the dmesg are a bit concerning as this really shouldn't be possible to trigger even if user space is doing something wrong.
Can you file a gitlab issue against Xe here: https://gitlab.freedesktop.org/drm/xe/kernel/issues/new
TBH, I have no idea if running Xen / Qubes OS + Xe is something anyone at Intel has tried out, so please include instructions on to how reproduce and we will see in someone on engineering team can take a look at this and if issues in Xe KMD exist, try to get these fixed.
Matt
Quoted 9 lines
Not long after GPU errors, nvme driver fails due to full swiotlb.
Any ideas?
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 123 lines · Matthew Brost
Quoted 102 lines · Marek Marczykowski-Górecki
Quoted 101 lines · Marek Marczykowski-Górecki
Quoted 57 lines · Marek Marczykowski-Górecki
Quoted 12 lines · Marek Marczykowski-Górecki
Quoted 10 lines · Roger Pau Monné
Quoted 6 lines · Marek Marczykowski-Górecki
Hi,
Recently I started testing compatibility with Intel Lunar Lake. This is the first one that uses "xe" instead of "i915" Linux driver for iGPU. I test it with Qubes OS 4.3, which uses Xen 4.19.4 and PV dom0 running Linux 6.17.9 in this test.
Not sure it's going to help a lot, but does using a PVH dom0 make any difference?
Quoted 7 lines
Ok, now with the correct Xen version, it's better with PVH dom0. At least on the login screen and few applications (from both dom0 and domU) I don't see the glitches anymore. I can't do a full test, because PCI passthrough doesn't seem to work with PVH dom0 on Xen 4.19 - and I need it to start most VMs.
So, if the above test is representative, it's only about PV dom0.
Some further observations:
1. My initial impression that Xen 4.17.6 is not affected is false. Apparently I got lucky and didn't waited long enough for glitches to appear. Unfortunately this means I have no way to bisect this...
1a. Updated test procedure - either:
- start Qubes OS in full (including default system domUs) and try to open an app in one of them (for example file manager or pdf viewer)
- start Linux up to lightdm login page, log in, log out, click on a few lightdm menus (session type selector, poewroff menu etc)The second version works even if toolstack version in dom0 doesn't match Xen version. If no glitches are observed after doing either of those procedures, assume it's good.
2. Xen staging is affected too. As well as Xen staging-4.19 without any qubes patches.
3. After enabling
CONFIG_DEBUGin Xen, the xe.ko fails to load firmware:xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status = 0x40000056, time = 0ms, freq = 1850MHz (req 1850MHz), done = -1 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: load failed: status: Reset = 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01 xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: firmware production part check failure xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-EPROTO) xe 0000:00:02.0: probe with driver xe failed with error -71
CONFIG_DEBUGis the only change between "xe.ko loads fine but there are glitches later on" and "xe.ko fails to load at all". Full console logs:
https://gist.github.com/marmarek/47b5e62a2cdbae6678c2aecc5283cd3f, there are 3 files:
-CONFIG_DEBUG=n
-CONFIG_DEBUG=y
-CONFIG_DEBUG=y+ iommu=debugQuoted 5 lines
4. Updating to Linux 7.0-rc1 doesn't help, for example:
https://openqa.qubes-os.org/tests/168119#step/desktop_linux_manager_create_qube/11Generally, it does feel like a bug in xe.ko, but I can't exclude some issue on Xen side too (especially given point 3 above).
After waiting some time (Linux 6.19.5 this time), Xen
CONFIG_DEBUG=n, I get some timeout messages:[ 8.122120] xe 0000:00:02.0: [drm] [ENCODER:204:DDI A/PHY A] failed to retrieve link info, disabling eDP [ 8.148476] xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 [ 8.803845] xe 0000:00:02.0: [drm] Tile0: GT0: ccs1 fused off [ 8.804208] xe 0000:00:02.0: [drm] Tile0: GT0: ccs2 fused off [ 8.804556] xe 0000:00:02.0: [drm] Tile0: GT0: ccs3 fused off [ 8.822426] xe 0000:00:02.0: [drm] Tile0: GT1: Using GuC firmware from xe/lnl_guc_70.bin version 70.53.0 [ 8.827140] xe 0000:00:02.0: [drm] Tile0: GT1: Using HuC firmware from xe/lnl_huc.bin version 9.4.13 [ 8.829478] xe 0000:00:02.0: [drm] Tile0: GT1: Using GSC firmware from xe/lnl_gsc_1.bin version 104.0.5.1429 [ 8.852923] xe 0000:00:02.0: [drm] Tile0: GT1: vcs1 fused off [ 8.853513] xe 0000:00:02.0: [drm] Tile0: GT1: vcs2 fused off [ 8.854090] xe 0000:00:02.0: [drm] Tile0: GT1: vcs3 fused off [ 8.854706] xe 0000:00:02.0: [drm] Tile0: GT1: vcs4 fused off [ 8.855310] xe 0000:00:02.0: [drm] Tile0: GT1: vcs5 fused off [ 8.855904] xe 0000:00:02.0: [drm] Tile0: GT1: vcs6 fused off [ 8.856495] xe 0000:00:02.0: [drm] Tile0: GT1: vcs7 fused off [ 8.857079] xe 0000:00:02.0: [drm] Tile0: GT1: vecs1 fused off [ 8.857675] xe 0000:00:02.0: [drm] Tile0: GT1: vecs2 fused off [ 8.858272] xe 0000:00:02.0: [drm] Tile0: GT1: vecs3 fused off [ 8.975881] xe 0000:00:02.0: [drm] Registered 3 planes with drm panic [ 8.976586] [drm] Initialized xe 1.1.0 for 0000:00:02.0 on minor 0 [ 8.980882] ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 9.033754] xe 0000:00:02.0: [drm] Tile0: GT1: found GSC cv104.1.0 ... [ 1218.319232] xe 0000:00:02.0: [drm] Tile0: GT0: Engine reset: engine_class=rcs, logical_mask: 0x1, guc_id=3 [ 1218.319890] xe 0000:00:02.0: [drm] Tile0: GT0: Timedout job: seqno=9883, lrc_seqno=9883, guc_id=3, flags=0x0 in Xorg [3245] [ 1218.320736] xe 0000:00:02.0: [drm] Xe device coredump has been created [ 1218.321140] xe 0000:00:02.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data [ 1222.285626] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out [ 1232.525685] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out [ 1232.526280] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out [ 1242.765717] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed out [ 1253.005696] xe 0000:00:02.0: [drm] *ERROR* flip_done timed out [ 1253.006248] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] commit wait timed out [ 1263.245599] xe 0000:00:02.0: [drm] *ERROR* [CRTC:88:pipe A] flip_done timed outQuoted 3 lines
Quoted 3 lines
The glitches appear much earlier, though.
Would content of/sys/class/drm/card0/device/devcoredump/databe useful for debugging this?Yes, it would. Jobs hanging can be a bug anywhere in the stack (e.g., Hardware bug, KMD bug, UMD bug, application bug, etc...) but the devcoredump would give us some hints.
Quoted 7 lines
Quoted 3 lines
Full log at https://openqa.qubes-os.org/tests/168813/file/serial0.txt
(warning, almost 200MB of those errors...)The issue still happens with Linux 7.0.12. Current log (quite similar to the previous one):
https://openqa.qubes-os.org/tests/184602/logfile?filename=serial0.txtHmm, the 'not started' messages in the dmesg are a bit concerning as this really shouldn't be possible to trigger even if user space is doing something wrong.
Quoted 6 lines
Can you file a gitlab issue against Xe here: https://gitlab.freedesktop.org/drm/xe/kernel/issues/new
TBH, I have no idea if running Xen / Qubes OS + Xe is something anyone at Intel has tried out, so please include instructions on to how reproduce and we will see in someone on engineering team can take a look at this and if issues in Xe KMD exist, try to get these fixed.
I've opened https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8382
including the crashdump and fresh logs.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab