Skip to contentSkip to conversation

meta-yocto: bump qemu preferred version to 4.4

PATCH In review Bruce Ashfield 4 participants 15 messages Feb 11, 2016 – Mar 10, 2016

lore
Share & tools
lore
#0 lore raw

cover meta-yocto: bump qemu preferred version to 4.4

Hi all,

4.4 has been in test in various forms for over a month now, without major issues on any of the image types I've tried. Now that we have -stable updates to 4.4 (as well as -rt updates), things are looking good to make it the default for the qemu machines.

I'm also at the limit of the combos I can build and test, so this needs to go out for wider testing.

Cheers,

Bruce

The following changes since commit 4e7320cc81178fe17c870e987a526d3f29caf906:

linux-yocto/4.1: galileo backports and support (2016-02-11 10:01:55 -0500)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib zedd/kernel-yocto
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=zedd/kernel-yocto

Bruce Ashfield (1):
poky: update qemu* to prefer 4.4 kernel

 meta-yocto/conf/distro/poky.conf | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/1 poky: update qemu* to prefer 4.4 kernel

4.4 is out and has had enough mileage to be the default for the qemu machines. Tested with sato, minimal and kernel dev image types.

---
 meta-yocto/conf/distro/poky.conf | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
meta-yocto/conf/distro/poky.conf +7 -7
index dec364498553..ede728dde1fd 100644
@@ -18,13 +18,13 @@ POKY_DEFAULT_EXTRA_RRECOMMENDS = "kernel-module-af-packet"
 
 DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT} ${DISTRO_FEATURES_LIBC} ${POKY_DEFAULT_DISTRO_FEATURES}"
 
-PREFERRED_VERSION_linux-yocto ?= "4.1%"
-PREFERRED_VERSION_linux-yocto_qemux86 ?= "4.1%"
-PREFERRED_VERSION_linux-yocto_qemux86-64 ?= "4.1%"
-PREFERRED_VERSION_linux-yocto_qemuarm ?= "4.1%"
-PREFERRED_VERSION_linux-yocto_qemumips ?= "4.1%"
-PREFERRED_VERSION_linux-yocto_qemumips64 ?= "4.1%"
-PREFERRED_VERSION_linux-yocto_qemuppc ?= "4.1%"
+PREFERRED_VERSION_linux-yocto ?= "4.4%"
+PREFERRED_VERSION_linux-yocto_qemux86 ?= "4.4%"
+PREFERRED_VERSION_linux-yocto_qemux86-64 ?= "4.4%"
+PREFERRED_VERSION_linux-yocto_qemuarm ?= "4.4%"
+PREFERRED_VERSION_linux-yocto_qemumips ?= "4.4%"
+PREFERRED_VERSION_linux-yocto_qemumips64 ?= "4.4%"
+PREFERRED_VERSION_linux-yocto_qemuppc ?= "4.4%"
 
 SDK_NAME = "${DISTRO}-${TCLIBC}-${SDK_ARCH}-${IMAGE_BASENAME}-${TUNE_PKGARCH}"
 SDKPATH = "/opt/${DISTRO}/${SDK_VERSION}"
Signature · 1 line
2.1.0
#2 lore raw

Re: [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 3 lines · Bruce Ashfield

4.4 is out and has had enough mileage to be the default for the qemu machines. Tested with sato, minimal and kernel dev image types.

This mostly worked except for qemux86 which is showing X failing on all builds. Its quite reproducible, "MACHINE=qemux86 bitbake core-image -sato -c testimage" shows it.

https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/6
48/steps/Running%20Sanity%20Tests/logs/stdio
is one example from the autobuilder.

Poking at the image manually, it appears Xorg fails due to:

[3587706.730] (EE) XKB: Could not invoke xkbcomp
[3587706.730] (EE) XKB: Couldn't compile keymap
[3587706.731] (EE) XKB: Failed to load keymap. Loading default keymap
instead.
[3587706.765] (EE) XKB: Could not invoke xkbcomp
[3587706.765] (EE) XKB: Couldn't compile keymap

which prompted me to look at dmesg:

EXT4-fs (vda): re-mounted. Opts: data=ordered
random: dd urandom read with 51 bits of entropy available
x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

------------[ cut here ]------------
WARNING: CPU: 0 PID: 705 at /media/build1/poky/build/tmp/work-shared/qemux86/kernel-source/arch/x86/mm/pat.c:985 untrack_pfn+0xaf/0xc0()
Modules linked in: uvesafb
CPU: 0 PID: 705 Comm: Xorg Not tainted 4.4.1-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
 00000000 00000000 cf14dd78 c1397ab2 00000000 cf14dda8 c1051477 c1aa4f6c
 00000000 000002c1 c1a9fa4c 000003d9 c104b98f c104b98f cf244000 b6355000
 00000000 cf14ddb8 c1051552 00000009 00000000 cf14dde0 c104b98f cf14ddd0
Call Trace:
 [<c1397ab2>] dump_stack+0x4b/0x79
 [<c1051477>] warn_slowpath_common+0x87/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c1051552>] warn_slowpath_null+0x22/0x30
 [<c104b98f>] untrack_pfn+0xaf/0xc0
 [<c104d54c>] ? kmap_atomic_prot+0x3c/0xf0
 [<c114e17f>] unmap_single_vma+0x4ef/0x500
 [<c114f007>] unmap_vmas+0x37/0x50
 [<c1154f8f>] exit_mmap+0x5f/0xf0
 [<c104eedd>] mmput+0x2d/0xb0
 [<c105009c>] copy_process+0xd2c/0x13c0
 [<c1050892>] _do_fork+0x82/0x340
 [<c105f2d1>] ? SyS_rt_sigaction+0x51/0xa0
 [<c1050c3c>] SyS_clone+0x2c/0x30
 [<c1001a03>] do_syscall_32_irqs_on+0x53/0xb0
 [<c189a94a>] entry_INT80_32+0x2a/0x2a
---[ end trace be3e0a61097feddc ]---

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining Installing knfsd (copyright (C) 1996 [email protected]).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period (net c1c271c0)

which fits since do_fork() is probably failing and would cause this error. It looks like the above is only a warning but could be fatal later I guess.

A quick look at upstream makes me wonder about:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d9fe4fab11976e56b2e992980bf6ce948bdf02ac

which changes the code in this area.

Any idea what is going on here?

Cheers,

Richard

#3 lore raw

Re: [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 6 lines · Richard Purdie

A quick look at upstream makes me wonder about:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commi
t/?id=d9fe4fab11976e56b2e992980bf6ce948bdf02ac

which changes the code in this area.

FWIW this doesn't help, the problem still exists after applying it.

Setting CONFIG_X86_PAT to is not set in the defconfig did "fix" it, unsurprisingly.

Cheers,

Richard

#4 lore raw

Re: [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 7 lines · Richard Purdie
Quoted 6 lines · Richard Purdie

A quick look at upstream makes me wonder about:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/com
mi
t/?id=d9fe4fab11976e56b2e992980bf6ce948bdf02ac

Quoted 6 lines
Quoted 1 line

which changes the code in this area.

FWIW this doesn't help, the problem still exists after applying it.

Setting CONFIG_X86_PAT to is not set in the defconfig did "fix" it, unsurprisingly.

Also, in the last set of builds, despite booting with "nopat", we see:

https://autobuilder.yoctoproject.org/main/builders/nightly-x86-lsb/buil
ds/637/steps/Running%20Sanity%20Tests/logs/stdio

Central error: [ 9.298049] Failed to add WC MTRR for [fd000000-fdffffff]; performance may suffer.

Why we only see this on the lsb run I don't know but that address looks like the one causing problems elsewhere with PAT. It could therefore look like this is related to uvesafb.

Cheers,

Richard

#5 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

I'm moving the discussion to OE-Core and pulling in some kernel people. I think I understand what is wrong and how to fix it but I could use someone who actually knows this code.

To summarise the story so far, on qemux86, X doesn't start and there is a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

------------[ cut here ]------------
WARNING: CPU: 0 PID: 705 at /media/build1/poky/build/tmp/work-shared/qemux86/kernel-source/arch/x86/mm/pat.c:985 untrack_pfn+0xaf/0xc0()
Modules linked in: uvesafb
CPU: 0 PID: 705 Comm: Xorg Not tainted 4.4.1-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
 00000000 00000000 cf14dd78 c1397ab2 00000000 cf14dda8 c1051477 c1aa4f6c
 00000000 000002c1 c1a9fa4c 000003d9 c104b98f c104b98f cf244000 b6355000
 00000000 cf14ddb8 c1051552 00000009 00000000 cf14dde0 c104b98f cf14ddd0
Call Trace:
 [<c1397ab2>] dump_stack+0x4b/0x79
 [<c1051477>] warn_slowpath_common+0x87/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c1051552>] warn_slowpath_null+0x22/0x30
 [<c104b98f>] untrack_pfn+0xaf/0xc0
 [<c104d54c>] ? kmap_atomic_prot+0x3c/0xf0
 [<c114e17f>] unmap_single_vma+0x4ef/0x500
 [<c114f007>] unmap_vmas+0x37/0x50
 [<c1154f8f>] exit_mmap+0x5f/0xf0
 [<c104eedd>] mmput+0x2d/0xb0
 [<c105009c>] copy_process+0xd2c/0x13c0
 [<c1050892>] _do_fork+0x82/0x340
 [<c105f2d1>] ? SyS_rt_sigaction+0x51/0xa0
 [<c1050c3c>] SyS_clone+0x2c/0x30
 [<c1001a03>] do_syscall_32_irqs_on+0x53/0xb0
 [<c189a94a>] entry_INT80_32+0x2a/0x2a
---[ end trace be3e0a61097feddc ]---

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

The entry in question is setup by uvesafb which in its uvesafb_ioremap() function calls ioremap_wc().

It appears that Xorg mmaps this from userspace, then later does a fork() to execute a utility. At this point, when creating the vmas for the new process, the pat code says "eeek!" as the protection mode for the new vmas don't match the old one, returns -EINVAL, the process dies and X goes with it.

There are a few hammers we can hit this with, we can boot with "nopat" option which makes the problem go away, or turn off CONFIG_X86_PAT. No surprises there. Changing uvesafb to use mtrr=0 doesn't help since the ioremap_wc call still happens.

The real issue is the "expected mapping type uncached-minus for got write-combining" message, it all goes wrong from there.

Upon looking at the code and scratching my head for a long while, I notice that there are two ways of representing the protection mode data, "enum page_cache_mode" and "pgprot_t & _PAGE_CACHE_MASK".

The exact meaning of pgprot_t depends on which CPU you're running, older CPUs have errata meaning only a small number of bits can be used. The exact mapping table is determined by __cachemode2pte_tbl and is updated at boot by calls from update_cache_mode_entry().

The result of this if you map enum -> pgprot_t, then try to do pgprot_t -> enum, you can get different values since its not a 1:1 mapping.

This means the comparison in reserve_pfn_range() where it does "pcm != want_pcm" isn't correct and can trigger even in cases where there isn't a problem.

This can be "fixed" by doing cachemode2protval(pcm) != cachemode2protval(want_pcm) and checking whether the protection bits match, rather than the enum values, since in reality this is what we really care about.

I can confirm that if I make that change, X boots up just fine.

The problem is I really have no idea what I'm doing :).

Could someone who understands this code have a look and see whether the above makes sense and if it does, perhaps open a discussion with upstream about how to fix this properly (assuming my change isn't actually the correct fix)?

We don't see this on qemux86-64 since that has more PAT bits working and hence the values map correctly.

Bruce: Would you accept a patch doing the above for now?

Cheers,

Richard

#6 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 80 lines · Richard Purdie

I'm moving the discussion to OE-Core and pulling in some kernel people. I think I understand what is wrong and how to fix it but I could use someone who actually knows this code.

To summarise the story so far, on qemux86, X doesn't start and there is a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

------------[ cut here ]------------
WARNING: CPU: 0 PID: 705 at
/media/build1/poky/build/tmp/work-shared/qemux86/kernel-source/arch/x86/mm/pat.c:985
untrack_pfn+0xaf/0xc0()
Modules linked in: uvesafb
CPU: 0 PID: 705 Comm: Xorg Not tainted 4.4.1-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS

rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
00000000 00000000 cf14dd78 c1397ab2 00000000 cf14dda8 c1051477 c1aa4f6c 00000000 000002c1 c1a9fa4c 000003d9 c104b98f c104b98f cf244000 b6355000 00000000 cf14ddb8 c1051552 00000009 00000000 cf14dde0 c104b98f cf14ddd0

Call Trace:
 [<c1397ab2>] dump_stack+0x4b/0x79
 [<c1051477>] warn_slowpath_common+0x87/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c1051552>] warn_slowpath_null+0x22/0x30
 [<c104b98f>] untrack_pfn+0xaf/0xc0
 [<c104d54c>] ? kmap_atomic_prot+0x3c/0xf0
 [<c114e17f>] unmap_single_vma+0x4ef/0x500
 [<c114f007>] unmap_vmas+0x37/0x50
 [<c1154f8f>] exit_mmap+0x5f/0xf0
 [<c104eedd>] mmput+0x2d/0xb0
 [<c105009c>] copy_process+0xd2c/0x13c0
 [<c1050892>] _do_fork+0x82/0x340
 [<c105f2d1>] ? SyS_rt_sigaction+0x51/0xa0
 [<c1050c3c>] SyS_clone+0x2c/0x30
 [<c1001a03>] do_syscall_32_irqs_on+0x53/0xb0
 [<c189a94a>] entry_INT80_32+0x2a/0x2a
---[ end trace be3e0a61097feddc ]---

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

The entry in question is setup by uvesafb which in its uvesafb_ioremap() function calls ioremap_wc().

It appears that Xorg mmaps this from userspace, then later does a fork() to execute a utility. At this point, when creating the vmas for the new process, the pat code says "eeek!" as the protection mode for the new vmas don't match the old one, returns -EINVAL, the process dies and X goes with it.

There are a few hammers we can hit this with, we can boot with "nopat" option which makes the problem go away, or turn off CONFIG_X86_PAT. No surprises there. Changing uvesafb to use mtrr=0 doesn't help since the ioremap_wc call still happens.

The real issue is the "expected mapping type uncached-minus for got write-combining" message, it all goes wrong from there.

Upon looking at the code and scratching my head for a long while, I notice that there are two ways of representing the protection mode data, "enum page_cache_mode" and "pgprot_t & _PAGE_CACHE_MASK".

The exact meaning of pgprot_t depends on which CPU you're running, older CPUs have errata meaning only a small number of bits can be used. The exact mapping table is determined by __cachemode2pte_tbl and is updated at boot by calls from update_cache_mode_entry().

The result of this if you map enum -> pgprot_t, then try to do pgprot_t -> enum, you can get different values since its not a 1:1 mapping.

This means the comparison in reserve_pfn_range() where it does "pcm != want_pcm" isn't correct and can trigger even in cases where there isn't a problem.

This can be "fixed" by doing cachemode2protval(pcm) != cachemode2protval(want_pcm) and checking whether the protection bits match, rather than the enum values, since in reality this is what we really care about.

Quoted 9 lines

I can confirm that if I make that change, X boots up just fine.

The problem is I really have no idea what I'm doing :).

Could someone who understands this code have a look and see whether the above makes sense and if it does, perhaps open a discussion with upstream about how to fix this properly (assuming my change isn't actually the correct fix)?

I'm not familiar with this code either, but I'll start doing some bisects to see if
we can isolate the commit that introduced the behaviour change. This is the sort of thing that is very rare, and really isn't supposed to happen. So until
I read the commit log where the change was triggered, I'd lean to it being a bug
and something to report and fix upstream.

I know that I booted sato with the 4.4 update, do we have any other confirmations
that this worked, or was I the only one doing 4.4 boot tests until I sent the
patch to bump the preferred version ?

Quoted 6 lines

We don't see this on qemux86-64 since that has more PAT bits working and hence the values map correctly.

Bruce: Would you accept a patch doing the above for now?

I would. I'd rather not paper over this by changing the config fragments, since
once we throw that switch, it'll most likely sit there .. covered up.

Bruce

Quoted 12 lines

Cheers,

Richard

--
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"
#7 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

In reply to Richard Purdie
Quoted 46 lines · [Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel] On 13/02/2016…

I'm moving the discussion to OE-Core and pulling in some kernel people. I think I understand what is wrong and how to fix it but I could use someone who actually knows this code.

To summarise the story so far, on qemux86, X doesn't start and there is a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

------------[ cut here ]------------
WARNING: CPU: 0 PID: 705 at /media/build1/poky/build/tmp/work-shared/qemux86/kernel-source/arch/x86/mm/pat.c:985 untrack_pfn+0xaf/0xc0()
Modules linked in: uvesafb
CPU: 0 PID: 705 Comm: Xorg Not tainted 4.4.1-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
 00000000 00000000 cf14dd78 c1397ab2 00000000 cf14dda8 c1051477 c1aa4f6c
 00000000 000002c1 c1a9fa4c 000003d9 c104b98f c104b98f cf244000 b6355000
 00000000 cf14ddb8 c1051552 00000009 00000000 cf14dde0 c104b98f cf14ddd0
Call Trace:
 [<c1397ab2>] dump_stack+0x4b/0x79
 [<c1051477>] warn_slowpath_common+0x87/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c1051552>] warn_slowpath_null+0x22/0x30
 [<c104b98f>] untrack_pfn+0xaf/0xc0
 [<c104d54c>] ? kmap_atomic_prot+0x3c/0xf0
 [<c114e17f>] unmap_single_vma+0x4ef/0x500
 [<c114f007>] unmap_vmas+0x37/0x50
 [<c1154f8f>] exit_mmap+0x5f/0xf0
 [<c104eedd>] mmput+0x2d/0xb0
 [<c105009c>] copy_process+0xd2c/0x13c0
 [<c1050892>] _do_fork+0x82/0x340
 [<c105f2d1>] ? SyS_rt_sigaction+0x51/0xa0
 [<c1050c3c>] SyS_clone+0x2c/0x30
 [<c1001a03>] do_syscall_32_irqs_on+0x53/0xb0
 [<c189a94a>] entry_INT80_32+0x2a/0x2a
---[ end trace be3e0a61097feddc ]---

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

The entry in question is setup by uvesafb which in its uvesafb_ioremap() function calls ioremap_wc().

It appears that Xorg mmaps this from userspace, then later does a fork() to execute a utility. At this point, when creating the vmas for the new process, the pat code says "eeek!" as the protection mode for the new vmas don't match the old one, returns -EINVAL, the process dies and X goes with it.

Quoted 4 lines

There are a few hammers we can hit this with, we can boot with "nopat" option which makes the problem go away, or turn off CONFIG_X86_PAT. No surprises there. Changing uvesafb to use mtrr=0 doesn't help since the ioremap_wc call still happens.

Disabling PAT for qemu wouldn't be some horrible crime in the end; the help text for the Kconfig option itself says:

Say N here if you see bootup problems (boot crash, boot hang, spontaneous reboots) or a non-working video driver.

...and in theory PAT and the older MTRR are supposed to be performance enhancements but not critical to have present. I find it hard to get excited about qemu video performance through the vesa driver. :) That said, it would be nice to fully understand what went pear shaped.

Quoted 20 lines

The real issue is the "expected mapping type uncached-minus for got write-combining" message, it all goes wrong from there.

Upon looking at the code and scratching my head for a long while, I notice that there are two ways of representing the protection mode data, "enum page_cache_mode" and "pgprot_t & _PAGE_CACHE_MASK".

The exact meaning of pgprot_t depends on which CPU you're running, older CPUs have errata meaning only a small number of bits can be used. The exact mapping table is determined by __cachemode2pte_tbl and is updated at boot by calls from update_cache_mode_entry().

The result of this if you map enum -> pgprot_t, then try to do pgprot_t -> enum, you can get different values since its not a 1:1 mapping.

This means the comparison in reserve_pfn_range() where it does "pcm != want_pcm" isn't correct and can trigger even in cases where there isn't a problem.

Quoted 8 lines

This can be "fixed" by doing cachemode2protval(pcm) != cachemode2protval(want_pcm) and checking whether the protection bits match, rather than the enum values, since in reality this is what we really care about.

I can confirm that if I make that change, X boots up just fine.

The problem is I really have no idea what I'm doing :).

I know the feeling. :) Usually I find that being able to pinpoint the
exact commit where things failed adds that final bit of information needed to get to the bottom of things. Bruce and I fought with disks disappearing on qemu versatile a while back and with a bisect traced it down to some cryptic PCI swizzle mess. But without the bisect pointing us at where it went wrong, I'm not sure what we'd have done. This case is probably not that bad; it sounds like you've got 95% of it figured out already.

Anyway to that end, I'm assuming here if we insert the 4.1 kernel (presumably which also has PAT enabled) and leave X11 and qemu alone, things work. If so, we can use "debugpat" bootarg or debugfs to compare the 4.1 and 4.4 PAT entries (as per Documentation/x86/pat.txt) to see where things differ between the two kernels.

And/or we can look at some of the relevant changes between the two versions (see below) and spot test reverts of any that look suspect. Or, just jump to a brute force bisect, while keeping an eye on the .config file along the way to ensure it remains consistent as we go.

If this is still unresolved Tues when I'm back in the office and more easily able to test gfx issues, I'll look at doing bisection.

P.
--

Note: below listing does not account for gregKH stable or yocto changes!

paul@acer:~/git/linux-head$ git log --oneline ^v4.1 v4.4 arch/x86/mm/pat.c 
35a5a10 x86/mm/pat: Extend set_page_memtype() to support Write-Through type
d1b4bfb x86/mm/pat: Add pgprot_writethrough()
0d69bdf x86/mm/pat: Change reserve_memtype() for Write-Through type
d79a40c x86/mm/pat: Use 7th PAT MSR slot for Write-Through PAT type
7202fdb x86/mm/pat: Remove pat_enabled() checks
9cd25aa x86/mm/pat: Emulate PAT when it is disabled
9dac629 x86/mm/pat: Untangle pat_init()
fbe7193 x86/mm/pat: Export pat_enabled()
cb32edf x86/mm/pat: Wrap pat_enabled into a function API
9e76561 x86/mm/pat: Convert to pr_*() usage
b73522e x86/mm/mtrr: Enhance MTRR checks in kernel mapping helpers
paul@acer:~/git/linux-head$ git log --oneline ^v4.1 v4.4 arch/x86/include/asm/pgtable_types.h
70f15287 x86/mm: Fix regression with huge pages on PAE
f70abb0 x86/asm: Fix pud/pmd interfaces to handle large PAT bit
4be4c1f x86/asm: Add pud/pmd mask interfaces to handle large PAT bit
d1b4bfb x86/mm/pat: Add pgprot_writethrough()
paul@acer:~/git/linux-head$ git log --oneline ^v4.1 v4.4 drivers/video/fbdev/uvesafb.c
9c27847 kernel/params: constify struct kernel_param_ops uses
paul@acer:~/git/linux-head$ git log --no-merges --oneline ^v4.1 v4.4 arch/x86/mm/ioremap*
8a0a5da x86/mm: Fix newly introduced printk format warnings
9a58eeb x86/mm: Remove region_is_ram() call from ioremap
1c9cf9b x86/mm: Move warning from __ioremap_check_ram() to the call site
623dffb x86/mm/pat: Add set_memory_wt() for Write-Through type
d838270 x86/mm, asm-generic: Add ioremap_wt() for creating Write-Through mappings
7202fdb x86/mm/pat: Remove pat_enabled() checks
1e6277d x86/mm: Mark arch_ioremap_p{m,u}d_supported() __init
cb32edf x86/mm/pat: Wrap pat_enabled into a function API
e4b6be3 x86/mm: Add ioremap_uc() helper to map memory uncacheable (not UC-)
562bfca x86/mm: Clean up types in xlate_dev_mem_ptr() some more
#8 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

In reply to Richard Purdie
Quoted 8 lines · [Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel] On 13/02/2016…

I'm moving the discussion to OE-Core and pulling in some kernel people. I think I understand what is wrong and how to fix it but I could use someone who actually knows this code.

To summarise the story so far, on qemux86, X doesn't start and there is a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

So Bruce helped me set up a reproducer locally today since he'd already invested the time on that, and then I boiled that down to divorce it from the slower steps of build-deploy-boot to make the bisect something that mortal humans could tolerate.

Amusingly enough that led to:

commit 9cd25aac1f44f269de5ecea11f7d927f37f1d01c

Author: Borislav Petkov <[email protected]>
Date:   Thu Jun 4 18:55:10 2015 +0200

x86/mm/pat: Emulate PAT when it is disabled

So while some of us were joking on IRC about the validity of forcibly disabling PAT (via cmdline or Kconfig) as a workaround, the one line shortlog above tells us that it wasn't so off the mark after all.

Bruce and I will decide what to do with this tomorrow, but since Richard spent so much time on it, I thought he'd like to know this in the
interim. Good times. :-/

Paul.
--

Quoted 87 lines
------------[ cut here ]------------
WARNING: CPU: 0 PID: 705 at /media/build1/poky/build/tmp/work-shared/qemux86/kernel-source/arch/x86/mm/pat.c:985 untrack_pfn+0xaf/0xc0()
Modules linked in: uvesafb
CPU: 0 PID: 705 Comm: Xorg Not tainted 4.4.1-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
 00000000 00000000 cf14dd78 c1397ab2 00000000 cf14dda8 c1051477 c1aa4f6c
 00000000 000002c1 c1a9fa4c 000003d9 c104b98f c104b98f cf244000 b6355000
 00000000 cf14ddb8 c1051552 00000009 00000000 cf14dde0 c104b98f cf14ddd0
Call Trace:
 [<c1397ab2>] dump_stack+0x4b/0x79
 [<c1051477>] warn_slowpath_common+0x87/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c104b98f>] ? untrack_pfn+0xaf/0xc0
 [<c1051552>] warn_slowpath_null+0x22/0x30
 [<c104b98f>] untrack_pfn+0xaf/0xc0
 [<c104d54c>] ? kmap_atomic_prot+0x3c/0xf0
 [<c114e17f>] unmap_single_vma+0x4ef/0x500
 [<c114f007>] unmap_vmas+0x37/0x50
 [<c1154f8f>] exit_mmap+0x5f/0xf0
 [<c104eedd>] mmput+0x2d/0xb0
 [<c105009c>] copy_process+0xd2c/0x13c0
 [<c1050892>] _do_fork+0x82/0x340
 [<c105f2d1>] ? SyS_rt_sigaction+0x51/0xa0
 [<c1050c3c>] SyS_clone+0x2c/0x30
 [<c1001a03>] do_syscall_32_irqs_on+0x53/0xb0
 [<c189a94a>] entry_INT80_32+0x2a/0x2a
---[ end trace be3e0a61097feddc ]---

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

The entry in question is setup by uvesafb which in its uvesafb_ioremap() function calls ioremap_wc().

It appears that Xorg mmaps this from userspace, then later does a fork() to execute a utility. At this point, when creating the vmas for the new process, the pat code says "eeek!" as the protection mode for the new vmas don't match the old one, returns -EINVAL, the process dies and X goes with it.

There are a few hammers we can hit this with, we can boot with "nopat" option which makes the problem go away, or turn off CONFIG_X86_PAT. No surprises there. Changing uvesafb to use mtrr=0 doesn't help since the ioremap_wc call still happens.

The real issue is the "expected mapping type uncached-minus for got write-combining" message, it all goes wrong from there.

Upon looking at the code and scratching my head for a long while, I notice that there are two ways of representing the protection mode data, "enum page_cache_mode" and "pgprot_t & _PAGE_CACHE_MASK".

The exact meaning of pgprot_t depends on which CPU you're running, older CPUs have errata meaning only a small number of bits can be used. The exact mapping table is determined by __cachemode2pte_tbl and is updated at boot by calls from update_cache_mode_entry().

The result of this if you map enum -> pgprot_t, then try to do pgprot_t -> enum, you can get different values since its not a 1:1 mapping.

This means the comparison in reserve_pfn_range() where it does "pcm != want_pcm" isn't correct and can trigger even in cases where there isn't a problem.

This can be "fixed" by doing cachemode2protval(pcm) != cachemode2protval(want_pcm) and checking whether the protection bits match, rather than the enum values, since in reality this is what we really care about.

I can confirm that if I make that change, X boots up just fine.

The problem is I really have no idea what I'm doing :).

Could someone who understands this code have a look and see whether the above makes sense and if it does, perhaps open a discussion with upstream about how to fix this properly (assuming my change isn't actually the correct fix)?

We don't see this on qemux86-64 since that has more PAT bits working and hence the values map correctly.

Bruce: Would you accept a patch doing the above for now?

Cheers,

Richard

#9 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 24 lines · Paul Gortmaker
Quoted 8 lines · [Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel] On 13/02/2016…

I'm moving the discussion to OE-Core and pulling in some kernel people. I think I understand what is wrong and how to fix it but I could use someone who actually knows this code.

To summarise the story so far, on qemux86, X doesn't start and there is a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

So Bruce helped me set up a reproducer locally today since he'd already invested the time on that, and then I boiled that down to divorce it from the slower steps of build-deploy-boot to make the bisect something that mortal humans could tolerate.

Amusingly enough that led to:

commit 9cd25aac1f44f269de5ecea11f7d927f37f1d01c

Author: Borislav Petkov <[email protected]>
Date:   Thu Jun 4 18:55:10 2015 +0200

x86/mm/pat: Emulate PAT when it is disabled

Quoted 7 lines

So while some of us were joking on IRC about the validity of forcibly disabling PAT (via cmdline or Kconfig) as a workaround, the one line shortlog above tells us that it wasn't so off the mark after all.

Bruce and I will decide what to do with this tomorrow, but since Richard spent so much time on it, I thought he'd like to know this in the
interim. Good times. :-/

As another follow up. The thread can be summarized as "It doesn't look like it should have worked before, and qemu's pat emulation may be the issue'.

The suggestion is to run with 'nopat', which is what Richard originally did.

So I'm going to prep a patch that drops the kernel patch, and leaves nopat enabled on the qemu command line. That should get us put back together in a semi-permanent way.

Bruce

Quoted 91 lines

Paul.
--

Quoted 87 lines
------------[ cut here ]------------
WARNING: CPU: 0 PID: 705 at /media/build1/poky/build/tmp/work-shared/qemux86/kernel-source/arch/x86/mm/pat.c:985 untrack_pfn+0xaf/0xc0()
Modules linked in: uvesafb
CPU: 0 PID: 705 Comm: Xorg Not tainted 4.4.1-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
  00000000 00000000 cf14dd78 c1397ab2 00000000 cf14dda8 c1051477 c1aa4f6c
  00000000 000002c1 c1a9fa4c 000003d9 c104b98f c104b98f cf244000 b6355000
  00000000 cf14ddb8 c1051552 00000009 00000000 cf14dde0 c104b98f cf14ddd0
Call Trace:
  [<c1397ab2>] dump_stack+0x4b/0x79
  [<c1051477>] warn_slowpath_common+0x87/0xc0
  [<c104b98f>] ? untrack_pfn+0xaf/0xc0
  [<c104b98f>] ? untrack_pfn+0xaf/0xc0
  [<c1051552>] warn_slowpath_null+0x22/0x30
  [<c104b98f>] untrack_pfn+0xaf/0xc0
  [<c104d54c>] ? kmap_atomic_prot+0x3c/0xf0
  [<c114e17f>] unmap_single_vma+0x4ef/0x500
  [<c114f007>] unmap_vmas+0x37/0x50
  [<c1154f8f>] exit_mmap+0x5f/0xf0
  [<c104eedd>] mmput+0x2d/0xb0
  [<c105009c>] copy_process+0xd2c/0x13c0
  [<c1050892>] _do_fork+0x82/0x340
  [<c105f2d1>] ? SyS_rt_sigaction+0x51/0xa0
  [<c1050c3c>] SyS_clone+0x2c/0x30
  [<c1001a03>] do_syscall_32_irqs_on+0x53/0xb0
  [<c189a94a>] entry_INT80_32+0x2a/0x2a
---[ end trace be3e0a61097feddc ]---

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

The entry in question is setup by uvesafb which in its uvesafb_ioremap() function calls ioremap_wc().

It appears that Xorg mmaps this from userspace, then later does a fork() to execute a utility. At this point, when creating the vmas for the new process, the pat code says "eeek!" as the protection mode for the new vmas don't match the old one, returns -EINVAL, the process dies and X goes with it.

There are a few hammers we can hit this with, we can boot with "nopat" option which makes the problem go away, or turn off CONFIG_X86_PAT. No surprises there. Changing uvesafb to use mtrr=0 doesn't help since the ioremap_wc call still happens.

The real issue is the "expected mapping type uncached-minus for got write-combining" message, it all goes wrong from there.

Upon looking at the code and scratching my head for a long while, I notice that there are two ways of representing the protection mode data, "enum page_cache_mode" and "pgprot_t & _PAGE_CACHE_MASK".

The exact meaning of pgprot_t depends on which CPU you're running, older CPUs have errata meaning only a small number of bits can be used. The exact mapping table is determined by __cachemode2pte_tbl and is updated at boot by calls from update_cache_mode_entry().

The result of this if you map enum -> pgprot_t, then try to do pgprot_t -> enum, you can get different values since its not a 1:1 mapping.

This means the comparison in reserve_pfn_range() where it does "pcm != want_pcm" isn't correct and can trigger even in cases where there isn't a problem.

This can be "fixed" by doing cachemode2protval(pcm) != cachemode2protval(want_pcm) and checking whether the protection bits match, rather than the enum values, since in reality this is what we really care about.

I can confirm that if I make that change, X boots up just fine.

The problem is I really have no idea what I'm doing :).

Could someone who understands this code have a look and see whether the above makes sense and if it does, perhaps open a discussion with upstream about how to fix this properly (assuming my change isn't actually the correct fix)?

We don't see this on qemux86-64 since that has more PAT bits working and hence the values map correctly.

Bruce: Would you accept a patch doing the above for now?

Cheers,

Richard

#10 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 48 lines · Bruce Ashfield
Quoted 38 lines · Paul Gortmaker

[Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel] On

Quoted 6 lines · 13/02/2016 (Sat 17:17) Richard Purdie wrote:

I'm moving the discussion to OE-Core and pulling in some kernel people.
I think I understand what is wrong and how to fix it but I could use
someone who actually knows this code.

Quoted 6 lines

To summarise the story so far, on qemux86, X doesn't start and there is
a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

So Bruce helped me set up a reproducer locally today since he'd already
invested the time on that, and then I boiled that down to divorce it
from the slower steps of build-deploy-boot to make the bisect something
that mortal humans could tolerate.

Amusingly enough that led to:

commit 9cd25aac1f44f269de5ecea11f7d927f37f1d01c

Author: Borislav Petkov <[email protected]>
Date:   Thu Jun 4 18:55:10 2015 +0200

x86/mm/pat: Emulate PAT when it is disabled

So while some of us were joking on IRC about the validity of forcibly
disabling PAT (via cmdline or Kconfig) as a workaround, the one line
shortlog above tells us that it wasn't so off the mark after all.

Quoted 4 lines

Bruce and I will decide what to do with this tomorrow, but since Richard
spent so much time on it, I thought he'd like to know this in the
interim. Good times. :-/

As another follow up. The thread can be summarized as "It doesn't look like it should have worked before, and qemu's pat emulation may be the issue'.

Quoted 7 lines

The suggestion is to run with 'nopat', which is what Richard originally
did.

So I'm going to prep a patch that drops the kernel patch, and leaves nopat enabled on the qemu command line. That should get us put back together in a semi-permanent way.

How sure are we this is a bug in QEMU's pat emulation? If that is the case we should file a bug against qemu and try and fix it rather than work around it...

Cheers,

Richard

#11 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 53 lines · Richard Purdie
Quoted 52 lines · Bruce Ashfield
Quoted 38 lines · Paul Gortmaker

[Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel] On

Quoted 6 lines · 13/02/2016 (Sat 17:17) Richard Purdie wrote:

I'm moving the discussion to OE-Core and pulling in some kernel people.
I think I understand what is wrong and how to fix it but I could use
someone who actually knows this code.

Quoted 6 lines

To summarise the story so far, on qemux86, X doesn't start and there is
a backtrace in the logs:

x86/PAT: Xorg:705 map pfn expected mapping type uncached-minus for [mem 0xfd000000-0xfdffffff], got write-combining

So Bruce helped me set up a reproducer locally today since he'd already
invested the time on that, and then I boiled that down to divorce it
from the slower steps of build-deploy-boot to make the bisect something
that mortal humans could tolerate.

Amusingly enough that led to:

commit 9cd25aac1f44f269de5ecea11f7d927f37f1d01c

Author: Borislav Petkov <[email protected]>
Date:   Thu Jun 4 18:55:10 2015 +0200

x86/mm/pat: Emulate PAT when it is disabled

So while some of us were joking on IRC about the validity of forcibly
disabling PAT (via cmdline or Kconfig) as a workaround, the one line
shortlog above tells us that it wasn't so off the mark after all.

Quoted 4 lines

Bruce and I will decide what to do with this tomorrow, but since Richard
spent so much time on it, I thought he'd like to know this in the
interim. Good times. :-/

As another follow up. The thread can be summarized as "It doesn't look like it should have worked before, and qemu's pat emulation may be the issue'.

The suggestion is to run with 'nopat', which is what Richard originally
did.

Quoted 7 lines
Quoted 3 lines

So I'm going to prep a patch that drops the kernel patch, and leaves nopat enabled on the qemu command line. That should get us put back together in a semi-permanent way.

How sure are we this is a bug in QEMU's pat emulation? If that is the case we should file a bug against qemu and try and fix it rather than work around it...

It could still be something that the kernel can work around, Toshi did say:

There is a matter of how qemu emulates CPU features. There is no such Intel CPU that supports PAT w/o MTRR. This is why the current code assumes this dependency.

Which is likely the trigger, we've send information about the cpu to him, and with that there's a chance for a pat fix.

He repeated our thought of running with 'nopat' while a fix is considered.

It may be some time before that happens, and I was going to test with the kernel patch dropped, and nopat in the qemu boot args. If that works, I'd rather run with that, and then revisit when (if) there's more changes upstream.

Bruce

Quoted 5 lines

Cheers,

Richard

#12 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 33 lines · Bruce Ashfield
Quoted 15 lines · Richard Purdie
Quoted 8 lines · Bruce Ashfield

As another follow up. The thread can be summarized as "It doesn't look like it should have worked before, and qemu's pat emulation may be the issue'.

The suggestion is to run with 'nopat', which is what Richard originally
did.

Quoted 5 lines

So I'm going to prep a patch that drops the kernel patch, and leaves
nopat enabled on the qemu command line. That should get us put back
together in a semi-permanent way.

Quoted 5 lines

How sure are we this is a bug in QEMU's pat emulation? If that is the
case we should file a bug against qemu and try and fix it rather than
work around it...

It could still be something that the kernel can work around, Toshi did say:

There is a matter of how qemu emulates CPU features. There is no such
Intel CPU that supports PAT w/o MTRR. This is why the current code assumes this dependency.

Which is likely the trigger, we've send information about the cpu to him, and with that there's a chance for a pat fix.

Quoted 7 lines

He repeated our thought of running with 'nopat' while a fix is considered.

It may be some time before that happens, and I was going to test with the kernel patch dropped, and nopat in the qemu boot args. If that works, I'd rather run with that, and then revisit when (if) there's more changes upstream.

Reading the other thread, it looks like if MTRR is disabled, PAT needs to be disabled too. That sounds like a simple enough patch which is going upstream imminently so I think the preferred solution is to get that into our kernels and then drop my patch?

Cheers,

Richard

#13 lore raw

Re: [poky] [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

Quoted 42 lines · Richard Purdie
Quoted 33 lines · Bruce Ashfield
Quoted 15 lines · Richard Purdie
Quoted 8 lines · Bruce Ashfield

As another follow up. The thread can be summarized as "It doesn't look like it should have worked before, and qemu's pat emulation may be the issue'.

The suggestion is to run with 'nopat', which is what Richard originally
did.

Quoted 5 lines

So I'm going to prep a patch that drops the kernel patch, and leaves
nopat enabled on the qemu command line. That should get us put back
together in a semi-permanent way.

Quoted 5 lines

How sure are we this is a bug in QEMU's pat emulation? If that is the
case we should file a bug against qemu and try and fix it rather than
work around it...

It could still be something that the kernel can work around, Toshi did say:

There is a matter of how qemu emulates CPU features. There is no such
Intel CPU that supports PAT w/o MTRR. This is why the current code assumes this dependency.

Which is likely the trigger, we've send information about the cpu to him, and with that there's a chance for a pat fix.

Quoted 7 lines

He repeated our thought of running with 'nopat' while a fix is considered.

It may be some time before that happens, and I was going to test with the kernel patch dropped, and nopat in the qemu boot args. If that works, I'd rather run with that, and then revisit when (if) there's more changes upstream.

Quoted 4 lines

Reading the other thread, it looks like if MTRR is disabled, PAT needs to be disabled too. That sounds like a simple enough patch which is going upstream imminently so I think the preferred solution is to get that into our kernels and then drop my patch?

Yep. Looks like there's a patch to disable the pat based on the flags of the cpu model being proposed.

When that's out, I'll merge it, and then we'll drop this. So for now, we wait a bit longer.

Bruce

Quoted 5 lines

Cheers,

Richard

#14 lore raw

Re: [PATCH 1/1] poky: update qemu* to prefer 4.4 kernel

In reply to Richard Purdie
Quoted 20 lines · Richard Purdie
Quoted 7 lines · Richard Purdie
Quoted 6 lines · Richard Purdie

A quick look at upstream makes me wonder about:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/com
mi
t/?id=d9fe4fab11976e56b2e992980bf6ce948bdf02ac

Quoted 6 lines
Quoted 1 line

which changes the code in this area.

FWIW this doesn't help, the problem still exists after applying it.

Setting CONFIG_X86_PAT to is not set in the defconfig did "fix" it, unsurprisingly.

Also, in the last set of builds, despite booting with "nopat", we see:

https://autobuilder.yoctoproject.org/main/builders/nightly-x86-lsb/buil
ds/637/steps/Running%20Sanity%20Tests/logs/stdio

Quoted 7 lines

Central error: [ 9.298049] Failed to add WC MTRR for
[fd000000-fdffffff]; performance may suffer.

Why we only see this on the lsb run I don't know but that address looks like the one causing problems elsewhere with PAT. It could therefore look like this is related to uvesafb.

Interesting. And of course as I was mentioning in my update discussion, that I ran out of cycles to test the 3x kernel types with LSB .. murphy's law I suppose.

Bruce

Quoted 10 lines

Cheers,

Richard
--
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"