Linux 2.6.39-rc3

Let's hope the release cycle continues like this. I _like_ it when people really seem to follow the whole "big changes during the merge window" rules.

Quoted 3 lines · Linus Torvalds

Let's hope the release cycle continues like this. I _like_ it when people really seem to follow the whole "big changes during the merge window" rules.

Sorry for disturbing the silence, but radeon seems to have issues. I tested -rc3 (and after that -rc1 which also has the issue) on my Laptop and it just reboots after (or while?) GFX initialization. The last lines of dmesg are:

 Freeing unused kernel memory: 624k freed
 Write protecting the kernel read-only data: 8192k
 Freeing unused kernel memory: 1456k freed
 Freeing unused kernel memory: 16k freed
 udev: starting version 151
 udevd (62): /proc/62/oom_adj is deprecated, please use /proc/62/oom_score_adj instead. [drm] Initialized drm 1.1.0 20060810
 [drm] radeon defaulting to kernel modesetting.
 [drm] radeon kernel modesetting enabled.
 radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
 [drm] initializing kernel modesetting (RS880 0x1002:0x9712).
 [drm] register mmio base: 0xD6400000
 [drm] register mmio size: 65536
 ATOM BIOS: HP_TAG
 radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used) radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF [drm] Detected VRAM RAM=320M, BAR=256M
 [drm] RAM width 32bits DDR
 [TTM] Zone  kernel: Available graphics memory: 1896512 kiB.
 usb 7-2: new full speed USB device number 2 using ohci_hcd
 [TTM] Initializing pool allocator.
 [drm] radeon: 320M of VRAM memory ready
 [drm] radeon: 512M of GTT memory ready.
 [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
 [drm] Driver supports precise vblank timestamp query.
 [drm] radeon: irq initialized.
 [drm] GART: num cpu pages 131072, num gpu pages 131072
 [drm] Loading RS780 Microcode
 radeon 0000:01:05.0: WB enabled
 [drm] ring test succeeded in 1 usecs
 [drm] radeon: ib pool ready.

The card is a Radeon Mobility 4200:

01:05.0 VGA compatible controller: ATI Technologies Inc M880G [Mobility Radeon HD 4200]
       Subsystem: Hewlett-Packard Company Device 307e
       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
       Latency: 0, Cache Line Size: 64 bytes
       Interrupt: pin A routed to IRQ 18
       Region 0: Memory at c0000000 (32-bit, prefetchable) [size=256M]
       Region 1: I/O ports at 6000 [size=256]
       Region 2: Memory at d6400000 (32-bit, non-prefetchable) [size=64K]
       Region 5: Memory at d6300000 (32-bit, non-prefetchable) [size=1M]
       Capabilities: [50] Power Management version 3
               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
       Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
               Address: 0000000000000000  Data: 0000
       Kernel driver in use: radeon
       Kernel modules: radeon

The problem does not happen with 2.6.38. I try to bisect this further down to a commit. Alex, please let me know if you need any further information.

       Joerg

Quoted 62 lines · Joerg Roedel

Quoted 3 lines · Linus Torvalds

Let's hope the release cycle continues like this. I _like_ it when people really seem to follow the whole "big changes during the merge window" rules.

Sorry for disturbing the silence, but radeon seems to have issues. I tested -rc3 (and after that -rc1 which also has the issue) on my Laptop and it just reboots after (or while?) GFX initialization. The last lines of dmesg are:

 Freeing unused kernel memory: 624k freed
 Write protecting the kernel read-only data: 8192k
 Freeing unused kernel memory: 1456k freed
 Freeing unused kernel memory: 16k freed
 udev: starting version 151
 udevd (62): /proc/62/oom_adj is deprecated, please use /proc/62/oom_score_adj instead. [drm] Initialized drm 1.1.0 20060810
 [drm] radeon defaulting to kernel modesetting.
 [drm] radeon kernel modesetting enabled.
 radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
 [drm] initializing kernel modesetting (RS880 0x1002:0x9712).
 [drm] register mmio base: 0xD6400000
 [drm] register mmio size: 65536
 ATOM BIOS: HP_TAG
 radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used) radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF [drm] Detected VRAM RAM=320M, BAR=256M
 [drm] RAM width 32bits DDR
 [TTM] Zone  kernel: Available graphics memory: 1896512 kiB.
 usb 7-2: new full speed USB device number 2 using ohci_hcd
 [TTM] Initializing pool allocator.
 [drm] radeon: 320M of VRAM memory ready
 [drm] radeon: 512M of GTT memory ready.
 [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
 [drm] Driver supports precise vblank timestamp query.
 [drm] radeon: irq initialized.
 [drm] GART: num cpu pages 131072, num gpu pages 131072
 [drm] Loading RS780 Microcode
 radeon 0000:01:05.0: WB enabled
 [drm] ring test succeeded in 1 usecs
 [drm] radeon: ib pool ready.

The card is a Radeon Mobility 4200:

01:05.0 VGA compatible controller: ATI Technologies Inc M880G [Mobility Radeon HD 4200]
       Subsystem: Hewlett-Packard Company Device 307e
       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
       Latency: 0, Cache Line Size: 64 bytes
       Interrupt: pin A routed to IRQ 18
       Region 0: Memory at c0000000 (32-bit, prefetchable) [size=256M]
       Region 1: I/O ports at 6000 [size=256]
       Region 2: Memory at d6400000 (32-bit, non-prefetchable) [size=64K]
       Region 5: Memory at d6300000 (32-bit, non-prefetchable) [size=1M]
       Capabilities: [50] Power Management version 3
               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
       Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
               Address: 0000000000000000  Data: 0000
       Kernel driver in use: radeon
       Kernel modules: radeon

Quoted 2 lines

The problem does not happen with 2.6.38. I try to bisect this further down to a commit. Alex, please let me know if you need any further information.

If you can bisect it, that would be great. Thanks,

Bisecting actually gave a very weird result. It points to

	d2137d5af4259f50c19addb8246a186c9ffac325

which is a merge-commit in the x86 tree. Even more weird is that this notebook is the only machine with these symptoms, all my other boxes are fine.
During the bisect I tested commits from Yinghai which were good. It seems like the problem appeared with the merge.

Quoted 5 lines

Quoted 3 lines

The problem does not happen with 2.6.38. I try to bisect this further down to a commit. Alex, please let me know if you need any further information.

If you can bisect it, that would be great. Thanks,

Bisecting actually gave a very weird result. It points to

	d2137d5af4259f50c19addb8246a186c9ffac325

which is a merge-commit in the x86 tree. Even more weird is that this notebook is the only machine with these symptoms, all my other boxes are fine.

During the bisect I tested commits from Yinghai which were good. It seems like the problem appeared with the merge.

Could you please send the before/after bootlog (in particular all memory init messages included) and your .config?

before: f005fe12b90c: x86-64: Move out cleanup higmap [_brk_end, _end) out of init_memory_mapping()
after: d2137d5af425: Merge branch 'linus' into x86/bootmem

I've Cc:-ed more people who might have an idea about it.

Quoted 7 lines · Ingo Molnar

Could you please send the before/after bootlog (in particular all memory init messages included) and your .config?

before: f005fe12b90c: x86-64: Move out cleanup higmap [_brk_end, _end) out of init_memory_mapping()
after: d2137d5af425: Merge branch 'linus' into x86/bootmem

I've Cc:-ed more people who might have an idea about it.

Okay, I have done some more bisecting and debugging today.

Quoted 11 lines · Joerg Roedel

Quoted 7 lines · Ingo Molnar

Could you please send the before/after bootlog (in particular all memory init messages included) and your .config?

before: f005fe12b90c: x86-64: Move out cleanup higmap [_brk_end, _end) out of init_memory_mapping()
after: d2137d5af425: Merge branch 'linus' into x86/bootmem

I've Cc:-ed more people who might have an idea about it.

Okay, I have done some more bisecting and debugging today.

First of all, *huge* thanks for this effort. At least we need to track down the bits that need to be reverted -- it is past rc3, and it's time to see what we should revert and tell the submitter to try again next cycle.

This looks to be the same issue as in bugzilla 33012:

	https://bugzilla.kernel.org/show_bug.cgi?id=33012

... so it would be good if we could keep the information in there.

First of all, I bisected between v2.6.37-rc2..f005fe12b90c which where only a couple of patches and merged v2.6.38-rc4 in at every step. There was no failure found.
Then I tried this again, but this time I merged v2.6.38-rc5 at every step and was successful. The bad commit in this branch turned out to be

	1a4a678b12c84db9ae5dce424e0e97f0559bb57c

which is related to memblock.

Then I tried to find out which change between 2.6.38-rc4 and 2.6.38-rc5 is needed to trigger the failure, so I used f005fe12b90c as a base, bisected between v2.6.38-rc4..v2.6.38-rc5 and merged every bisect step into the base and tested. Here the bad commit turned out to be

	e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20

which is related to gart. It turned out that the gart aperture on that box is on another position with these patches. Before it was as 0xa4000000 and now it is at 0xa0000000. It seems like this has something to do with the root-cause.

Reverting commit 1a4a678b12c84db9ae5dce424e0e97f0559bb57c fixes the problem btw. and booting with iommu=soft also works, but I have no idea yet why the aperture at that address is a problem (with the patch reverted the aperture lands at 0x80000000).

Quoted 23 lines · Joerg Roedel

First of all, I bisected between v2.6.37-rc2..f005fe12b90c which where only a couple of patches and merged v2.6.38-rc4 in at every step. There was no failure found.
Then I tried this again, but this time I merged v2.6.38-rc5 at every step and was successful. The bad commit in this branch turned out to be

	1a4a678b12c84db9ae5dce424e0e97f0559bb57c

which is related to memblock.

Then I tried to find out which change between 2.6.38-rc4 and 2.6.38-rc5 is needed to trigger the failure, so I used f005fe12b90c as a base, bisected between v2.6.38-rc4..v2.6.38-rc5 and merged every bisect step into the base and tested. Here the bad commit turned out to be

	e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20

which is related to gart. It turned out that the gart aperture on that box is on another position with these patches. Before it was as 0xa4000000 and now it is at 0xa0000000. It seems like this has something to do with the root-cause.

Quoted 5 lines

Reverting commit 1a4a678b12c84db9ae5dce424e0e97f0559bb57c fixes the problem btw. and booting with iommu=soft also works, but I have no idea yet why the aperture at that address is a problem (with the patch reverted the aperture lands at 0x80000000).

Does reverting e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20 solve the problem for you?

1a4a678b12c84db9ae5dce424e0e97f0559bb57c is a memory-allocation-order patch, which have a nasty tendency to unmask bugs elsewhere in the kernel. However, e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20 looks positively strange (and it doesn't exactly help that the description is written in Yinghai-ese and is therefore nearly impossible to decode, never mind tell if it is remotely correct.)

On Wed, Apr 13, 2011 at 08:46:09AM +0200, Ingo Molnar wrote:
First of all, I bisected between v2.6.37-rc2..f005fe12b90c which where only a couple of patches and merged v2.6.38-rc4 in at every step. There was no failure found.
Then I tried this again, but this time I merged v2.6.38-rc5 at every step and was successful. The bad commit in this branch turned out to be

	1a4a678b12c84db9ae5dce424e0e97f0559bb57c

which is related to memblock.

Then I tried to find out which change between 2.6.38-rc4 and 2.6.38-rc5 is needed to trigger the failure, so I used f005fe12b90c as a base, bisected between v2.6.38-rc4..v2.6.38-rc5 and merged every bisect step into the base and tested. Here the bad commit turned out to be

	e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20

which is related to gart. It turned out that the gart aperture on that box is on another position with these patches. Before it was as 0xa4000000 and now it is at 0xa0000000. It seems like this has something to do with the root-cause.

Reverting commit 1a4a678b12c84db9ae5dce424e0e97f0559bb57c fixes the problem btw. and booting with iommu=soft also works, but I have no idea yet why the aperture at that address is a problem (with the patch reverted the aperture lands at 0x80000000).

I have put some debug-data online. There is my .config and two dmesg-files for good (==2.6.39-rc3 + revert) and bad (==2.6.39-rc3) I also created these dmesg-files again with memblock=debug, maybe that helps to find the problem. The files are at

	http://www.8bytes.org/~joro/debug/

thanks for the bisecting...

so those two patches uncover some problems.

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000

so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

but your radeon does use [0xa0000000, 0xbfffffff)

Quoted 14 lines · Yinghai Lu

thanks for the bisecting...

so those two patches uncover some problems.

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000

Quoted 1 line

so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

It is actually beyond 4GB on that machine, this value read here is from the previous kernel-boot. The BIOS does not reset these values on a reboot.

Quoted 1 line

but your radeon does use [0xa0000000, 0xbfffffff)

Yes, I suspected that too (and spent a few hours reading radeon code), but then I talked the Alex Deucher and he explained that these addresses which the driver prints for GTT and VRAM are in the GPU address space and do not refer to system ram. So this shouldn't be the problem.

Quoted 3 lines · Yinghai Lu

can you try following change ? it will push gart to 0x80000000

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 10 lines

arch/x86/kernel/aperture_64.c +1 -1

index 86d1ad4..3b6a9d5 100644
@@ -83,7 +83,7 @@ static u32 __init allocate_aperture(void)
        * so don't use 512M below as gart iommu, leave the space for kernel
        * code for safe
        */
-       addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+       addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

What are all the magic numbers, and why would 0x80000000 be special?

Why don't we write code that just works?

Or absent a "just works" set of patches, why don't we revert to code that has years of testing?

This kind of "I broke things, so now I will jiggle things randomly until they unbreak" is not acceptable.

Either explain why that fixes a real BUG (and why the magic constants need to be what they are), or just revert the patch that caused the problem, and go back to the allocation patters that have years of experience.

Guys, we've had this discussion before, in PCI allocation. We don't do this. We tried switching the PCI region allocations to top-down, and IT WAS A FAILURE. We reverted it to what we had years of testing with.

Don't just make random changes. There really are only two acceptable models of development: "think and analyze" or "years and years of testing on thousands of machines". Those two really do work.

Quoted 2 lines

What are all the magic numbers, and why would 0x80000000 be special?

that is the old value when kernel was doing bottom-up bootmem allocation.

Quoted 17 lines

Why don't we write code that just works?

Or absent a "just works" set of patches, why don't we revert to code that has years of testing?

This kind of "I broke things, so now I will jiggle things randomly until they unbreak" is not acceptable.

Either explain why that fixes a real BUG (and why the magic constants need to be what they are), or just revert the patch that caused the problem, and go back to the allocation patters that have years of experience.

Guys, we've had this discussion before, in PCI allocation. We don't do this. We tried switching the PCI region allocations to top-down, and IT WAS A FAILURE. We reverted it to what we had years of testing with.

Quoted 3 lines

Don't just make random changes. There really are only two acceptable models of development: "think and analyze" or "years and years of testing on thousands of machines". Those two really do work.

We did do the analyzing, and only difference seems to be:

good one is using 0x80000000
and bad one is using 0xa0000000.

We try to figure out if it needs low address and it happen to work because kernel was doing bottom up allocation.

Quoted 4 lines · Yinghai Lu

Quoted 2 lines

What are all the magic numbers, and why would 0x80000000 be special?

that is the old value when kernel was doing bottom-up bootmem allocation.

I understand, BUT THAT IS STILL A TOTALLY MAGIC NUMBER!

It makes it come out the same ON THAT ONE MACHINE. So no, it's not "the old value". It's a random value that gets the old value in one specific case.

Quoted 18 lines

Quoted 18 lines

Why don't we write code that just works?

Or absent a "just works" set of patches, why don't we revert to code that has years of testing?

This kind of "I broke things, so now I will jiggle things randomly until they unbreak" is not acceptable.

Either explain why that fixes a real BUG (and why the magic constants need to be what they are), or just revert the patch that caused the problem, and go back to the allocation patters that have years of experience.

Guys, we've had this discussion before, in PCI allocation. We don't do this. We tried switching the PCI region allocations to top-down, and IT WAS A FAILURE. We reverted it to what we had years of testing with.

Quoted 5 lines

Quoted 3 lines

Don't just make random changes. There really are only two acceptable models of development: "think and analyze" or "years and years of testing on thousands of machines". Those two really do work.

We did do the analyzing, and only difference seems to be:

No.

Yinghai, we have had this discussion before, and dammit, you need to understand the difference between "understanding the problem" and "put in random values until it works on one machine".

There was absolutely _zero_ analysis done. You do not actually understand WHY the numbers matter. You just look at two random numbers, and one works, the other does not. That's not "analyzing". That's just "random number games".

If you cannot see and understand the difference between an actual analytical solution where you _understand_ what the code is doing and why, and "random numbers that happen to work on one machine", I don't know what to tell you.

Quoted 5 lines

good one is using 0x80000000
and bad one is using 0xa0000000.

We try to figure out if it needs low address and it happen to work because kernel was doing bottom up allocation.

No.

Let me repeat my point one more time.

You have TWO choices. Not more, not less:

- choice #1: go back to the old allocation model. It's tested. It
doesn't regress. Admittedly we may not know exactly _why_ it works, and it might not work on all machines, but it doesn't cause regressions (ie the machines it doesn't work on it _never_ worked on).

And this doesn't mean "old value for that _one_ machine". It means
"old value for _every_ machine". So it means we revert the whole bottom-down thing entirely. Not just "change one random number so that the totally different allocation pattern happens to give the same result on one particular machine".

Quite frankly, I don't see the point of doing top-to-bottom anyway,
so I think we should do this regardless. Just revert the whole "allocate from top". It didn't work for PCI, it's not working for this case either. Stop doing it.

- Choice #2: understand exactly _what_ goes wrong, and fix it
analytically (ie by _understanding_ the problem, and being able to solve it exactly, and in a way you can argue about without having to resort to "magic happens").

Now, the whole analytic approach (aka "computer sciency" approach), where you can actually think about the problem without having any pesky "reality" impact the solution is obviously the one we tend to prefer. Sadly, it's seldom the one we can use in reality when it comes to things like resource allocation, since we end up starting off with often buggy approximations of what the actual hardware is all about (ie broken firmware tables).

So I'd love to know exactly why one random number works, and why another one doesn't. But as long as we do _not_ know the "Why" of it, we will have to revert.

It really is that simple. It's _always_ that simple.

So the numbers shouldn't be "magic", they should have real explanations. And in the absense of real explanation, the model that works is "this is what we've always done". Including, very much, the whole allocation order. Not just one random number on one random machine.

- Choice #2: understand exactly _what_ goes wrong, and fix it
analytically (ie by _understanding_ the problem, and being able to solve it exactly, and in a way you can argue about without having to resort to "magic happens").

Now, the whole analytic approach (aka "computer sciency" approach), where you can actually think about the problem without having any pesky "reality" impact the solution is obviously the one we tend to prefer. Sadly, it's seldom the one we can use in reality when it comes to things like resource allocation, since we end up starting off with often buggy approximations of what the actual hardware is all about (ie broken firmware tables).

So I'd love to know exactly why one random number works, and why another one doesn't. But as long as we do _not_ know the "Why" of it, we will have to revert.

Yes.  However, even if we *do* revert (and the time is running short on not reverting) I would like to understand this particular one, simply because I think it may very well be a problem that is manifesting itself in other ways on other systems.

The other thing that this has uncovered is that we already have a bunch of complete b*llsh*t magic numbers in this

Quoted 5 lines · H. Peter Anvin

Yes.  However, even if we *do* revert (and the time is running short on not reverting) I would like to understand this particular one, simply because I think it may very well be a problem that is manifesting itself in other ways on other systems.

Quoted 1 line

of complete b*llsh*t magic numbers in this

Quoted 6 lines · Linus Torvalds

Quoted 5 lines · H. Peter Anvin

Yes.  However, even if we *do* revert (and the time is running short on not reverting) I would like to understand this particular one, simply because I think it may very well be a problem that is manifesting itself in other ways on other systems.

sorry, fingerfart. Anyway, I agree 100%.

we definitely want to also understand the reason for things not
working, even if we do revert..

Hello,

Quoted 9 lines · Linus Torvalds

Quoted 6 lines · Linus Torvalds

Quoted 5 lines · H. Peter Anvin

Yes.  However, even if we *do* revert (and the time is running short on not reverting) I would like to understand this particular one, simply because I think it may very well be a problem that is manifesting itself in other ways on other systems.

Quoted 4 lines

sorry, fingerfart. Anyway, I agree 100%.

we definitely want to also understand the reason for things not
working, even if we do revert..

There were (and still are) places where memblock callers implemented ad-hoc top-down allocation by stepping down start limit until allocation succeeds. Several of them have been removed since top-down became the default behavior, so simply reverting the commit is likely to cause subtle issues. Maybe the best approach is introducing @topdown parameter and use it selectively for pure memory allocations.

Quoted 6 lines · Linus Torvalds

Quoted 5 lines · H. Peter Anvin

Yes.  However, even if we *do* revert (and the time is running short on not reverting) I would like to understand this particular one, simply because I think it may very well be a problem that is manifesting itself in other ways on other systems.

sorry, fingerfart. Anyway, I agree 100%.

we definitely want to also understand the reason for things not
working, even if we do revert..

we definitely want to also understand the reason for things not
working, even if we do revert..

From aaacff8db50b6ed4345e337ecbe53e505699c7e5 Mon Sep 17 00:00:00 2001

Quoted 2 lines · Linus Torvalds

we definitely want to also understand the reason for things not
working, even if we do revert..

Okay, here it is.

After experimenting with different configurations for the north-bridge it turned out that a GART related MCE fires at the time the machine reboots. BIOSes configure the machine to sync-flood in that case which causes a reboot.

After decoding the MCE it turned out to be a GART TBL Wlk Error. Such errors can happen if devices (speculativly) access GART ranges mapped invalid. The AMD BKDG for Fam10h CPUs recommends to disable these errors at all. But unfortunatly some BIOSes (including the one on my laptop) forget to do this.

Below is a patch which disables these errors if the BIOS didn't do it. It fixes the problem on my site.

Ok, but how did the allocation changes start triggering this error in v2.6.39-rc1? There must still be some layout specific thing here, right? Do we understand the details of that as well?

Quoted 3 lines · Ingo Molnar

Ok, but how did the allocation changes start triggering this error in v2.6.39-rc1? There must still be some layout specific thing here, right? Do we understand the details of that as well?

No, I must admit that I lack enough knowledge about the GPU hardware to make an guess how this tanslation-request happened. All I can tell is the address that was reported in the MCE, it is 0xa0001000 (==the second page of the GART aperture).

Maybe Alex can help here. Alex, may it be possible that the GPU generates DMA requests in the GTT area before the GTT is activated (or the activation is completed)? Or can you imagine any other reason?

Ok, but how did the allocation changes start triggering this error in v2.6.39-rc1? There must still be some layout specific thing here, right? Do we understand the details of that as well?

Quoted 3 lines · Ingo Molnar

Ok, but how did the allocation changes start triggering this error in v2.6.39-rc1? There must still be some layout specific thing here, right? Do we understand the details of that as well?

Well, thinking again about this, the GPU likely generated this DMA request before too (which has an address in the range configured for the GTT on the card), but nobody noticed because they just hit main memory. And with the allocation changes in 39-rc1 the GART aperture started to be on the same address as the GTT (in their respective address spaces) so that the DMA request hit the GART. This caused the MCE and the sync-flood.
The open question is why the GPU generates a DMA request with an address that is configured as the GTT base (+1 page) on the card.

       Joerg

Do you also got the write if you load radeon with radeon.no_wb=1 ? I think at this address it's the wb page, or maybe the cp as wb likely take only one page

Quoted 3 lines · Jerome Glisse

Do you also got the write if you load radeon with radeon.no_wb=1 ? I think at this address it's the wb page, or maybe the cp as wb likely take only one page

radeon.no_wb=1 makes no difference. The box still reboots.

       Joerg

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Quoted 3 lines · Jerome Glisse

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Okay, I found the place in source that triggers this. It happens in the function r600_ib_test. The interesting thing is that not the ib-command itself is responsible but the fence that is emitted afterwards (proved by removing the fence command, where the problem went away). I don't know enough about the command semantics to make a guess what goes wrong there. But maybe you GPU folks have an idea?

       Joerg

Quoted 3 lines · Jerome Glisse

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Okay, I found the place in source that triggers this. It happens in the function r600_ib_test. The interesting thing is that not the ib-command itself is responsible but the fence that is emitted afterwards (proved by removing the fence command, where the problem went away). I don't know enough about the command semantics to make a guess what goes wrong there. But maybe you GPU folks have an idea?

       Joerg

_______________________________________________
dri-devel mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/dri-devel

Quoted 6 lines · Joerg Roedel

Quoted 3 lines · Jerome Glisse

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Quoted 7 lines

Okay, I found the place in source that triggers this. It happens in the function r600_ib_test. The interesting thing is that not the ib-command itself is responsible but the fence that is emitted afterwards (proved by removing the fence command, where the problem went away). I don't know enough about the command semantics to make a guess what goes wrong there. But maybe you GPU folks have an idea?

I can't think of anything off hand.  It might be worth disabling the call to r600_ib_test() in r600_init() and then seeing if you get any errors when the fences are used later on when X starts or just at that point in the module load sequence.  What's odd is that when you tested radeon.no_wb=1 you got the same behavior as that disables shadowing of fence writes to gpu gart mem, so it wouldn't be writing to memory in that case.

Alex

Quoted 23 lines · Alex Deucher

Quoted 6 lines · Joerg Roedel

Quoted 3 lines · Jerome Glisse

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Quoted 7 lines

Okay, I found the place in source that triggers this. It happens in the function r600_ib_test. The interesting thing is that not the ib-command itself is responsible but the fence that is emitted afterwards (proved by removing the fence command, where the problem went away). I don't know enough about the command semantics to make a guess what goes wrong there. But maybe you GPU folks have an idea?

I can't think of anything off hand.  It might be worth disabling the call to r600_ib_test() in r600_init() and then seeing if you get any errors when the fences are used later on when X starts or just at that point in the module load sequence.  What's odd is that when you tested radeon.no_wb=1 you got the same behavior as that disables shadowing of fence writes to gpu gart mem, so it wouldn't be writing to memory in that case.

Quoted 2 lines

Alex

It might be the irq ring write that is faulty.

Cheers,
Jerome

Quoted 24 lines · Jerome Glisse

Quoted 23 lines · Alex Deucher

Quoted 6 lines · Joerg Roedel

Quoted 3 lines · Jerome Glisse

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Quoted 7 lines

Okay, I found the place in source that triggers this. It happens in the function r600_ib_test. The interesting thing is that not the ib-command itself is responsible but the fence that is emitted afterwards (proved by removing the fence command, where the problem went away). I don't know enough about the command semantics to make a guess what goes wrong there. But maybe you GPU folks have an idea?

I can't think of anything off hand.  It might be worth disabling the call to r600_ib_test() in r600_init() and then seeing if you get any errors when the fences are used later on when X starts or just at that point in the module load sequence.  What's odd is that when you tested radeon.no_wb=1 you got the same behavior as that disables shadowing of fence writes to gpu gart mem, so it wouldn't be writing to memory in that case.

Quoted 4 lines

Quoted 2 lines

Alex

It might be the irq ring write that is faulty.

That's disabled with no_wb=1 as well.

Alex

Quoted 28 lines · Alex Deucher

Quoted 27 lines · Jerome Glisse

Quoted 25 lines · Alex Deucher

Quoted 6 lines · Joerg Roedel

Quoted 3 lines · Jerome Glisse

If you want to go the printk way you can add printk before each test ring_test, ib_test in r600.c this 2 functions are the own that might trigger the first GPU gart activities.

Quoted 7 lines

Okay, I found the place in source that triggers this. It happens in the function r600_ib_test. The interesting thing is that not the ib-command itself is responsible but the fence that is emitted afterwards (proved by removing the fence command, where the problem went away). I don't know enough about the command semantics to make a guess what goes wrong there. But maybe you GPU folks have an idea?

I can't think of anything off hand.  It might be worth disabling the call to r600_ib_test() in r600_init() and then seeing if you get any errors when the fences are used later on when X starts or just at that point in the module load sequence.  What's odd is that when you tested radeon.no_wb=1 you got the same behavior as that disables shadowing of fence writes to gpu gart mem, so it wouldn't be writing to memory in that case.

Alex

Quoted 6 lines

Quoted 1 line

It might be the irq ring write that is faulty.

That's disabled with no_wb=1 as well.

Alex

I mean the irq interrupt ring, i don't see this being disabled when no_wb=1

Cheers,
Jerome

Quoted 2 lines · Linus Torvalds

we definitely want to also understand the reason for things not
working, even if we do revert..

Okay, here it is.

After experimenting with different configurations for the north-bridge it turned out that a GART related MCE fires at the time the machine reboots. BIOSes configure the machine to sync-flood in that case which causes a reboot.

After decoding the MCE it turned out to be a GART TBL Wlk Error. Such errors can happen if devices (speculativly) access GART ranges mapped invalid. The AMD BKDG for Fam10h CPUs recommends to disable these errors at all. But unfortunatly some BIOSes (including the one on my laptop) forget to do this.

Below is a patch which disables these errors if the BIOS didn't do it. It fixes the problem on my site.

Alexandre, can you try this patch on your machine too, please?

Regards,

Joerg

From aaacff8db50b6ed4345e337ecbe53e505699c7e5 Mon Sep 17 00:00:00 2001
From: Joerg Roedel <[email protected]>
Date: Fri, 15 Apr 2011 14:47:40 +0200
Subject: [PATCH] x86/amd: Disable GartTlbWlkErr when BIOS forgets it

This patch disables GartTlbWlk errors on AMD Fam10h CPUs if the BIOS forgets to do is (or is just too old). Letting these errors enabled can cause a sync-flood on the CPU causing a reboot.

This patch is the fix for

	https://bugzilla.kernel.org/show_bug.cgi?id=33012

on my machine.

Signed-off-by Joerg Roedel <[email protected]>

What about tagging this patch for stable/longterm releases?

Potentially there are other cases where certain combinations of hardware(GPUs)/drivers/whatsoever might trigger a GartTlbWlkErr. If the BIOS doesn't follow the BKDG recommendation to mask these errors, the system will hang/reboot. Thus I think having this quirk in .32 and .38 (at least) is useful.

Quoted 2 lines · Linus Torvalds

we definitely want to also understand the reason for things not
working, even if we do revert..

Okay, here it is.

After experimenting with different configurations for the north-bridge it turned out that a GART related MCE fires at the time the machine reboots. BIOSes configure the machine to sync-flood in that case which causes a reboot.

After decoding the MCE it turned out to be a GART TBL Wlk Error. Such errors can happen if devices (speculativly) access GART ranges mapped invalid. The AMD BKDG for Fam10h CPUs recommends to disable these errors at all. But unfortunatly some BIOSes (including the one on my laptop) forget to do this.

Below is a patch which disables these errors if the BIOS didn't do it. It fixes the problem on my site.

Alexandre, can you try this patch on your machine too, please?

Regards,

Joerg

From aaacff8db50b6ed4345e337ecbe53e505699c7e5 Mon Sep 17 00:00:00 2001
From: Joerg Roedel <[email protected]>
Date: Fri, 15 Apr 2011 14:47:40 +0200
Subject: [PATCH] x86/amd: Disable GartTlbWlkErr when BIOS forgets it

This patch disables GartTlbWlk errors on AMD Fam10h CPUs if the BIOS forgets to do is (or is just too old). Letting these errors enabled can cause a sync-flood on the CPU causing a reboot.

This patch is the fix for

	https://bugzilla.kernel.org/show_bug.cgi?id=33012

on my machine.

Signed-off-by Joerg Roedel <[email protected]>

---
 arch/x86/include/asm/msr-index.h |    4 ++++
 arch/x86/kernel/cpu/amd.c        |   19 +++++++++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)

arch/x86/include/asm/msr-index.h +4 -0

index fd5a1f3..3cce714 100644
@@ -96,11 +96,15 @@
 #define MSR_IA32_MC0_ADDR		0x00000402
 #define MSR_IA32_MC0_MISC		0x00000403
 
+#define MSR_AMD64_MC0_MASK		0xc0010044
+
 #define MSR_IA32_MCx_CTL(x)		(MSR_IA32_MC0_CTL + 4*(x))
 #define MSR_IA32_MCx_STATUS(x)		(MSR_IA32_MC0_STATUS + 4*(x))
 #define MSR_IA32_MCx_ADDR(x)		(MSR_IA32_MC0_ADDR + 4*(x))
 #define MSR_IA32_MCx_MISC(x)		(MSR_IA32_MC0_MISC + 4*(x))
 
+#define MSR_AMD64_MCx_MASK(x)		(MSR_AMD64_MC0_MASK + (x))
+
 /* These are consecutive and not in the normal 4er MCE bank block */
 #define MSR_IA32_MC0_CTL2		0x00000280
 #define MSR_IA32_MCx_CTL2(x)		(MSR_IA32_MC0_CTL2 + (x))

arch/x86/kernel/cpu/amd.c +12 -0

index 3ecece0..3532d3b 100644
@@ -615,6 +615,25 @@ static void __cpuinit init_amd(struct cpuinfo_x86 *c)
 	/* As a rule processors have APIC timer running in deep C states */
 	if (c->x86 >= 0xf && !cpu_has_amd_erratum(amd_erratum_400))
 		set_cpu_cap(c, X86_FEATURE_ARAT);
+
+	/*
+	 * Disable GART TLB Walk Errors on Fam10h. We do this here
+	 * because this is always needed when GART is enabled, even in a
+	 * kernel which has no MCE support built in.
+	 */
+	if (c->x86 == 0x10) {
+		/*
+		 * BIOS should disable GartTlbWlk Errors themself. If
+		 * it doesn't do it here as suggested by the BKDG.
+		 *
+		 * Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=33012

Ok, I'll test it today. Should I apply it on a clean rc3 without any of the other patches?

BTW, may I suggest adding the info under bug 33012 in kernel bugzilla? This could be useful in the future.

Quoted 2 lines · Alexandre Demers

Ok, I'll test it today. Should I apply it on a clean rc3 without any of the other patches?

Yes, apply it just on -rc3 without any other patch.

Quoted 2 lines

BTW, may I suggest adding the info under bug 33012 in kernel bugzilla? This could be useful in the future.

Cool, thanks

Joerg

Quoted 5 lines · Joerg Roedel

Quoted 2 lines · Alexandre Demers

Ok, I'll test it today. Should I apply it on a clean rc3 without any of the other patches?

Yes, apply it just on -rc3 without any other patch.

Quoted 6 lines

Quoted 2 lines

BTW, may I suggest adding the info under bug 33012 in kernel bugzilla? This could be useful in the future.

Cool, thanks

Joerg

The patch was applied and tested. It looks fine, I'm able to boot without problem.

Joerg, mind submitting it with a changelog that includes everything we learned about this bug and all the Tested-by's in place?

Is anyone of the opinion that we should try to revert the allocation order/alignment changes in addition to this fix?

Quoted 4 lines · Ingo Molnar

Joerg, mind submitting it with a changelog that includes everything we learned about this bug and all the Tested-by's in place?

Quoted 2 lines

Is anyone of the opinion that we should try to revert the allocation order/alignment changes in addition to this fix?

We should figure out what is written to 0xa0001000 (main memory) by GPU before internal GART is setup.

Joerg,
can you insert some dump code in the drm/radon code to find out which function cause the problem?

Quoted 4 lines · Ingo Molnar

Joerg, mind submitting it with a changelog that includes everything we learned about this bug and all the Tested-by's in place?

Quoted 2 lines

Is anyone of the opinion that we should try to revert the allocation order/alignment changes in addition to this fix?

We should figure out what is written to 0xa0001000 (main memory) by GPU before internal GART is setup.

Joerg,
can you insert some dump code in the drm/radon code to find out which function cause the problem?

Quoted 11 lines · Alexandre Demers

Quoted 10 lines · Joerg Roedel

Quoted 2 lines · Alexandre Demers

Ok, I'll test it today. Should I apply it on a clean rc3 without any of the other patches?

Yes, apply it just on -rc3 without any other patch.

Quoted 2 lines

BTW, may I suggest adding the info under bug 33012 in kernel bugzilla? This could be useful in the future.

Cool, thanks

Quoted 3 lines

Quoted 1 line

Joerg

The patch was applied and tested. It looks fine, I'm able to boot without problem.

Joerg, mind submitting it with a changelog that includes everything we learned about this bug and all the Tested-by's in place?

Quoted 18 lines · Ingo Molnar

Quoted 14 lines · Alexandre Demers

Quoted 11 lines · Joerg Roedel

Quoted 2 lines · Alexandre Demers

Ok, I'll test it today. Should I apply it on a clean rc3 without any of the other patches?

Yes, apply it just on -rc3 without any other patch.

Quoted 2 lines

BTW, may I suggest adding the info under bug 33012 in kernel bugzilla? This could be useful in the future.

Cool, thanks

Joerg

The patch was applied and tested. It looks fine, I'm able to boot without problem.

Quoted 2 lines

Joerg, mind submitting it with a changelog that includes everything we learned about this bug and all the Tested-by's in place?

Looks like I am too late, it is already applied. But the changelog contains a link to the korg-bugzilla which has all information too. So the information is not lost.

Quoted 5 lines · H. Peter Anvin

Yes.  However, even if we *do* revert (and the time is running short on not reverting) I would like to understand this particular one, simply because I think it may very well be a problem that is manifesting itself in other ways on other systems.

Quoted 1 line

of complete b*llsh*t magic numbers in this

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 2 lines · Yinghai Lu

arch/x86/kernel/aperture_64.c +1 -1

-	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

Btw, while looking at this code I wondered why the 512M goal is enforced by the alignment. Start could be set to 512M instead and the alignment can be aper_size as it should. Any reason for such a big alignment?

P.S.: The box is still in the office, I will try this debug-patch
tomorrow.

Quoted 8 lines · Joerg Roedel

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 2 lines · Yinghai Lu

arch/x86/kernel/aperture_64.c +1 -1

-	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

Btw, while looking at this code I wondered why the 512M goal is enforced by the alignment. Start could be set to 512M instead and the alignment can be aper_size as it should. Any reason for such a big alignment?

when using bootmem, try to use big alignment (512M ), so we could avoid take ram range below 512M.

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 2 lines · Yinghai Lu

arch/x86/kernel/aperture_64.c +1 -1

-	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

Btw, while looking at this code I wondered why the 512M goal is enforced by the alignment. Start could be set to 512M instead and the alignment can be aper_size as it should. Any reason for such a big alignment?

Joerg

P.S.: The box is still in the office, I will try this debug-patch
tomorrow.

Quoted 8 lines · Joerg Roedel

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 2 lines · Yinghai Lu

arch/x86/kernel/aperture_64.c +1 -1

-	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

Btw, while looking at this code I wondered why the 512M goal is enforced by the alignment. Start could be set to 512M instead and the alignment can be aper_size as it should. Any reason for such a big alignment?

Quoted 4 lines

Joerg

P.S.: The box is still in the office, I will try this debug-patch
tomorrow.

The only reason that I can think of is that the aperture itself can be huge, and perhaps 512 MiB is the biggest such known.

Quoted 14 lines · H. Peter Anvin

Quoted 12 lines · Joerg Roedel

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 2 lines · Yinghai Lu

arch/x86/kernel/aperture_64.c +1 -1

-	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

Btw, while looking at this code I wondered why the 512M goal is enforced by the alignment. Start could be set to 512M instead and the alignment can be aper_size as it should. Any reason for such a big alignment?

Joerg

P.S.: The box is still in the office, I will try this debug-patch
tomorrow.

Quoted 2 lines

The only reason that I can think of is that the aperture itself can be huge, and perhaps 512 MiB is the biggest such known.

Well, that would work as well by just using aper_size as alignment, the aperture needs to be aligned on its size anyway. This code only runs when Linux allocates the aperture itself and if I am mistaken is uses always 64MB when doing this.

Quoted 15 lines · Joerg Roedel

Quoted 14 lines · H. Peter Anvin

Quoted 12 lines · Joerg Roedel

arch/x86/kernel/aperture_64.c static u32 __init allocate_aperture(void) · line 86

Quoted 2 lines · Yinghai Lu

arch/x86/kernel/aperture_64.c +1 -1

-	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<20);
+	addr = memblock_find_in_range(0, 1ULL<<32, aper_size, 512ULL<<21);

Btw, while looking at this code I wondered why the 512M goal is enforced by the alignment. Start could be set to 512M instead and the alignment can be aper_size as it should. Any reason for such a big alignment?

Joerg

P.S.: The box is still in the office, I will try this debug-patch
tomorrow.

Quoted 7 lines

Quoted 2 lines

The only reason that I can think of is that the aperture itself can be huge, and perhaps 512 MiB is the biggest such known.

Well, that would work as well by just using aper_size as alignment, the aperture needs to be aligned on its size anyway. This code only runs when Linux allocates the aperture itself and if I am mistaken is uses always 64MB when doing this.

Yes, I would agree with that. The sane thing would be to set the base to whatever address needs to be guarded against (WHICH SHOULD BE MOTIVATED), and use aper_size as alignment, *unless* we are only using the initial portion of a much larger hardware structure that needs natural alignment (which isn't clear to me, I do know we sometimes use only a fraction of the GART, but that doesn't mean we need to naturally-align the entire thing, nor that 512 MiB is sufficient to do so.)

Quoted 28 lines · Joerg Roedel

On Wed, Apr 13, 2011 at 08:46:09AM +0200, Ingo Molnar wrote:
First of all, I bisected between v2.6.37-rc2..f005fe12b90c which where only a couple of patches and merged v2.6.38-rc4 in at every step. There was no failure found.
Then I tried this again, but this time I merged v2.6.38-rc5 at every step and was successful. The bad commit in this branch turned out to be

      1a4a678b12c84db9ae5dce424e0e97f0559bb57c

which is related to memblock.

Then I tried to find out which change between 2.6.38-rc4 and 2.6.38-rc5 is needed to trigger the failure, so I used f005fe12b90c as a base, bisected between v2.6.38-rc4..v2.6.38-rc5 and merged every bisect step into the base and tested. Here the bad commit turned out to be

      e6d2e2b2b1e1455df16d68a78f4a3874c7b3ad20

which is related to gart. It turned out that the gart aperture on that box is on another position with these patches. Before it was as 0xa4000000 and now it is at 0xa0000000. It seems like this has something to do with the root-cause.

Reverting commit 1a4a678b12c84db9ae5dce424e0e97f0559bb57c fixes the problem btw. and booting with iommu=soft also works, but I have no idea yet why the aperture at that address is a problem (with the patch reverted the aperture lands at 0x80000000).

Quoted 6 lines

I have put some debug-data online. There is my .config and two dmesg-files for good (==2.6.39-rc3 + revert) and bad (==2.6.39-rc3) I also created these dmesg-files again with memblock=debug, maybe that helps to find the problem. The files are at

      http://www.8bytes.org/~joro/debug/

thanks for the bisecting...

so those two patches uncover some problems.

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64 [    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000

so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

but your radeon does use [0xa0000000, 0xbfffffff)

[    4.281993] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used) [    4.290672] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF [    4.298550] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.309857] [drm] RAM width 32bits DDR
[    4.313748] [TTM] Zone  kernel: Available graphics memory: 1896524 kiB.
[    4.320379] [TTM] Initializing pool allocator.
[    4.324948] [drm] radeon: 320M of VRAM memory ready
[    4.329832] [drm] radeon: 512M of GTT memory ready.

and the one seems working:

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0x80000000-0x83ffffff]       aperture64 [    0.000000] Mapping aperture over 65536 KB of RAM @ 80000000
[    0.000000]     memblock_x86_reserve_range: [0xacb6bdc0-0xacb6bddf]          BOOTMEM

will use different position...

[    4.250159] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used) [    4.258830] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF [    4.266742] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.271549] [drm] RAM width 32bits DDR
[    4.275435] [TTM] Zone  kernel: Available graphics memory: 1896526 kiB.
[    4.282066] [TTM] Initializing pool allocator.
[    4.282085] usb 7-2: new full speed USB device number 2 using ohci_hcd
[    4.293076] [drm] radeon: 320M of VRAM memory ready
[    4.298277] [drm] radeon: 512M of GTT memory ready.
[    4.303218] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    4.309854] [drm] Driver supports precise vblank timestamp query.
[    4.315970] [drm] radeon: irq initialized.
[    4.320094] [drm] GART: num cpu pages 131072, num gpu pages 131072

So question is why radeon is using the address [0xa0000000 - 0xc000000], and in E820 it is RAM ....

[    0.000000]  BIOS-e820: 0000000000100000 - 00000000acb8d000 (usable) [    0.000000]  BIOS-e820: 00000000acb8d000 - 00000000acb8f000 (reserved) [    0.000000]  BIOS-e820: 00000000acb8f000 - 00000000afce9000 (usable) [    0.000000]  BIOS-e820: 00000000afce9000 - 00000000afd21000 (reserved) [    0.000000]  BIOS-e820: 00000000afd21000 - 00000000afd4f000 (usable) [    0.000000]  BIOS-e820: 00000000afd4f000 - 00000000afdcf000 (reserved) [    0.000000]  BIOS-e820: 00000000afdcf000 - 00000000afecf000 (ACPI NVS) [    0.000000]  BIOS-e820: 00000000afecf000 - 00000000afeff000 (ACPI data) [    0.000000]  BIOS-e820: 00000000afeff000 - 00000000aff00000 (usable)

so looks bios program wrong address to the radon card?

Thanks

Yinghai Lu

so those two patches uncover some problems.

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000

so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

but your radeon does use [0xa0000000, 0xbfffffff)

[    4.281993] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used)
[    4.290672] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[    4.298550] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.309857] [drm] RAM width 32bits DDR
[    4.313748] [TTM] Zone  kernel: Available graphics memory: 1896524 kiB.
[    4.320379] [TTM] Initializing pool allocator.
[    4.324948] [drm] radeon: 320M of VRAM memory ready
[    4.329832] [drm] radeon: 512M of GTT memory ready.

and the one seems working:

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0x80000000-0x83ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ 80000000
[    0.000000]     memblock_x86_reserve_range: [0xacb6bdc0-0xacb6bddf]          BOOTMEM

will use different position...

[    4.250159] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used)
[    4.258830] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[    4.266742] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.271549] [drm] RAM width 32bits DDR
[    4.275435] [TTM] Zone  kernel: Available graphics memory: 1896526 kiB.
[    4.282066] [TTM] Initializing pool allocator.
[    4.282085] usb 7-2: new full speed USB device number 2 using ohci_hcd
[    4.293076] [drm] radeon: 320M of VRAM memory ready
[    4.298277] [drm] radeon: 512M of GTT memory ready.
[    4.303218] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    4.309854] [drm] Driver supports precise vblank timestamp query.
[    4.315970] [drm] radeon: irq initialized.
[    4.320094] [drm] GART: num cpu pages 131072, num gpu pages 131072

So question is why radeon is using the address [0xa0000000 - 0xc000000], and in E820 it is RAM ....

[    0.000000]  BIOS-e820: 0000000000100000 - 00000000acb8d000 (usable)
[    0.000000]  BIOS-e820: 00000000acb8d000 - 00000000acb8f000 (reserved)
[    0.000000]  BIOS-e820: 00000000acb8f000 - 00000000afce9000 (usable)
[    0.000000]  BIOS-e820: 00000000afce9000 - 00000000afd21000 (reserved)
[    0.000000]  BIOS-e820: 00000000afd21000 - 00000000afd4f000 (usable)
[    0.000000]  BIOS-e820: 00000000afd4f000 - 00000000afdcf000 (reserved)
[    0.000000]  BIOS-e820: 00000000afdcf000 - 00000000afecf000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000afecf000 - 00000000afeff000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000afeff000 - 00000000aff00000 (usable)

so looks bios program wrong address to the radon card?

Quoted 67 lines · Yinghai Lu

so those two patches uncover some problems.

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0xa0000000-0xa3ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ a0000000

so kernel try to reallocate apperture. because BIOS allocated is pointed to RAM or size is too small.

but your radeon does use [0xa0000000, 0xbfffffff)

[    4.281993] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used)
[    4.290672] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[    4.298550] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.309857] [drm] RAM width 32bits DDR
[    4.313748] [TTM] Zone  kernel: Available graphics memory: 1896524 kiB.
[    4.320379] [TTM] Initializing pool allocator.
[    4.324948] [drm] radeon: 320M of VRAM memory ready
[    4.329832] [drm] radeon: 512M of GTT memory ready.

and the one seems working:

[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ a0000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000]     memblock_x86_reserve_range: [0x80000000-0x83ffffff]       aperture64
[    0.000000] Mapping aperture over 65536 KB of RAM @ 80000000
[    0.000000]     memblock_x86_reserve_range: [0xacb6bdc0-0xacb6bddf]          BOOTMEM

will use different position...

[    4.250159] radeon 0000:01:05.0: VRAM: 320M 0x00000000C0000000 - 0x00000000D3FFFFFF (320M used)
[    4.258830] radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[    4.266742] [drm] Detected VRAM RAM=320M, BAR=256M
[    4.271549] [drm] RAM width 32bits DDR
[    4.275435] [TTM] Zone  kernel: Available graphics memory: 1896526 kiB.
[    4.282066] [TTM] Initializing pool allocator.
[    4.282085] usb 7-2: new full speed USB device number 2 using ohci_hcd
[    4.293076] [drm] radeon: 320M of VRAM memory ready
[    4.298277] [drm] radeon: 512M of GTT memory ready.
[    4.303218] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    4.309854] [drm] Driver supports precise vblank timestamp query.
[    4.315970] [drm] radeon: irq initialized.
[    4.320094] [drm] GART: num cpu pages 131072, num gpu pages 131072

So question is why radeon is using the address [0xa0000000 - 0xc000000], and in E820 it is RAM ....

[    0.000000]  BIOS-e820: 0000000000100000 - 00000000acb8d000 (usable)
[    0.000000]  BIOS-e820: 00000000acb8d000 - 00000000acb8f000 (reserved)
[    0.000000]  BIOS-e820: 00000000acb8f000 - 00000000afce9000 (usable)
[    0.000000]  BIOS-e820: 00000000afce9000 - 00000000afd21000 (reserved)
[    0.000000]  BIOS-e820: 00000000afd21000 - 00000000afd4f000 (usable)
[    0.000000]  BIOS-e820: 00000000afd4f000 - 00000000afdcf000 (reserved)
[    0.000000]  BIOS-e820: 00000000afdcf000 - 00000000afecf000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000afecf000 - 00000000afeff000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000afeff000 - 00000000aff00000 (usable)

Quoted 2 lines

so looks bios program wrong address to the radon card?

Okay, staring at this, it definitely seems toxic to overlay the GART over memory areas reserved by the BIOS. If I were to guess, I would say that the problem here seems to be that the kernel thinks it is overlaying 64 MiB of memory, but the actual GART is in fact 512 MiB in size -- 131072 CPU pages -- which now overlaps the BIOS reserved areas.

Alex D., could you comment on the "num cpu pages" bit?

Quoted 7 lines

Okay, staring at this, it definitely seems toxic to overlay the GART over memory areas reserved by the BIOS. If I were to guess, I would say that the problem here seems to be that the kernel thinks it is overlaying 64 MiB of memory, but the actual GART is in fact 512 MiB in size -- 131072 CPU pages -- which now overlaps the BIOS reserved areas.

Quoted 1 line

Alex D., could you comment on the "num cpu pages" bit?

These are not CPU addresses. I think we've stated that already. Not the droids.

the num cpu pages is how many CPU pages would be needed to fill the GPU GTT, for those crazy cases where CPU pagesize != GPU pagesize.

Quoted 3 lines · Yinghai Lu

so looks bios program wrong address to the radon card?

Okay, staring at this, it definitely seems toxic to overlay the GART over memory areas reserved by the BIOS. If I were to guess, I would say that the problem here seems to be that the kernel thinks it is overlaying 64 MiB of memory, but the actual GART is in fact 512 MiB in size -- 131072 CPU pages -- which now overlaps the BIOS reserved areas.

Alex D., could you comment on the "num cpu pages" bit?

Quoted 5 lines · H. Peter Anvin

Quoted 3 lines · Yinghai Lu

so looks bios program wrong address to the radon card?

Quoted 7 lines

Okay, staring at this, it definitely seems toxic to overlay the GART over memory areas reserved by the BIOS.  If I were to guess, I would say that the problem here seems to be that the kernel thinks it is overlaying 64 MiB of memory, but the actual GART is in fact 512 MiB in size -- 131072 CPU pages -- which now overlaps the BIOS reserved areas.

Alex D., could you comment on the "num cpu pages" bit?

Okay, I tried the debug-patch from Yinghai (posted to the bugzilla):

drivers/gpu/drm/radeon/radeon_device.c +2 -0

@@ -325,6 +325,8 @@ void radeon_gtt_location(struct radeon_device *rdev, struct radeon_mc *mc)
                        mc->gtt_size = size_bf;
                }
                mc->gtt_start = (mc->vram_start & ~mc->gtt_base_align) - mc->gtt_size;
+               if (mc->gtt_start == 0xa0000000)
+                       mc->gtt_start = 0x80000000;
        } else {
                if (mc->gtt_size > size_af) {
                        dev_warn(rdev->dev, "limiting GTT\n");

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Alex, can you comment on this?

Quoted 5 lines · H. Peter Anvin

Quoted 3 lines · Yinghai Lu

so looks bios program wrong address to the radon card?

Quoted 7 lines

Okay, staring at this, it definitely seems toxic to overlay the GART over memory areas reserved by the BIOS. If I were to guess, I would say that the problem here seems to be that the kernel thinks it is overlaying 64 MiB of memory, but the actual GART is in fact 512 MiB in size -- 131072 CPU pages -- which now overlaps the BIOS reserved areas.

Alex D., could you comment on the "num cpu pages" bit?

Okay, I tried the debug-patch from Yinghai (posted to the bugzilla):

drivers/gpu/drm/radeon/radeon_device.c +2 -0

@@ -325,6 +325,8 @@ void radeon_gtt_location(struct radeon_device *rdev, struct radeon_mc *mc)
                        mc->gtt_size = size_bf;
                }
                mc->gtt_start = (mc->vram_start & ~mc->gtt_base_align) - mc->gtt_size;
+               if (mc->gtt_start == 0xa0000000)
+                       mc->gtt_start = 0x80000000;
        } else {
                if (mc->gtt_size > size_af) {
                        dev_warn(rdev->dev, "limiting GTT\n");

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Alex, can you comment on this?

I'd strongly suggest we revert back to the old and proven allocation order, as long as it results in valid layouts. Even if we figure out this particular GART/GTT assumption there might be a dozen others in other types of hardware.

Quoted 5 lines · H. Peter Anvin

Quoted 3 lines · Yinghai Lu

so looks bios program wrong address to the radon card?

Quoted 7 lines

Okay, staring at this, it definitely seems toxic to overlay the GART over memory areas reserved by the BIOS.  If I were to guess, I would say that the problem here seems to be that the kernel thinks it is overlaying 64 MiB of memory, but the actual GART is in fact 512 MiB in size -- 131072 CPU pages -- which now overlaps the BIOS reserved areas.

Alex D., could you comment on the "num cpu pages" bit?

Okay, I tried the debug-patch from Yinghai (posted to the bugzilla):

drivers/gpu/drm/radeon/radeon_device.c +2 -0

@@ -325,6 +325,8 @@ void radeon_gtt_location(struct radeon_device *rdev, struct radeon_mc *mc)
                        mc->gtt_size = size_bf;
                }
                mc->gtt_start = (mc->vram_start & ~mc->gtt_base_align) - mc->gtt_size;
+               if (mc->gtt_start == 0xa0000000)
+                       mc->gtt_start = 0x80000000;
        } else {
                if (mc->gtt_size > size_af) {
                        dev_warn(rdev->dev, "limiting GTT\n");

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Alex, can you comment on this?

Regards,

       Joerg

Quoted 4 lines · Joerg Roedel

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Quoted 1 line

Alex, can you comment on this?

As Dave said, they are completely different addresses spaces. You could put the GPU aperture at 0 if you wanted (in fact we do on some chips). Perhaps there's some strange interaction with the nb gart since the nb gart on that chipset was designed to be used for graphics and the rs780/880 can be configured to use an agp aperture. Unfortunately, I'm not that familiar with the nb gart.

Quoted 5 lines · Alex Deucher

Quoted 4 lines · Joerg Roedel

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Quoted 8 lines

Quoted 1 line

Alex, can you comment on this?

As Dave said, they are completely different addresses spaces.  You could put the GPU aperture at 0 if you wanted (in fact we do on some chips).  Perhaps there's some strange interaction with the nb gart since the nb gart on that chipset was designed to be used for graphics and the rs780/880 can be configured to use an agp aperture. Unfortunately, I'm not that familiar with the nb gart.

Actually, the nb gart is part of the cpu. It is part of the cpu north bridge and can translate io and cpu accesses. In fact, it is a remapper of physical memory addresses.

The problem seems to be related to specific gpu chips. On another notebook with an hd3000 card gtt and the nb gart aperture are both on 0xa0000000 too but the box works fine. I havn't tested with an hd5000 yet. The failing notebook has an hd4200 mobility.

Btw. what happens if the gpu accesses an unmapped address in the gtt range?

Quoted 3 lines · Joerg Roedel

Actually, the nb gart is part of the cpu. It is part of the cpu north bridge and can translate io and cpu accesses. In fact, it is a remapper of physical memory addresses.

I know what it's for. In the IGP graphics chip is also part of the north bridge, but it may not be related at all.

Quoted 4 lines

The problem seems to be related to specific gpu chips. On another notebook with an hd3000 card gtt and the nb gart aperture are both on 0xa0000000 too but the box works fine. I havn't tested with an hd5000 yet. The failing notebook has an hd4200 mobility.

What exact model is the hd3000? Is it IGP GPU or a discrete GPU? It
it's an IGP, it's identical to the hd4200 programming-wise.

Quoted 2 lines

Btw. what happens if the gpu accesses an unmapped address in the gtt range?

It's redirected to a dummy page.

Quoted 15 lines · Joerg Roedel

Quoted 5 lines · Alex Deucher

Quoted 4 lines · Joerg Roedel

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Quoted 8 lines

Quoted 1 line

Alex, can you comment on this?

As Dave said, they are completely different addresses spaces.  You could put the GPU aperture at 0 if you wanted (in fact we do on some chips).  Perhaps there's some strange interaction with the nb gart since the nb gart on that chipset was designed to be used for graphics and the rs780/880 can be configured to use an agp aperture. Unfortunately, I'm not that familiar with the nb gart.

Quoted 3 lines

Actually, the nb gart is part of the cpu. It is part of the cpu north bridge and can translate io and cpu accesses. In fact, it is a remapper of physical memory addresses.

I know what it's for. In the IGP graphics chip is also part of the north bridge, but it may not be related at all.

Quoted 1 line

Quoted 4 lines

The problem seems to be related to specific gpu chips. On another notebook with an hd3000 card gtt and the nb gart aperture are both on 0xa0000000 too but the box works fine. I havn't tested with an hd5000 yet. The failing notebook has an hd4200 mobility.

What exact model is the hd3000? Is it IGP GPU or a discrete GPU? It
it's an IGP, it's identical to the hd4200 programming-wise.

Quoted 5 lines · Alex Deucher

Quoted 4 lines · Joerg Roedel

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Quoted 8 lines

Quoted 1 line

Alex, can you comment on this?

As Dave said, they are completely different addresses spaces. You could put the GPU aperture at 0 if you wanted (in fact we do on some chips). Perhaps there's some strange interaction with the nb gart since the nb gart on that chipset was designed to be used for graphics and the rs780/880 can be configured to use an agp aperture. Unfortunately, I'm not that familiar with the nb gart.

Actually, the nb gart is part of the cpu. It is part of the cpu north bridge and can translate io and cpu accesses. In fact, it is a remapper of physical memory addresses.

The problem seems to be related to specific gpu chips. On another notebook with an hd3000 card gtt and the nb gart aperture are both on 0xa0000000 too but the box works fine.

Quoted 19 lines · Joerg Roedel

Quoted 5 lines · Alex Deucher

Quoted 4 lines · Joerg Roedel

And this makes a difference, with this change on-top of -rc3 the box boots fine. So there seems to be some dependency between the GART base and the GTT base even when they are in different address spaces.

Quoted 8 lines

Quoted 1 line

Alex, can you comment on this?

As Dave said, they are completely different addresses spaces. You could put the GPU aperture at 0 if you wanted (in fact we do on some chips). Perhaps there's some strange interaction with the nb gart since the nb gart on that chipset was designed to be used for graphics and the rs780/880 can be configured to use an agp aperture. Unfortunately, I'm not that familiar with the nb gart.

Actually, the nb gart is part of the cpu. It is part of the cpu north bridge and can translate io and cpu accesses. In fact, it is a remapper of physical memory addresses.

Quoted 3 lines

The problem seems to be related to specific gpu chips. On another notebook with an hd3000 card gtt and the nb gart aperture are both on 0xa0000000 too but the box works fine.

Wasn't the working theory that the problem occurs if those two values aren't the same?

It's been another almost spookily calm week. Usually this kind of calmness happens much later in the -rc series (during -rc7 or -rc8, say), but I'm not going to complain. I'm just still waiting for the other shoe to drop.

however, the output of mount looks very confused..

.38:
/dev/mapper/vg_adamo-lv_home on /home type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)

.39:
- on /home type 79a9-4526-888c-1f86d35a6704 (rw,relatime,ext4)

It looks like /proc/self/mountinfo broke abi.

.38:
48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered

.39:
46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

looks like this was caused by 93f1c20bc8cdb757be50566eff88d65c3b26881f

perhaps adding that string to the end of the line would preserve what mount expects ?

Quoted 4 lines · Dave Jones

looks like this was caused by 93f1c20bc8cdb757be50566eff88d65c3b26881f

perhaps adding that string to the end of the line would preserve what mount expects ?

Care to test? Otherwise I'll just revert the thing.. It's clearly not valid behavior to randomly add some new field into the middle of a /proc file.

Quoted 13 lines · Dave Jones

however, the output of mount looks very confused..

.38:
/dev/mapper/vg_adamo-lv_home on /home type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)

.39:
- on /home type 79a9-4526-888c-1f86d35a6704 (rw,relatime,ext4)

It looks like /proc/self/mountinfo broke abi.

.38:
48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered

Quoted 2 lines

.39:
46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

looks like this was caused by 93f1c20bc8cdb757be50566eff88d65c3b26881f

perhaps adding that string to the end of the line would preserve what mount expects ?

Quoted 18 lines · Dave Jones

On Tue, Apr 12, 2011 at 03:09:34PM -0400, Dave Jones wrote:

 > however, the output of mount looks very confused..
 >
 > .38:
 > /dev/mapper/vg_adamo-lv_home on /home type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)
 >
 > .39:
 > - on /home type 79a9-4526-888c-1f86d35a6704 (rw,relatime,ext4)
 >
 > It looks like /proc/self/mountinfo broke abi.
 >
 > .38:
 > 48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered
 >
 > .39:
 > 46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

Quoted 3 lines

looks like this was caused by 93f1c20bc8cdb757be50566eff88d65c3b26881f

perhaps adding that string to the end of the line would preserve what mount expects ?

uuid:<value> is the option field  as per Documentation/filesystem/proc.txt. There was an error in libmount parsing which got fixed upstream recently

Quoted 3 lines

uuid:<value> is the option field  as per Documentation/filesystem/proc.txt. There was an error in libmount parsing which got fixed upstream recently

Just a simple question about this approach in general? A filesystem UUID can be changed on disk at any time (tune2fs -U ...).

Your code looks like you copy the bytes to the in-kernel superblock structure without noticing any later changes on disk? How is that supposed to work?

On Tue, Apr 19, 2011 at 12:57:27AM +0200, Kay Sievers wrote:

 > > uuid:<value> is the option field  as per
 > > Documentation/filesystem/proc.txt. There was an error in libmount > > parsing which got fixed upstream recently
 >
 > Just a simple question about this approach in general? A filesystem > UUID can be changed on disk at any time (tune2fs -U ...). >
 > Your code looks like you copy the bytes to the in-kernel superblock > structure without noticing any later changes on disk? How is that > supposed to work?

I thought tune2fs on a mounted filesystem was always a "you get to keep both pieces if it breaks" situation.

Quoted 3 lines

Your code looks like you copy the bytes to the in-kernel superblock structure without noticing any later changes on disk? How is that supposed to work?

I thought tune2fs on a mounted filesystem was always a "you get to keep both pieces if it breaks" situation.

Quoted 19 lines · Aneesh Kumar K.V

Quoted 18 lines · Dave Jones

On Tue, Apr 12, 2011 at 03:09:34PM -0400, Dave Jones wrote:

 > however, the output of mount looks very confused..
 >
 > .38:
 > /dev/mapper/vg_adamo-lv_home on /home type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)
 >
 > .39:
 > - on /home type 79a9-4526-888c-1f86d35a6704 (rw,relatime,ext4)
 >
 > It looks like /proc/self/mountinfo broke abi.
 >
 > .38:
 > 48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered
 >
 > .39:
 > 46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

Quoted 7 lines

Quoted 3 lines

looks like this was caused by 93f1c20bc8cdb757be50566eff88d65c3b26881f

perhaps adding that string to the end of the line would preserve what mount expects ?

uuid:<value> is the option field  as per Documentation/filesystem/proc.txt. There was an error in libmount parsing which got fixed upstream recently

Just a simple question about this approach in general? A filesystem UUID can be changed on disk at any time (tune2fs -U ...).

Your code looks like you copy the bytes to the in-kernel superblock structure without noticing any later changes on disk? How is that supposed to work?

Quoted 29 lines · Kay Sievers

Quoted 19 lines · Aneesh Kumar K.V

Quoted 18 lines · Dave Jones

Quoted 15 lines · Dave Jones

however, the output of mount looks very confused..

.38:
/dev/mapper/vg_adamo-lv_home on /home type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)

.39:
- on /home type 79a9-4526-888c-1f86d35a6704 (rw,relatime,ext4)

It looks like /proc/self/mountinfo broke abi.

.38:
48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered

.39:
46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

Quoted 7 lines

Quoted 3 lines

looks like this was caused by 93f1c20bc8cdb757be50566eff88d65c3b26881f

perhaps adding that string to the end of the line would preserve what mount expects ?

uuid:<value> is the option field as per Documentation/filesystem/proc.txt. There was an error in libmount parsing which got fixed upstream recently

Quoted 7 lines

Just a simple question about this approach in general? A filesystem UUID can be changed on disk at any time (tune2fs -U ...).

Your code looks like you copy the bytes to the in-kernel superblock structure without noticing any later changes on disk? How is that supposed to work?

Isn't that true even for the fsid returned by statfs ?. IIUC tune2fs won't change even the ext4_super_block.s_uuid .

-aneesh

Quoted 6 lines · Kay Sievers

Just a simple question about this approach in general? A filesystem UUID can be changed on disk at any time (tune2fs -U ...).

Your code looks like you copy the bytes to the in-kernel superblock structure without noticing any later changes on disk? How is that supposed to work?

Isn't that true even for the fsid returned by statfs ?.  IIUC tune2fs won't change even the ext4_super_block.s_uuid .

Quoted 4 lines · Linus Torvalds

It's been another almost spookily calm week. Usually this kind of calmness happens much later in the -rc series (during -rc7 or -rc8, say), but I'm not going to complain. I'm just still waiting for the other shoe to drop.

Here's an odd one.

my laptop's fstab has

/dev/mapper/vg_adamo-lv_home /home ext4 defaults 1 2

on 2.6.38, /proc/mounts contains ..

/dev/mapper/vg_adamo-lv_home /home ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0

on 2.6.39rc3 it looks like..

/dev/mapper/vg_adamo-lv_home /home ext4 rw,seclabel,relatime,user_xattr,barrier=1,data=ordered 0 0

Which looks like ea6633369458992241599c9d9ebadffaeddec164, so nothing untoward..

however, the output of mount looks very confused..

.38:
/dev/mapper/vg_adamo-lv_home on /home type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)

.39:
- on /home type 79a9-4526-888c-1f86d35a6704 (rw,relatime,ext4)

It looks like /proc/self/mountinfo broke abi.

.38:
48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered

.39:
46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

This file contains lines of the form:

36 35 98:0 /mnt1 /mnt2 rw,noatime master:1 - ext3 /dev/root rw,errors=continue
(1)(2)(3) (4) (5) (6) (7) (8) (9) (10) (11)

(1) mount ID: unique identifier of the mount (may be reused after umount)
(2) parent ID: ID of parent (or of self for the top of the mount tree)
(3) major:minor: value of st_dev for files on filesystem
(4) root: root of the mount within the filesystem
(5) mount point: mount point relative to the process's root
(6) mount options: per mount options
(7) optional fields: zero or more fields of the form "tag[:value]"
(8) separator: marks the end of the optional fields
(9) filesystem type: name of filesystem of the form "type[.subtype]"
(10) mount source: filesystem specific information or "none"
(11) super options: per super block options

Dave

Quoted 8 lines

It looks like /proc/self/mountinfo broke abi.

.38:
48 45 253:3 / /home rw,relatime - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,barrier=1,data=ordered

.39:
46 22 253:3 / /home rw,relatime uuid:f3971858-79a9-4526-888c-1f86d35a6704 - ext4 /dev/mapper/vg_adamo-lv_home rw,seclabel,user_xattr,barrier=1,data=ordered

so it's supposed to be like this, from Documentation/filesystems/proc.txt:

Quoted 5 lines

This file contains lines of the form:

36 35 98:0 /mnt1 /mnt2 rw,noatime master:1 - ext3 /dev/root rw,errors=continue
(1)(2)(3) (4) (5) (6) (7) (8) (9) (10) (11)

Quoted 11 lines

(1) mount ID: unique identifier of the mount (may be reused after umount)
(2) parent ID: ID of parent (or of self for the top of the mount tree)
(3) major:minor: value of st_dev for files on filesystem
(4) root: root of the mount within the filesystem
(5) mount point: mount point relative to the process's root
(6) mount options: per mount options
(7) optional fields: zero or more fields of the form "tag[:value]"
(8) separator: marks the end of the optional fields
(9) filesystem type: name of filesystem of the form "type[.subtype]"
(10) mount source: filesystem specific information or "none"
(11) super options: per super block options

it does seem that the new UUID info is in a perfectly fine place (the optional fields slot), at least per the docs, so I guess I might blame the mount binary for not following the aforementioned rules...

Maybe Karel knows? cc'd...

so it's supposed to be like this, from Documentation/filesystems/proc.txt:

Quoted 4 lines

This file contains lines of the form:

36 35 98:0 /mnt1 /mnt2 rw,noatime master:1 - ext3 /dev/root rw,errors=continue (1)(2)(3)   (4)   (5)      (6)      (7)   (8) (9)   (10)         (11)

It's been another almost spookily calm week. Usually this kind of calmness happens much later in the -rc series (during -rc7 or -rc8, say), but I'm not going to complain. I'm just still waiting for the other shoe to drop.

Yep, I had to hit a buglet too. Looks like block layer, ->request_fn (do_ide_request) called in IRQ disabled region. Process is blkid, i.e. some udev helper. Kernel is -rc3 + Joerg's USB quirk fix which should be unrelated.

Happens when I put a cd in the drive. More info upon request.

[20933.365059] BUG: sleeping function called from invalid context at drivers/ide/ide-io.c:468
[20933.365113] in_atomic(): 1, irqs_disabled(): 0, pid: 5817, name: blkid
[20933.365166] no locks held by blkid/5817.
[20933.365193] Pid: 5817, comm: blkid Not tainted 2.6.39-rc3-00001-g1b521ee #9
[20933.365228] Call Trace:
[20933.365282]  [<ffffffff8102db2b>] __might_sleep+0x103/0x108
[20933.365312]  [<ffffffff812e61c8>] do_ide_request+0x4a/0x58e
[20933.365362]  [<ffffffff811951af>] ? cfq_prio_tree_add+0xb3/0xc2
[20933.365390]  [<ffffffff811974f3>] ? cfq_add_rq_rb+0xb1/0xc5
[20933.365437]  [<ffffffff81197579>] ? cfq_insert_request+0x72/0x433
[20933.365465]  [<ffffffff81187fd8>] __blk_run_queue+0x80/0xee
[20933.365511]  [<ffffffff81188141>] flush_plug_list+0xfb/0x139
[20933.365540]  [<ffffffff810a9a86>] ? sleep_on_page+0x12/0x12
[20933.365586]  [<ffffffff81188199>] __blk_flush_plug+0x1a/0x3a
[20933.365613]  [<ffffffff81442913>] schedule+0x3d2/0xb4b
[20933.365663]  [<ffffffff811a2cee>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[20933.365692]  [<ffffffff8100387f>] ? do_softirq+0x77/0x85
[20933.365739]  [<ffffffff81446244>] ? retint_restore_args+0xe/0xe
[20933.365767]  [<ffffffff810a9a86>] ? sleep_on_page+0x12/0x12
[20933.365812]  [<ffffffff81443377>] preempt_schedule+0x37/0x4b
[20933.365839]  [<ffffffff81445f92>] _raw_spin_unlock_irqrestore+0x64/0x69
[20933.365878]  [<ffffffff81057b3b>] prepare_to_wait_exclusive+0x6c/0x77
[20933.365925]  [<ffffffff81443806>] __wait_on_bit_lock+0x34/0x8f
[20933.365953]  [<ffffffff810a9a00>] __lock_page_killable+0x66/0x6d
[20933.366000]  [<ffffffff810579a1>] ? autoremove_wake_function+0x3d/0x3d
[20933.366090]  [<ffffffff810ab56d>] generic_file_aio_read+0x491/0x67c
[20933.366123]  [<ffffffff810e8137>] do_sync_read+0xcb/0x108
[20933.366183]  [<ffffffff81068f4d>] ? trace_hardirqs_on+0xd/0xf
[20933.366217]  [<ffffffff810e8bce>] vfs_read+0xb3/0x13b
[20933.366270]  [<ffffffff810e8d1f>] sys_read+0x4d/0x77
[20933.366303]  [<ffffffff81446aeb>] system_call_fastpath+0x16/0x1b

Thanks.

--
Regards/Gruss,
   Boris.

Quoted 5 lines · Borislav Petkov

Yep, I had to hit a buglet too. Looks like block layer, ->request_fn (do_ide_request) called in IRQ disabled region. Process is blkid, i.e. some udev helper. Kernel is -rc3 + Joerg's USB quirk fix which should be unrelated.

I think this particular backtrace should be fixed by commit 6631e635c65d ("block: don't flush plugged IO on forced preemtion scheduling"), although even without preempt scheduling, I don't think it's at all ok to sleep inside __blk_run_queue.

Jens? Even from a _regular_ schedule, it would not be ok if we end up sleeping - we're caching things like the request-queue, and we have preempt_disable() inside the scheduler for a very good reason.

Jens already has a fix in his tree to always offload the block I/O submission to blockd for this case.