DMA cache consistency bug introduced in 2.6.28 (Was: Re: [Fdutils] Cannot format floppies under kernel 2.6.*?)

Quoted 8 lines · Mark Hounschell

Quoted 4 lines · Alain Knaff

Quoted 1 line

Should I do more work in between?

No, but make sure to look at track 0... Other tracks will still have the error, as there was nothing forcing a memory flush between track 0 and 1...

Ok track 0

[...]

Quoted 8 lines

0: 0
1: 0
2: 0
3: 4f <--
4: 0
5: 1
6: 2
no disk change

Yeah, that's what I meant... So the memory flusher program didn't manage to clear up the inconsistency...

So either my theory is wrong, or the memory flusher program was not efficient enough.... hmmm, maybe doing some surfing in between the formats, or doing another kernel compilation might be a better test.

Alain

1. initial contents: 33 44 55 66
2. one DMA transfer is performed
3. program changes buffer to: 77 88 99 aa
4. new DMA transfer is performed => instead it transmits 33 88 99 aa (i.e. first byte is from previous contents)

This used to work in 2.6.27.41, but broke in 2.6.28 . It doesn't happen on all hardware though.

It does indeed seem to be related to a DMA-side cache (rather than the processor's cache not being flushed to main memory), as doing lots of memory intensive work (kernel compilation) between 2 and 3 doesn't fix the problem.

In the diff between 2.6.27.41 and 2.6.28, I noticed a lot of changes in arch/x86/kernel/amd_iommu.c and related files, could any of these have triggered this behavior?

On x86, where all caches are supposed to be totally coherent (except for I$ under very special circumstances),

Quoted 6 lines · Alain Knaff

1. initial contents: 33 44 55 66
2. one DMA transfer is performed
3. program changes buffer to: 77 88 99 aa
4. new DMA transfer is performed => instead it transmits 33 88 99 aa (i.e. first byte is from previous contents)

Quoted 2 lines

This used to work in 2.6.27.41, but broke in 2.6.28 . It doesn't happen on all hardware though.

Do you have a list of hardware it works on? Especially chipsets.

I'm not entirely surprised. Actual CPU bugs are pretty rare in the x86 world. But chipset bugs? Another thing entirely. There are buffers and caches there, and those are sometimes software-visible. The most obvious case of that is just the IOMMU's themselves, but from your description I don't think you actually change the DMA _mappings_ do you? Just the actual buffer (that was then mapped earlier)?

So I don't think it's the IOMMU code itself necessarily, although an IOMMU may well be involved (eg I could easily see a few cachelines worth of actual DMA data caching going on in the whole IOMMU too)

And to some degree the floppy driver might be _more_ likely to see some kinds of bugs, because it uses that crazy legacy DMA engine. So it's not

going to go through the regular PCI DMA hardware paths, it's going to go through its own special paths that nobody else uses any more (and thus has probably not had as much testing).

Quoted 3 lines

In the diff between 2.6.27.41 and 2.6.28, I noticed a lot of changes in arch/x86/kernel/amd_iommu.c and related files, could any of these have triggered this behavior?

Could it have triggered? Sure. Chipset caches are often flushed by certain trivial operations (often the caches are small, and operations like "any PIO access" will make sure they are flushed). Different IOMMU flush patterns could easily account for it.

But I think we'd like to see a list of hardware where this can be triggered,

and quite frankly, a 'git bisect' would be absolutely wonderful

especially if the list of hardware is not showing any really obvious patterns (and I assume they aren't all _that_ obvious, or you'd have mentioned them).

Linus

For the moment, I have a very small sample of hardware:
1. One machine which works (my own): Athlon XP 1800+ processor
2. One which doesn't work (Mark's)

I might get access to a wider sample of boxen in a week or so, in order to do some stats.

What's the easiest way to find out the chipset?

Here's already the output of lspci from my machine (works):

00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge 00:01.0 PCI bridge: VIA Technologies, Inc. VT8235 PCI Bridge 00:11.0 ISA bridge: VIA Technologies, Inc. VT8235 ISA Bridge

(It happens during formatting the floppy drive: here the first byte happens to be the trackid of the first physical sector of the track, and it always ends up being the track of the *previously* formatted track).

Quoted 2 lines

But I think we'd like to see a list of hardware where this can be triggered,

We'll get a list of 2 machines relatively quickly (unless other people would like to chime in: the test is easy, just fdformat a floppy disk), and more in a week or so.

Quoted 1 line

and quite frankly, a 'git bisect' would be absolutely wonderful

How exactly would I use this (command line sample)?

Quoted 3 lines · Alain Knaff

For the moment, I have a very small sample of hardware:
1. One machine which works (my own): Athlon XP 1800+ processor
2. One which doesn't work (Mark's)

Ok. I don't think I even have any machines with floppy drives any more (one external USB drive somewhere gathering dust just in case I ever encounter a floppy again).

Quoted 2 lines

I might get access to a wider sample of boxen in a week or so, in order to do some stats.

Ok, I was more thinking "we have a bugzilla with ten different people reporting this". If it's just a single machine, that's not going to be relevant.

Quoted 5 lines

What's the easiest way to find out the chipset?

Here's already the output of lspci from my machine (works):

Quoted 3 lines

00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge 00:01.0 PCI bridge: VIA Technologies, Inc. VT8235 PCI Bridge 00:11.0 ISA bridge: VIA Technologies, Inc. VT8235 ISA Bridge

Yeah, lspci (and generally only the northbridge and southbridge matters, the "ISA bridge" might technically be relevant, but since it's universally on the same die as the southbridge, I left it in there just for kicks).

Only the "it doesn't work on xyz" is likely interesting. The machines it works on are probably uninteresting statistically.

You'd need a git tree that contains both the working and non-working versions, and then literally just do

git bisect start
git bisect good <known good version number here> git bisect bad <known bad version here>

and it will give you a commit to try. Compile, test, see if it's good or bad, and do

git bisect [good|bad]

depending on the result. Rinse and repeat (depending on how tight the initial good/bad commits were, it will need 10-15 kernel tests).

So in this case, since apparently 2.6.27.41 is good, and 2.6.28 is not, it would be something like this:

	# clone hpa's tree that has all the stable releases in one place
	git clone git://git.kernel.org/pub/scm/linux/kernel/git/hpa/linux-2.6-allstable.git

	cd linux-2.6-allstable
	git bisect start
	git bisect bad v2.6.28
	git bisect good v2.6.27.41

and off you go.

NOTE! Bisection depends very much on the bug being 100% reproducible. If you ever mark a good kernel bad (because you messed up) or a bad kernel good (because the bug wasn't 100% reproducible, so you _thought_ it was good even though the bug was present and just happened to hide), the end result of the bisect will be totally unreliable and seriously screwed up.

So after a successful bisect, it is usually a good idea to try to go back to the original known-bad kernel, and then revert the commit that was indicated as the bad one (assuming the revert works - it could be that the bad one ends up being fundamental to other commits after it), and test that yes, that really fixes the bug.

It gets more complicated if the bisect hits kernels that you can't test because they have _unrelated_ issues on that machine (compile failures or just other bugs that hide the actual floppy behavior), but generally bisection is pretty simple. "man git-bisect" does have some extra pointers.

So git bisect may be somewhat time-consuming and mindless, but for reliably triggering bugs where nobody really knows what caused the bug it is a _really_ convenient thing to do. The only thing you need is a reliably triggering test-case, and some time.

Linus

[...]

Quoted 10 lines

You'd need a git tree that contains both the working and non-working versions, and then literally just do

git bisect start
git bisect good <known good version number here> git bisect bad <known bad version here>

and it will give you a commit to try. Compile, test, see if it's good or bad, and do

Quoted 4 lines

git bisect [good|bad]

depending on the result. Rinse and repeat (depending on how tight the initial good/bad commits were, it will need 10-15 kernel tests).

... and how do I check out the most recent good / oldest bad kernel for compilation?

Quoted 5 lines

So after a successful bisect, it is usually a good idea to try to go back to the original known-bad kernel, and then revert the commit that was indicated as the bad one (assuming the revert works - it could be that the bad one ends up being fundamental to other commits after it), and test that yes, that really fixes the bug.

What command lines would I use for that revert?

Quoted 11 lines · Alain Knaff

[...]

Quoted 10 lines

You'd need a git tree that contains both the working and non-working versions, and then literally just do

git bisect start
git bisect good <known good version number here> git bisect bad <known bad version here>

and it will give you a commit to try. Compile, test, see if it's good or bad, and do

Quoted 6 lines

Quoted 4 lines

git bisect [good|bad]

depending on the result. Rinse and repeat (depending on how tight the initial good/bad commits were, it will need 10-15 kernel tests).

... and how do I check out the most recent good / oldest bad kernel for compilation?

'git bisect' does all that for you. You don't need to check out the kernels you mark good or bad - git will just calculate the commit graphs, and pick a commit that is in the "middle" between them, and check out that commit.

Quoted 6 lines

Quoted 5 lines

So after a successful bisect, it is usually a good idea to try to go back to the original known-bad kernel, and then revert the commit that was indicated as the bad one (assuming the revert works - it could be that the bad one ends up being fundamental to other commits after it), and test that yes, that really fixes the bug.

What command lines would I use for that revert?

git revert <sha1-that-git-bisect-reported>

but even if that revert isn't successful, just the bisection result will be very interesting (assuming it all looks sane, of course - as mentioned, sometimes bisect results get screwed up because the bug isn't entirely reproducible due to timing etc).

Linus

Now, I only need to find a machine locally to test this on. Or Mark: are you confident in doing this yourself?

Quoted 4 lines · Alain Knaff

Now, I only need to find a machine locally to test this on. Or Mark: are you confident in doing this yourself?

I'll give it a shot. Sounds easy enough. If I have problems, I'll yell.

harley:/usr/src/linux-2.6-allstable # git bisect good
Bisecting: 2443 revisions left to test after this (roughly 11 steps) [db563fc2e80534f98c7f9121a6f7dfe41f177a79] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/sfrench/cifs-2.6

This one doesn't build:

CC [M] fs/ext3/super.o
fs/ext3/super.c: In function ‘ext3_quota_on’:
fs/ext3/super.c:2839: error: ‘nd’ undeclared (first use in this function) fs/ext3/super.c:2839: error: (Each undeclared identifier is reported only once fs/ext3/super.c:2839: error: for each function it appears in.) make[2]: *** [fs/ext3/super.o] Error 1
make[1]: *** [fs/ext3] Error 2
make: *** [fs] Error 2

I haven't yet determined that I can but, if I were to make a modification to the tree now to fix this would that screw up the bisect process?

This one doesn't build:

CC [M] fs/ext3/super.o
fs/ext3/super.c: In function ‘ext3_quota_on’:
fs/ext3/super.c:2839: error: ‘nd’ undeclared (first use in this function) fs/ext3/super.c:2839: error: (Each undeclared identifier is reported only once fs/ext3/super.c:2839: error: for each function it appears in.) make[2]: *** [fs/ext3/super.o] Error 1
make[1]: *** [fs/ext3] Error 2
make: *** [fs] Error 2

I haven't yet determined that I can but, if I were to make a modification to the tree now to fix this would that screw up the bisect process?

Quoted 12 lines · Mark Hounschell

This one doesn't build:

CC [M] fs/ext3/super.o
fs/ext3/super.c: In function ‘ext3_quota_on’:
fs/ext3/super.c:2839: error: ‘nd’ undeclared (first use in this function) fs/ext3/super.c:2839: error: (Each undeclared identifier is reported only once fs/ext3/super.c:2839: error: for each function it appears in.) make[2]: *** [fs/ext3/super.o] Error 1
make[1]: *** [fs/ext3] Error 2
make: *** [fs] Error 2

Quoted 2 lines

I haven't yet determined that I can but, if I were to make a modification to the tree now to fix this would that screw up the bisect process?

You can safely fix unrelated problems without screwing up the bisection. And in this case you can be pretty sure that this is unrelated, so it's all ok.

The fix for that silly problem is

(quoted diff) +1 -1

-                       path_put(&nd.path);
+                       path_put(&path);

(it's due to a silent merge failure - it merged cleanly, but semantics had changed in a branch and impacted code that was newly introduced in another branch).

Yep, thanks. I'm past that now. But haven't done a bisect [good|bad] on the results of that one yet. Did you see Alain's email response to my bisect progress report to him?

I'm still at a loss as to how to proceed?

Quoted 6 lines · Mark Hounschell

Yep, thanks. I'm past that now. But haven't done a bisect [good|bad] on the results of that one yet. Did you see Alain's email response to my bisect progress report to him?

I'm still at a loss as to how to proceed?

Ahh, the HPET issue.

That one is actually very interesting information, because we've had problems with HPET before. But what I would suggest is to try to continue to bisect with HPET enabled (to see the problem), and the commit that you couldn't even boot with HPET enabled you should not count as good or bad because you just don't know.

You can do "git bisect skip" to make git know that some particular commit is not a commit you can test, and you can also move away from a whole problematic region to another area by doing

git bisect visualize

to bring up a graphical gitk view of what all you have left to bisect, pick a good point (still _reasonably_ close to the middle) there, and do

	git reset --hard <the-point-you-want-to-test>

and try that kernel instead of the one git bisect suggested.

But this floppy DMA inconsistency being somehow HPET-related is interestign in itself. One thing that HPET does si to obviously change how we read the time - and what that can cause (totally indirectly) is that now we don't touch the southbridge with IO accesses nearly as much, because instead of going to the old 8253 PIT will touch the same legacy chip support that implements the floppy controller itself.

So it's entirely possible that the reason a non-HPET setup doesn't show this is that the accesses to the i8253 PIT part will "synchronize" the old floppy controller too, and hide some issue.

But still, I assume you had HPET enabled in 2.6.27, so it would be interesting to see exactly when the problem starts.

Linus

It looks like I may have to back up and first find the points that, let me, and stop me, booting with the HPET enabled. Before I change direction, can the git-bisect start sequence use the SHA1 id for the starting 'goods' and 'bads'? I don't see reference to that in the doc.

Quoted 5 lines · Mark Hounschell

It looks like I may have to back up and first find the points that, let me, and stop me, booting with the HPET enabled. Before I change direction, can the git-bisect start sequence use the SHA1 id for the starting 'goods' and 'bads'? I don't see reference to that in the doc.

You can always use a SHA1 id instead of a tag. So when you did

	git bisect good v2.6.17.4

you could always have replaced that "v2.6.17.4" with the SHA1 of the commit.

In git, the SHA1 ID's are the "real" names - the tags and branch names are purely for human-readable decoration. Git always turns them into SHA1 id's internally.

Linus

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] <[email protected]>
Date:   Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible. We retain the HPET timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when legacy mode is being used. We setup the remaining HPET timers as per CPU MSI based timers. This per CPU timer will eliminate the need for timer broadcasting with IRQ 0 when there is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available timers, CPUs that do not find any timer to use will continue using LAPIC and IRQ 0 broadcast.

Signed-off-by Venkatesh Pallipadi <[email protected]>

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not boot if I had hpet enabled. To get this one to boot (single user mode only) I had to add the the quiet cmdline option and following patch from to arch/x86/kernel/hpet.c

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct hpet_dev *dev)
{

        if (request_irq(dev->irq, hpet_interrupt_handler,

(quoted diff) +1 -1

-                       IRQF_SHARED|IRQF_NOBALANCING, dev->name, dev))
+                       IRQF_DISABLED|IRQF_NOBALANCING, dev->name, dev))
                return -1;
 
        disable_irq(dev->irq);

AND add the quiet cmdline option.

Also, of all the machines it does work on with hpets enabled, I don't see the HPET2 in /proc/interupts as below.

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:         82          0          3          0   IO-APIC-edge      timer
  1:          0          0       1712          6   IO-APIC-edge      i8042
  3:          0          0          6          0   IO-APIC-edge
  4:          0          0          6          0   IO-APIC-edge
  6:          0          0          4          0   IO-APIC-edge      floppy
  8:          0          0         60          0   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0      37798        179   IO-APIC-edge      i8042
 14:          0          0      16462         71   IO-APIC-edge      pata_atiixp
 15:          0          0       5713         17   IO-APIC-edge      pata_atiixp
 16:          0          0        904          2   IO-APIC-fasteoi   aic79xx, ohci_hcd:usb2, ohci_hcd:usb4, HDA Intel, ni-pci-gpib
 17:          0          0          2          0   IO-APIC-fasteoi   ehci_hcd:usb1, parport0, ni-pci-gpib
 18:          0          0      49940         90   IO-APIC-fasteoi   ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, nvidia
 19:          0          0        703          2   IO-APIC-fasteoi   aic7xxx, ehci_hcd:usb3, ttySLG0, eth1
 22:          0          0       1303         15   IO-APIC-fasteoi   ahci

24: 261763 0 0 0 HPET_MSI-edge hpet2

 29:          0          0        220          5   PCI-MSI-edge      sky2@pci:0000:04:00.0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:        138     271356     264446     261050   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
PND:          0          0          0          0   Performance pending work
RES:       4511       9275       8470       8086   Rescheduling interrupts
CAL:       3624       8666        523       4543   Function call interrupts
TLB:        981       1111       1065       1058   TLB shootdowns
ERR:          0
MIS:          0

[ Ingo, Venki and Shaohua added to cc: see the whole thread on lkml for
details, but Mark is basically chasing down a situation where the floppy driver seems to have trouble formatting floppies, and it happened between 2.6.27 and .28. The trouble seems to be that a DMA transfer of a memory block transfers the wrong value for the first byte of the block.

Which should be impossible, but whatever. Some part of the system has a cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot reproduce the bug if HPET is disabled in the BIOS or by using 'nohpet'. He found that out by pure luck while bisecting, because some time during his bisect, his machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_ to work. But 2.6.28 (and current -git) does not. Any ideas? ]

Quoted 38 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] <[email protected]>
Date:   Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible. We retain the HPET timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when legacy mode is being used. We setup the remaining HPET timers as per CPU MSI based timers. This per CPU timer will eliminate the need for timer broadcasting with IRQ 0 when there is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available timers, CPUs that do not find any timer to use will continue using LAPIC and IRQ 0 broadcast.

Signed-off-by Venkatesh Pallipadi <[email protected]>

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not boot if I had hpet enabled. To get this one to boot (single user mode only) I had to add the the quiet cmdline option and following patch from to arch/x86/kernel/hpet.c

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct hpet_dev *dev)
{

        if (request_irq(dev->irq, hpet_interrupt_handler,

(quoted diff) +1 -1

-                       IRQF_SHARED|IRQF_NOBALANCING, dev->name, dev))
+                       IRQF_DISABLED|IRQF_NOBALANCING, dev->name, dev))
                return -1;

Quoted 3 lines

        disable_irq(dev->irq);

AND add the quiet cmdline option.

Ok, so we know why HPET didn't boot for you, and that was fixed later (by that 5ceb1a04). But is this also when the floppy started mis-behaving?

IOW, _if_ you boot with that fix from commit 5ceb1a04 (and the quiet option - I wonder what that is about: do you have any ideas?), is the per-CPU HPET timer commit also the commit that causes floppy problems, or is this purely a "bisect when HPET became a boot-up problem"?

---

Quoted 35 lines

Also, of all the machines it does work on with hpets enabled, I don't see the HPET2 in /proc/interupts as below.

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:         82          0          3          0   IO-APIC-edge      timer
  1:          0          0       1712          6   IO-APIC-edge      i8042
  3:          0          0          6          0   IO-APIC-edge
  4:          0          0          6          0   IO-APIC-edge
  6:          0          0          4          0   IO-APIC-edge      floppy
  8:          0          0         60          0   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0      37798        179   IO-APIC-edge      i8042
 14:          0          0      16462         71   IO-APIC-edge      pata_atiixp
 15:          0          0       5713         17   IO-APIC-edge      pata_atiixp
 16:          0          0        904          2   IO-APIC-fasteoi   aic79xx, ohci_hcd:usb2, ohci_hcd:usb4, HDA Intel, ni-pci-gpib
 17:          0          0          2          0   IO-APIC-fasteoi   ehci_hcd:usb1, parport0, ni-pci-gpib
 18:          0          0      49940         90   IO-APIC-fasteoi   ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, nvidia
 19:          0          0        703          2   IO-APIC-fasteoi   aic7xxx, ehci_hcd:usb3, ttySLG0, eth1
 22:          0          0       1303         15   IO-APIC-fasteoi   ahci

24: 261763 0 0 0 HPET_MSI-edge hpet2

 29:          0          0        220          5   PCI-MSI-edge      sky2@pci:0000:04:00.0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:        138     271356     264446     261050   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
PND:          0          0          0          0   Performance pending work
RES:       4511       9275       8470       8086   Rescheduling interrupts
CAL:       3624       8666        523       4543   Function call interrupts
TLB:        981       1111       1065       1058   TLB shootdowns
ERR:          0
MIS:          0

Quoted 61 lines · Linus Torvalds

[ Ingo, Venki and Shaohua added to cc: see the whole thread on lkml for
details, but Mark is basically chasing down a situation where the floppy driver seems to have trouble formatting floppies, and it happened between 2.6.27 and .28. The trouble seems to be that a DMA transfer of a memory block transfers the wrong value for the first byte of the block.

Which should be impossible, but whatever. Some part of the system has a cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot reproduce the bug if HPET is disabled in the BIOS or by using 'nohpet'. He found that out by pure luck while bisecting, because some time during his bisect, his machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_ to work. But 2.6.28 (and current -git) does not. Any ideas? ]

Quoted 41 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] <[email protected]>
Date:   Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible. We retain the HPET timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when legacy mode is being used. We setup the remaining HPET timers as per CPU MSI based timers. This per CPU timer will eliminate the need for timer broadcasting with IRQ 0 when there is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available timers, CPUs that do not find any timer to use will continue using LAPIC and IRQ 0 broadcast.

Signed-off-by Venkatesh Pallipadi <[email protected]>

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not boot if I had hpet enabled. To get this one to boot (single user mode only) I had to add the the quiet cmdline option and following patch from to arch/x86/kernel/hpet.c

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct hpet_dev *dev)
{

        if (request_irq(dev->irq, hpet_interrupt_handler,

(quoted diff) +1 -1

-                       IRQF_SHARED|IRQF_NOBALANCING, dev->name, dev))
+                       IRQF_DISABLED|IRQF_NOBALANCING, dev->name, dev))
                return -1;
 
        disable_irq(dev->irq);

AND add the quiet cmdline option.

Quoted 3 lines

Ok, so we know why HPET didn't boot for you, and that was fixed later (by that 5ceb1a04). But is this also when the floppy started mis-behaving?

Commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0 is when the floppy stops working
and also when I could no longer boot with hpet enabled.

Quoted 66 lines · Mark Hounschell

Quoted 64 lines · Linus Torvalds

[ Ingo, Venki and Shaohua added to cc: see the whole thread on lkml for
details, but Mark is basically chasing down a situation where the floppy driver seems to have trouble formatting floppies, and it happened between 2.6.27 and .28. The trouble seems to be that a DMA transfer of a memory block transfers the wrong value for the first byte of the block.

Which should be impossible, but whatever. Some part of the system has a cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot reproduce the bug if HPET is disabled in the BIOS or by using 'nohpet'. He found that out by pure luck while bisecting, because some time during his bisect, his machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_ to work. But 2.6.28 (and current -git) does not. Any ideas? ]

Quoted 40 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] <[email protected]>
Date:   Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible. We retain the HPET timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when legacy mode is being used. We setup the remaining HPET timers as per CPU MSI based timers. This per CPU timer will eliminate the need for timer broadcasting with IRQ 0 when there is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available timers, CPUs that do not find any timer to use will continue using LAPIC and IRQ 0 broadcast.

Signed-off-by Venkatesh Pallipadi <[email protected]>

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not boot if I had hpet enabled. To get this one to boot (single user mode only) I had to add the the quiet cmdline option and following patch from to arch/x86/kernel/hpet.c

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct hpet_dev *dev)
{

        if (request_irq(dev->irq, hpet_interrupt_handler,

(quoted diff) +1 -1

-                       IRQF_SHARED|IRQF_NOBALANCING, dev->name, dev))
+                       IRQF_DISABLED|IRQF_NOBALANCING, dev->name, dev))
                return -1;
 
        disable_irq(dev->irq);

Quoted 1 line

AND add the quiet cmdline option.

Ok, so we know why HPET didn't boot for you, and that was fixed later (by that 5ceb1a04). But is this also when the floppy started mis-behaving?

Quoted 3 lines

Commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0 is when the floppy stops working
and also when I could no longer boot with hpet enabled.

I am missing something here. Commit 26afe5f2 is where system does not boot with HPET or is it where the floppy stops working when you boot with HPET enabled.

Can you try "idle=halt" with both .27 and .28 with /proc/interrupts output in each case. With that option, we should be using local APIC timer and PIT, HPET or HPET with MSI should not really matter. Does it still fail with .28 with that option?

Quoted 67 lines · Venkatesh

Quoted 66 lines · Mark Hounschell

Quoted 64 lines · Linus Torvalds

[ Ingo, Venki and Shaohua added to cc: see the whole thread on lkml for
details, but Mark is basically chasing down a situation where the floppy driver seems to have trouble formatting floppies, and it happened between 2.6.27 and .28. The trouble seems to be that a DMA transfer of a memory block transfers the wrong value for the first byte of the block.

Which should be impossible, but whatever. Some part of the system has a cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot reproduce the bug if HPET is disabled in the BIOS or by using 'nohpet'. He found that out by pure luck while bisecting, because some time during his bisect, his machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_ to work. But 2.6.28 (and current -git) does not. Any ideas? ]

Quoted 40 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] <[email protected]>
Date:   Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible. We retain the HPET timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when legacy mode is being used. We setup the remaining HPET timers as per CPU MSI based timers. This per CPU timer will eliminate the need for timer broadcasting with IRQ 0 when there is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available timers, CPUs that do not find any timer to use will continue using LAPIC and IRQ 0 broadcast.

Signed-off-by Venkatesh Pallipadi <[email protected]>

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not boot if I had hpet enabled. To get this one to boot (single user mode only) I had to add the the quiet cmdline option and following patch from to arch/x86/kernel/hpet.c

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct hpet_dev *dev)
{

        if (request_irq(dev->irq, hpet_interrupt_handler,

(quoted diff) +1 -1

-                       IRQF_SHARED|IRQF_NOBALANCING, dev->name, dev))
+                       IRQF_DISABLED|IRQF_NOBALANCING, dev->name, dev))
                return -1;
 
        disable_irq(dev->irq);

Quoted 1 line

AND add the quiet cmdline option.

Ok, so we know why HPET didn't boot for you, and that was fixed later (by that 5ceb1a04). But is this also when the floppy started mis-behaving?

Quoted 9 lines

Quoted 3 lines

Commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0 is when the floppy stops working
and also when I could no longer boot with hpet enabled.

I am missing something here. Commit 26afe5f2 is where system does not boot with HPET or is it where the floppy stops working when you boot with HPET enabled.

As it happens, both happen there. Commit 5ceb1a04 is where it starts booting _again_ with hpet enabled. So I took that patch (5ceb1a04) and applied it to (26afe5f2f) to be able to boot with hpet enabled. I had to use the quiet option to get to a login prompt, but there is where the floppy format first fails, just as it does in 2.6.28 and up.

Quoted 5 lines

Can you try "idle=halt" with both .27 and .28 with /proc/interrupts output in each case. With that option, we should be using local APIC timer and PIT, HPET or HPET with MSI should not really matter. Does it still fail with .28 with that option?

-----Original Message-----
From: Mark Hounschell [mailto:[email protected]]
Sent: Wednesday, December 23, 2009 5:03 AM
To: Pallipadi, Venkatesh
Cc: [email protected]; Linus Torvalds; Alain Knaff; Linux Kernel Mailing List; [email protected]; Li, Shaohua; Ingo Molnar
Subject: Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28 (Was: Re: Cannot format floppies under kernel 2.6.*?)

Quoted 5 lines · Mark Hounschell

Quoted 4 lines · Venkatesh

Quoted 3 lines · Mark Hounschell

Quoted 2 lines · Linus Torvalds

[ Ingo, Venki and Shaohua added to cc: see the whole

thread on lkml for

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

details, but Mark is basically chasing down a situation

where the floppy

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

driver seems to have trouble formatting floppies, and

it happened

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

between 2.6.27 and .28. The trouble seems to be that a

DMA transfer of a

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

memory block transfers the wrong value for the first

byte of the block.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Which should be impossible, but whatever. Some part of

the system has a

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot

reproduce the bug if

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

HPET is disabled in the BIOS or by using 'nohpet'. He

found that out by

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

pure luck while bisecting, because some time during his

bisect, his

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_

to work. But

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

2.6.28 (and current -git) does not. Any ideas? ]

Quoted 7 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] 

<[email protected]>

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Date: Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible.

We retain the HPET

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when

legacy mode is being used. We

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

setup the remaining HPET timers as per CPU MSI based

timers. This per CPU

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

timer will eliminate the need for timer broadcasting

with IRQ 0 when there

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available

timers, CPUs that do not

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

find any timer to use will continue using LAPIC and

IRQ 0 broadcast.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Signed-off-by Venkatesh Pallipadi

<[email protected]>

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not

boot if I had hpet

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

enabled. To get this one to boot (single user mode only)

I had to add the

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

the quiet cmdline option and following patch from to

arch/x86/kernel/hpet.c

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct

hpet_dev *dev)

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

{

        if (request_irq(dev->irq, hpet_interrupt_handler,

- IRQF_SHARED|IRQF_NOBALANCING,

dev->name, dev))

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

+ IRQF_DISABLED|IRQF_NOBALANCING,

dev->name, dev))

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 5 lines

                return -1;

        disable_irq(dev->irq);

AND add the quiet cmdline option.

Ok, so we know why HPET didn't boot for you, and that was

fixed later (by

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

that 5ceb1a04). But is this also when the floppy started

mis-behaving?

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0 is when

the floppy stops

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

working
and also when I could no longer boot with hpet enabled.

I am missing something here. Commit 26afe5f2 is where

system does not

Quoted 6 lines

Quoted 3 lines

boot with HPET or is it where the floppy stops working when you boot with HPET enabled.

As it happens, both happen there. Commit 5ceb1a04 is where it starts booting _again_ with hpet enabled. So I took that patch

(5ceb1a04) and

Quoted 1 line

applied it to (26afe5f2f) to be able to boot with hpet

enabled. I had to

Quoted 6 lines

use the quiet option to get to a login prompt, but there is where the floppy format first fails, just as it does in 2.6.28 and up.

Quoted 3 lines

Can you try "idle=halt" with both .27 and .28 with /proc/interrupts output in each case. With that option, we should be using local APIC timer and PIT, HPET or HPET with MSI should not really

matter. Does it

Quoted 2 lines

Quoted 2 lines

still fail with .28 with that option?

2.6.28 still fails with that option.

2.6.27.41 /proc/interrupts with idle=halt

CPU0 CPU1 CPU2 CPU3
0: 126 0 0 1
IO-APIC-edge timer
1: 0 0 1 157
IO-APIC-edge i8042

 3:          0          0          0          6   IO-APIC-edge
 4:          0          0          0          6   IO-APIC-edge
 6:          0          0          0          4   

IO-APIC-edge floppy
8: 0 0 0 1
IO-APIC-edge rtc0
9: 0 0 0 0
IO-APIC-fasteoi acpi
12: 0 0 1 128
IO-APIC-edge i8042
14: 0 0 34 4457 IO-APIC-edge
pata_atiixp
15: 0 0 4 480 IO-APIC-edge
pata_atiixp
16: 0 0 0 397 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel
17: 0 0 0 2 IO-APIC-fasteoi
ehci_hcd:usb1
18: 0 0 0 0 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
19: 0 0 0 142 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1
22: 0 0 4 1154
IO-APIC-fasteoi ahci
219: 0 0 3 63
PCI-MSI-edge eth0
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 91539 91964 92525 91181 Local timer
interrupts
RES: 2888 3873 2434 2721
Rescheduling interrupts
CAL: 240 245 247 84 function
call interrupts

TLB:        768        628        526        512   TLB shootdowns
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

2.6.28 /proc/interrupts with idle=halt

CPU0 CPU1 CPU2 CPU3
0: 126 0 2 0
IO-APIC-edge timer
1: 0 0 192 0
IO-APIC-edge i8042

 3:          0          0          6          0   IO-APIC-edge
 4:          0          0          6          0   IO-APIC-edge
 6:          0          0          4          0   

IO-APIC-edge floppy
8: 0 0 1 0
IO-APIC-edge rtc0
9: 0 0 0 0
IO-APIC-fasteoi acpi
12: 0 0 128 1
IO-APIC-edge i8042
14: 0 1 147114 396 IO-APIC-edge
pata_atiixp
15: 0 0 646 2 IO-APIC-edge
pata_atiixp
16: 0 0 396 0 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb2, ohci_hcd:usb4, HDA Intel
17: 0 0 0 0 IO-APIC-fasteoi
ehci_hcd:usb1
18: 0 0 0 0 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
19: 0 0 362 1 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb3, ttySLG0, eth1
22: 0 0 874 1
IO-APIC-fasteoi ahci
1274: 0 0 193 4
PCI-MSI-edge eth0
1279: 513207 0 0 0
HPET_MSI-edge hpet2
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 268 513395 513138 522088 Local timer
interrupts
RES: 3262 3679 2573 3746
Rescheduling interrupts
CAL: 131 166 57 147 Function
call interrupts

TLB:        680        438        450        639   TLB shootdowns
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

Quoted 224 lines

-----Original Message-----
From: Mark Hounschell [mailto:[email protected]]
Sent: Wednesday, December 23, 2009 5:03 AM
To: Pallipadi, Venkatesh
Cc: [email protected]; Linus Torvalds; Alain Knaff; Linux Kernel Mailing List; [email protected]; Li, Shaohua; Ingo Molnar
Subject: Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28 (Was: Re: Cannot format floppies under kernel 2.6.*?)

Quoted 5 lines · Mark Hounschell

Quoted 4 lines · Venkatesh

Quoted 3 lines · Mark Hounschell

Quoted 2 lines · Linus Torvalds

[ Ingo, Venki and Shaohua added to cc: see the whole

thread on lkml for

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

details, but Mark is basically chasing down a situation

where the floppy

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

driver seems to have trouble formatting floppies, and

it happened

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

between 2.6.27 and .28. The trouble seems to be that a

DMA transfer of a

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

memory block transfers the wrong value for the first

byte of the block.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Which should be impossible, but whatever. Some part of

the system has a

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot

reproduce the bug if

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

HPET is disabled in the BIOS or by using 'nohpet'. He

found that out by

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

pure luck while bisecting, because some time during his

bisect, his

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_

to work. But

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

2.6.28 (and current -git) does not. Any ideas? ]

Quoted 7 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] 

<[email protected]>

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Date: Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible.

We retain the HPET

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when

legacy mode is being used. We

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

setup the remaining HPET timers as per CPU MSI based

timers. This per CPU

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

timer will eliminate the need for timer broadcasting

with IRQ 0 when there

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available

timers, CPUs that do not

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

find any timer to use will continue using LAPIC and

IRQ 0 broadcast.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Signed-off-by Venkatesh Pallipadi

<[email protected]>

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not

boot if I had hpet

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

enabled. To get this one to boot (single user mode only)

I had to add the

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

the quiet cmdline option and following patch from to

arch/x86/kernel/hpet.c

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct

hpet_dev *dev)

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

{

        if (request_irq(dev->irq, hpet_interrupt_handler,

- IRQF_SHARED|IRQF_NOBALANCING,

dev->name, dev))

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

+ IRQF_DISABLED|IRQF_NOBALANCING,

dev->name, dev))

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 5 lines

                return -1;

        disable_irq(dev->irq);

AND add the quiet cmdline option.

Ok, so we know why HPET didn't boot for you, and that was

fixed later (by

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

that 5ceb1a04). But is this also when the floppy started

mis-behaving?

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0 is when

the floppy stops

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

working
and also when I could no longer boot with hpet enabled.

I am missing something here. Commit 26afe5f2 is where

system does not

Quoted 6 lines

Quoted 3 lines

boot with HPET or is it where the floppy stops working when you boot with HPET enabled.

As it happens, both happen there. Commit 5ceb1a04 is where it starts booting _again_ with hpet enabled. So I took that patch

(5ceb1a04) and

Quoted 1 line

applied it to (26afe5f2f) to be able to boot with hpet

enabled. I had to

Quoted 6 lines

use the quiet option to get to a login prompt, but there is where the floppy format first fails, just as it does in 2.6.28 and up.

Quoted 3 lines

Can you try "idle=halt" with both .27 and .28 with /proc/interrupts output in each case. With that option, we should be using local APIC timer and PIT, HPET or HPET with MSI should not really

matter. Does it

Quoted 2 lines

Quoted 2 lines

still fail with .28 with that option?

2.6.28 still fails with that option.

2.6.27.41 /proc/interrupts with idle=halt

CPU0 CPU1 CPU2 CPU3
0: 126 0 0 1
IO-APIC-edge timer
1: 0 0 1 157
IO-APIC-edge i8042

 3:          0          0          0          6   IO-APIC-edge
 4:          0          0          0          6   IO-APIC-edge
 6:          0          0          0          4   

IO-APIC-edge floppy
8: 0 0 0 1
IO-APIC-edge rtc0
9: 0 0 0 0
IO-APIC-fasteoi acpi
12: 0 0 1 128
IO-APIC-edge i8042
14: 0 0 34 4457 IO-APIC-edge
pata_atiixp
15: 0 0 4 480 IO-APIC-edge
pata_atiixp
16: 0 0 0 397 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel
17: 0 0 0 2 IO-APIC-fasteoi
ehci_hcd:usb1
18: 0 0 0 0 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
19: 0 0 0 142 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1
22: 0 0 4 1154
IO-APIC-fasteoi ahci
219: 0 0 3 63
PCI-MSI-edge eth0
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 91539 91964 92525 91181 Local timer
interrupts
RES: 2888 3873 2434 2721
Rescheduling interrupts
CAL: 240 245 247 84 function
call interrupts

TLB:        768        628        526        512   TLB shootdowns
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

2.6.28 /proc/interrupts with idle=halt

CPU0 CPU1 CPU2 CPU3
0: 126 0 2 0
IO-APIC-edge timer
1: 0 0 192 0
IO-APIC-edge i8042

 3:          0          0          6          0   IO-APIC-edge
 4:          0          0          6          0   IO-APIC-edge
 6:          0          0          4          0   

IO-APIC-edge floppy
8: 0 0 1 0
IO-APIC-edge rtc0
9: 0 0 0 0
IO-APIC-fasteoi acpi
12: 0 0 128 1
IO-APIC-edge i8042
14: 0 1 147114 396 IO-APIC-edge
pata_atiixp
15: 0 0 646 2 IO-APIC-edge
pata_atiixp
16: 0 0 396 0 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb2, ohci_hcd:usb4, HDA Intel
17: 0 0 0 0 IO-APIC-fasteoi
ehci_hcd:usb1
18: 0 0 0 0 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
19: 0 0 362 1 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb3, ttySLG0, eth1
22: 0 0 874 1
IO-APIC-fasteoi ahci
1274: 0 0 193 4
PCI-MSI-edge eth0
1279: 513207 0 0 0
HPET_MSI-edge hpet2
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 268 513395 513138 522088 Local timer
interrupts
RES: 3262 3679 2573 3746
Rescheduling interrupts
CAL: 131 166 57 147 Function
call interrupts

Quoted 5 lines

TLB:        680        438        450        639   TLB shootdowns
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

Hmm. Looks like hpet2 is still getting used instead of local APIC timer in .28 case.

I was expecting some low number in hpet2 and local timer on all CPU to be around the same value. Above shows CPU 0 is depending on hpet2 for some reason even with idle=halt. Can you send the output of below two in case of .28 /proc/timer_list

grep . /sys/devices/system/cpu/cpu0/cpuidle/*/*

Quoted 230 lines · Venkatesh

Quoted 224 lines

-----Original Message-----
From: Mark Hounschell [mailto:[email protected]]
Sent: Wednesday, December 23, 2009 5:03 AM
To: Pallipadi, Venkatesh
Cc: [email protected]; Linus Torvalds; Alain Knaff; Linux Kernel Mailing List; [email protected]; Li, Shaohua; Ingo Molnar
Subject: Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28 (Was: Re: Cannot format floppies under kernel 2.6.*?)

Quoted 5 lines · Mark Hounschell

Quoted 4 lines · Venkatesh

Quoted 3 lines · Mark Hounschell

Quoted 2 lines · Linus Torvalds

[ Ingo, Venki and Shaohua added to cc: see the whole

thread on lkml for

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

details, but Mark is basically chasing down a situation

where the floppy

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

driver seems to have trouble formatting floppies, and

it happened

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

between 2.6.27 and .28. The trouble seems to be that a

DMA transfer of a

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

memory block transfers the wrong value for the first

byte of the block.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Which should be impossible, but whatever. Some part of

the system has a

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

cached buffer that isn't flushed.

What gets _you_ guys involved is that Mark cannot

reproduce the bug if

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

HPET is disabled in the BIOS or by using 'nohpet'. He

found that out by

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

pure luck while bisecting, because some time during his

bisect, his

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

machine wouldn't even boot with HPET.

So the problem is: with HPET enabled, 2.6.27.4 _used_

to work. But

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

2.6.28 (and current -git) does not. Any ideas? ]

Quoted 7 lines · Mark Hounschell

Ok, I may have something that might help.

# git bisect bad
26afe5f2fbf06ea0765aaa316640c4dd472310c0 is the first bad commit
commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0
Author: [email protected] 

<[email protected]>

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Date: Fri Sep 5 18:02:18 2008 -0700

x86: HPET_MSI Initialise per-cpu HPET timers

Initialize a per CPU HPET MSI timer when possible.

We retain the HPET

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

timer 0 (IRQ 0) and timer 1 (IRQ 8) as is when

legacy mode is being used. We

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

setup the remaining HPET timers as per CPU MSI based

timers. This per CPU

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

timer will eliminate the need for timer broadcasting

with IRQ 0 when there

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

is non-functional LAPIC timer across CPU deep C-states.

If there are more CPUs than number of available

timers, CPUs that do not

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

find any timer to use will continue using LAPIC and

IRQ 0 broadcast.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Signed-off-by Venkatesh Pallipadi

<[email protected]>

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Signed-off-by Shaohua Li <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

And of coarse this was the first commit that I could not

boot if I had hpet

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

enabled. To get this one to boot (single user mode only)

I had to add the

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

the quiet cmdline option and following patch from to

arch/x86/kernel/hpet.c

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

commit 5ceb1a04187553e08c6ab60d30cee7c454ee139a

@ -445,7 +445,7 @@ static int hpet_setup_irq(struct

hpet_dev *dev)

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

{

        if (request_irq(dev->irq, hpet_interrupt_handler,

- IRQF_SHARED|IRQF_NOBALANCING,

dev->name, dev))

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

+ IRQF_DISABLED|IRQF_NOBALANCING,

dev->name, dev))

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 5 lines

                return -1;

        disable_irq(dev->irq);

AND add the quiet cmdline option.

Ok, so we know why HPET didn't boot for you, and that was

fixed later (by

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

that 5ceb1a04). But is this also when the floppy started

mis-behaving?

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Commit 26afe5f2fbf06ea0765aaa316640c4dd472310c0 is when

the floppy stops

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

working
and also when I could no longer boot with hpet enabled.

I am missing something here. Commit 26afe5f2 is where

system does not

Quoted 6 lines

Quoted 3 lines

boot with HPET or is it where the floppy stops working when you boot with HPET enabled.

As it happens, both happen there. Commit 5ceb1a04 is where it starts booting _again_ with hpet enabled. So I took that patch

(5ceb1a04) and

Quoted 1 line

applied it to (26afe5f2f) to be able to boot with hpet

enabled. I had to

Quoted 6 lines

use the quiet option to get to a login prompt, but there is where the floppy format first fails, just as it does in 2.6.28 and up.

Quoted 3 lines

Can you try "idle=halt" with both .27 and .28 with /proc/interrupts output in each case. With that option, we should be using local APIC timer and PIT, HPET or HPET with MSI should not really

matter. Does it

Quoted 2 lines

Quoted 2 lines

still fail with .28 with that option?

2.6.28 still fails with that option.

2.6.27.41 /proc/interrupts with idle=halt

CPU0 CPU1 CPU2 CPU3
0: 126 0 0 1
IO-APIC-edge timer
1: 0 0 1 157
IO-APIC-edge i8042

 3:          0          0          0          6   IO-APIC-edge
 4:          0          0          0          6   IO-APIC-edge
 6:          0          0          0          4   

IO-APIC-edge floppy
8: 0 0 0 1
IO-APIC-edge rtc0
9: 0 0 0 0
IO-APIC-fasteoi acpi
12: 0 0 1 128
IO-APIC-edge i8042
14: 0 0 34 4457 IO-APIC-edge
pata_atiixp
15: 0 0 4 480 IO-APIC-edge
pata_atiixp
16: 0 0 0 397 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel
17: 0 0 0 2 IO-APIC-fasteoi
ehci_hcd:usb1
18: 0 0 0 0 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
19: 0 0 0 142 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1
22: 0 0 4 1154
IO-APIC-fasteoi ahci
219: 0 0 3 63
PCI-MSI-edge eth0
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 91539 91964 92525 91181 Local timer
interrupts
RES: 2888 3873 2434 2721
Rescheduling interrupts
CAL: 240 245 247 84 function
call interrupts

TLB:        768        628        526        512   TLB shootdowns
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

2.6.28 /proc/interrupts with idle=halt

CPU0 CPU1 CPU2 CPU3
0: 126 0 2 0
IO-APIC-edge timer
1: 0 0 192 0
IO-APIC-edge i8042

 3:          0          0          6          0   IO-APIC-edge
 4:          0          0          6          0   IO-APIC-edge
 6:          0          0          4          0   

IO-APIC-edge floppy
8: 0 0 1 0
IO-APIC-edge rtc0
9: 0 0 0 0
IO-APIC-fasteoi acpi
12: 0 0 128 1
IO-APIC-edge i8042
14: 0 1 147114 396 IO-APIC-edge
pata_atiixp
15: 0 0 646 2 IO-APIC-edge
pata_atiixp
16: 0 0 396 0 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb2, ohci_hcd:usb4, HDA Intel
17: 0 0 0 0 IO-APIC-fasteoi
ehci_hcd:usb1
18: 0 0 0 0 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
19: 0 0 362 1 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb3, ttySLG0, eth1
22: 0 0 874 1
IO-APIC-fasteoi ahci
1274: 0 0 193 4
PCI-MSI-edge eth0
1279: 513207 0 0 0
HPET_MSI-edge hpet2
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 268 513395 513138 522088 Local timer
interrupts
RES: 3262 3679 2573 3746
Rescheduling interrupts
CAL: 131 166 57 147 Function
call interrupts

Quoted 5 lines

TLB:        680        438        450        639   TLB shootdowns
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

Quoted 4 lines

Hmm. Looks like hpet2 is still getting used instead of local APIC timer in .28 case.

I was expecting some low number in hpet2 and local timer on all CPU to be around the same value. Above shows CPU 0 is depending on hpet2 for some reason even with idle=halt. Can you send the output of below two in case of .28 /proc/timer_list

Attached.

Quoted 1 line

grep . /sys/devices/system/cpu/cpu0/cpuidle/*/*

I have no /sys/devices/system/cpu/cpu0/cpuidle on this machine. Maybe because of

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set

Would it be OK if when you ask for 2.6.28 info, I use a 2.6.32.2 kernel? That kernel also fails fdformat with hpet enabled on these machines.

Quoted 8 lines

Hmm. Looks like hpet2 is still getting used instead of local APIC timer in .28 case.

I was expecting some low number in hpet2 and local timer on all CPU to be around the same value. Above shows CPU 0 is depending on hpet2 for some reason even with idle=halt. Can you send the output of below two in case of .28 /proc/timer_list

Attached.

It's not using the lapic for CPU0.

Using the HPET as a per-cpu timer is some crazy sh*t, since it's pretty expensive to reprogram (compared to the local apic). And having different timers for different CPU's is just odd.

The fact that the timer subsystem can do this and it all (mostly) works at all is nice and impressive, but doesn't make it any less crazy ;)

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer broadcasts when needed to wake the other CPUs up from deep C, but for that it has to run with HPET. At least the other ones can still enjoy the LAPIC timer.

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

Quoted 1 line · Andi Kleen

Quoted 5 lines

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer broadcasts when needed to wake the other CPUs up from deep C, but for that it has to run with HPET. At least the other ones can still enjoy the LAPIC timer.

Ahh, ok, that makes sense. I was assuming the broadcast timer would act in that capacity, but..

Quoted 4 lines

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

We have indeed had historical issues with floppy and sleep states before.

-----Original Message-----
From: Linus Torvalds [mailto:[email protected]]
Sent: Wednesday, December 23, 2009 8:50 AM
To: Andi Kleen
Cc: Mark Hounschell; Pallipadi, Venkatesh; [email protected]; Alain Knaff; Linux Kernel Mailing List; [email protected]; Li, Shaohua; Ingo Molnar
Subject: Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28

Quoted 3 lines · Andi Kleen

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer

broadcasts

Quoted 1 line

when needed to wake the other CPUs up from deep C, but for

that it has

Quoted 2 lines

to run with HPET. At least the other ones can still enjoy the LAPIC timer.

Ahh, ok, that makes sense. I was assuming the broadcast timer would act in
that capacity, but..

Quoted 5 lines

Quoted 5 lines

HPET_MSI-edge hpet2
NMI: 0 0 0 0
Non-maskable interrupts
LOC: 268 513395 513138 522088 Local timer
interrupts

This is what I was thining yday and asked Mark to try idle=halt. This /proc/interrupts is with idle=halt when there should not be any C-states and broadcasts involved.

Quoted 8 lines · Linus Torvalds

It's not using the lapic for CPU0.

Using the HPET as a per-cpu timer is some crazy sh*t, since it's pretty expensive to reprogram (compared to the local apic). And having different timers for different CPU's is just odd.

The fact that the timer subsystem can do this and it all (mostly) works at all is nice and impressive, but doesn't make it any less crazy ;)

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer broadcasts when needed to wake the other CPUs up from deep C, but for that it has to run with HPET. At least the other ones can still enjoy the LAPIC timer.

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

I'll try the patch Linus provided now.

Quoted 2 lines · Mark Hounschell

I'll try the patch Linus provided now.

I doubt it matters - because if it did, it would matter for everybody, and the HPET thing shouldn't make any difference at all.

[ Or rather, it should matter for everybody trying to format a specific
format (without interleave it won't matter, and not all formats have any interleave - I think it was mainly used on 5.25" floppies and special formats). ]

Besides, maybe I was just mis-reading the code.

But getting some testing for the patch certainly won't hurt, so I'm not going to argue against it any more ;)

Quoted 17 lines · Andi Kleen

Quoted 8 lines · Linus Torvalds

It's not using the lapic for CPU0.

Using the HPET as a per-cpu timer is some crazy sh*t, since it's pretty expensive to reprogram (compared to the local apic). And having different timers for different CPU's is just odd.

The fact that the timer subsystem can do this and it all (mostly) works at all is nice and impressive, but doesn't make it any less crazy ;)

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer broadcasts when needed to wake the other CPUs up from deep C, but for that it has to run with HPET. At least the other ones can still enjoy the LAPIC timer.

Quoted 3 lines

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

I just did and /proc/interrupts looks the same and the floppy still does not format.

Quoted 18 lines · Mark Hounschell

Quoted 17 lines · Andi Kleen

Quoted 8 lines · Linus Torvalds

It's not using the lapic for CPU0.

Using the HPET as a per-cpu timer is some crazy sh*t, since it's pretty expensive to reprogram (compared to the local apic). And having different timers for different CPU's is just odd.

The fact that the timer subsystem can do this and it all (mostly) works at all is nice and impressive, but doesn't make it any less crazy ;)

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer broadcasts when needed to wake the other CPUs up from deep C, but for that it has to run with HPET. At least the other ones can still enjoy the LAPIC timer.

Quoted 7 lines

Quoted 3 lines

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

I just did and /proc/interrupts looks the same and the floppy still does not format.

Can you try this one line patch either on .28 or .32 (with /proc/interrupts output).
This disables hpet2 and lapic timer should then be used on CPU 0. If things work with this test patch, we will know that the failure is somehow related to HPET usage in MSI mode.

Thanks,
Venki

Reduce the rating of percpu hpet timer

Signed-off-by Venkatesh Pallipadi <[email protected]>

---
 arch/x86/kernel/hpet.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

arch/x86/kernel/hpet.c +0 -0

index cafb1c6..f89d17a 100644

drivers/block/floppy.c +1 -1

index 3266b4f..9c9148c 100644
@@ -2237,13 +2237,10 @@ static void setup_format_params(int track)
 	for (count = 1; count <= F_SECT_PER_TRACK; ++count) {
 		here[n].sect = count;
 		n = (n + il) % F_SECT_PER_TRACK;
-		if (here[n].sect) {	/* sector busy, find next free sector */
+		while (here[n].sect) {	/* sector busy, find next free sector */

Quoted 48 lines · Venkatesh

Quoted 18 lines · Mark Hounschell

Quoted 17 lines · Andi Kleen

Quoted 8 lines · Linus Torvalds

It's not using the lapic for CPU0.

Using the HPET as a per-cpu timer is some crazy sh*t, since it's pretty expensive to reprogram (compared to the local apic). And having different timers for different CPU's is just odd.

The fact that the timer subsystem can do this and it all (mostly) works at all is nice and impressive, but doesn't make it any less crazy ;)

I suspect it's a system where the APIC timer stops in deeper idle states and it supports them. In this case CPU #0 does timer broadcasts when needed to wake the other CPUs up from deep C, but for that it has to run with HPET. At least the other ones can still enjoy the LAPIC timer.

Quoted 7 lines

Quoted 3 lines

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

I just did and /proc/interrupts looks the same and the floppy still does not format.

Can you try this one line patch either on .28 or .32 (with /proc/interrupts output).
This disables hpet2 and lapic timer should then be used on CPU 0. If things work with this test patch, we will know that the failure is somehow related to HPET usage in MSI mode.

Thanks,
Venki

Reduce the rating of percpu hpet timer

Signed-off-by Venkatesh Pallipadi <[email protected]>

---
 arch/x86/kernel/hpet.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

arch/x86/kernel/hpet.c +0 -0

index cafb1c6..f89d17a 100644

arch/x86/kernel/hpet.c static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu) · line 486

Quoted 8 lines

arch/x86/kernel/hpet.c +1 -1

@@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
 	hpet_setup_irq(hdev);
 	evt->irq = hdev->irq;
 
-	evt->rating = 110;
+	evt->rating = 40;
 	evt->features = CLOCK_EVT_FEAT_ONESHOT;
 	if (hdev->flags & HPET_DEV_PERI_CAP)
 		evt->features |= CLOCK_EVT_FEAT_PERIODIC;

That made it work. Used 2.6.32.2

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:         82          0          0          1   IO-APIC-edge      timer
  1:          0          0          0         67   IO-APIC-edge      i8042
  3:          0          0          0          6   IO-APIC-edge
  4:          0          0          0          4   IO-APIC-edge
  6:          0          0          0          4   IO-APIC-edge      floppy
  8:          0          0          0          8   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0         10       1519   IO-APIC-edge      i8042
 14:          0          0         39      10995   IO-APIC-edge

pata_atiixp
15: 0 0 3 391 IO-APIC-edge
pata_atiixp
16: 0 0 2 606 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
17: 0 0 0 3 IO-APIC-fasteoi
ehci_hcd:usb1, parport0, ni-pci-gpib
18: 0 0 10 2168 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
19: 0 0 0 130 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1

 22:          0          0          8       1151   IO-APIC-fasteoi   ahci
 24:          0          0          0          0  HPET_MSI-edge      hpet2
 29:          0          0          0         48   PCI-MSI-edge
sky2@pci:0000:04:00.0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      34842      30177      29672      29632   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring
interrupts
PND:          0          0          0          0   Performance pending work
RES:      17501      20449      16670      11224   Rescheduling interrupts
CAL:      10554       2336       1102       1071   Function call interrupts
TLB:        364        562        753        468   TLB shootdowns
ERR:          0
MIS:          0

# fdformat /dev/fd0u1440
Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
Formatting ... done
Verifying ... done

MSI interrupt being delivered to CPU 0. I cannot think of any reason why this can break dma. We can probably try adding some dummy HPET read after dma write, to see if that flushes things properly.

Quoted 3 lines · Venkatesh

MSI interrupt being delivered to CPU 0. I cannot think of any reason why this can break dma. We can probably try adding some dummy HPET read after dma write, to see if that flushes things properly.

Shouldn't that be "... some dummy HPET read _before_ dma write...". In order to ensure that DMA cache is consistent _before_ dma controller reads it?

I removed that code when moving to 64bit (floppy driver disabling C1), but perhaps we need some variant of it again (but it's the first such report in many years). Although it would be sad to have it again on all systems.

Quoted 4 lines · On Wed, 23 Dec 2009 18:08:32 +0100 Andi Kleen <[email protected]> wrote:

I removed that code when moving to 64bit (floppy driver disabling C1), but perhaps we need some variant of it again (but it's the first such report in many years). Although it would be sad to have it again on all systems.

at least now we have the pmqos infrastructure, driver just needs to ask for 0 latency ;)

Quoted 8 lines · Arjan van de Ven

Quoted 4 lines · On Wed, 23 Dec 2009 18:08:32 +0100 Andi Kleen <[email protected]> wrote:

I removed that code when moving to 64bit (floppy driver disabling C1), but perhaps we need some variant of it again (but it's the first such report in many years). Although it would be sad to have it again on all systems.

Quoted 2 lines

at least now we have the pmqos infrastructure, driver just needs to ask for 0 latency ;)

Does pmqos work with apci=off etc.?

I didn't think it shut down
the classic "HLT" idle, does it?

Quoted 1 line

Does pmqos work with apci=off etc.?

yes

Quoted 2 lines

I didn't think it shut down
the classic "HLT" idle, does it?

it does if you specify a latency of 0; it will then go into the spin-only state until you give up your latency requirement

Quoted 7 lines

Quoted 1 line

Does pmqos work with apci=off etc.?

yes

Quoted 2 lines

I didn't think it shut down
the classic "HLT" idle, does it?

it does if you specify a latency of 0; it will then go into the spin-only state until you give up your latency requirement

I looked at it this evening, but it seems like pm_qos is not interrupt safe (e.g. calls blocking notifiers) and floppy currently does enable/disable_hlt from interrupts and bottom halves.

Would need some more infrastructure work or restructuring of the floppy driver.

-Andi

disable_hlt/enable_hlt was only needed to work around a bug on TM4000 (Texas Instrument) Laptops which were popular around 1994 / 1995.

Quoted 2 lines

disable_hlt/enable_hlt was only needed to work around a bug on TM4000 (Texas Instrument) Laptops which were popular around 1994 / 1995.

I don't think we can fully drop support for these systems.

Did they have an unique PCI ID or something else that could be tested for?

Perhaps it could be just a white list like dmi_year > 1995 to disable.

Depending on how often floppies are still used this might save non trivial amounts of power on newer systems :)

Anyways it would be probably good to convert this to the new infrastructure, and remove the old hooks, but the interrupt-context issue would need to be fixed first.

-Andi

Quoted 7 lines · Andi Kleen

Quoted 2 lines

disable_hlt/enable_hlt was only needed to work around a bug on TM4000 (Texas Instrument) Laptops which were popular around 1994 / 1995.

I don't think we can fully drop support for these systems.

Did they have an unique PCI ID or something else that could be tested for?

Floppy controllers are not PCI devices and thus have no PCI id unfortunately... :-(

Quoted 6 lines

Quoted 4 lines

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

We have indeed had historical issues with floppy and sleep states before.

I removed that code when moving to 64bit (floppy driver disabling C1), but perhaps we need some variant of it again (but it's the first such report in many years). Although it would be sad to have it again on all systems.

Quoted 11 lines

Quoted 6 lines

Quoted 4 lines

This might suggest that Mark's floppy controller doesn't like deep C? Mark, did you try booting with processor.max_cstate=1 and HPET enabled?

We have indeed had historical issues with floppy and sleep states before.

I removed that code when moving to 64bit (floppy driver disabling C1), but perhaps we need some variant of it again (but it's the first such report in many years). Although it would be sad to have it again on all systems.

C1 is hlt. Are you sure? I could see how C3 could cause problems (DMA
latency), but...

Can mark simply try with idle=poll?

Pavel

Quoted 71 lines

Quoted 21 lines

Can you try this one line patch either on .28 or .32 (with /proc/interrupts output).
This disables hpet2 and lapic timer should then be used on CPU 0. If things work with this test patch, we will know that the failure is somehow related to HPET usage in MSI mode.

Thanks,
Venki

Reduce the rating of percpu hpet timer

Signed-off-by Venkatesh Pallipadi <[email protected]>

---
 arch/x86/kernel/hpet.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

arch/x86/kernel/hpet.c +0 -0

index cafb1c6..f89d17a 100644

arch/x86/kernel/hpet.c static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu) · line 486

Quoted 8 lines

arch/x86/kernel/hpet.c +1 -1

@@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
 	hpet_setup_irq(hdev);
 	evt->irq = hdev->irq;
 
-	evt->rating = 110;
+	evt->rating = 40;
 	evt->features = CLOCK_EVT_FEAT_ONESHOT;
 	if (hdev->flags & HPET_DEV_PERI_CAP)
 		evt->features |= CLOCK_EVT_FEAT_PERIODIC;

That made it work. Used 2.6.32.2

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:         82          0          0          1   IO-APIC-edge      timer
  1:          0          0          0         67   IO-APIC-edge      i8042
  3:          0          0          0          6   IO-APIC-edge
  4:          0          0          0          4   IO-APIC-edge
  6:          0          0          0          4   IO-APIC-edge      floppy
  8:          0          0          0          8   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0         10       1519   IO-APIC-edge      i8042
 14:          0          0         39      10995   IO-APIC-edge

pata_atiixp
15: 0 0 3 391 IO-APIC-edge
pata_atiixp
16: 0 0 2 606 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
17: 0 0 0 3 IO-APIC-fasteoi
ehci_hcd:usb1, parport0, ni-pci-gpib
18: 0 0 10 2168 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
19: 0 0 0 130 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1

 22:          0          0          8       1151   IO-APIC-fasteoi   ahci
 24:          0          0          0          0  HPET_MSI-edge      hpet2
 29:          0          0          0         48   PCI-MSI-edge
sky2@pci:0000:04:00.0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      34842      30177      29672      29632   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring
interrupts
PND:          0          0          0          0   Performance pending work
RES:      17501      20449      16670      11224   Rescheduling interrupts
CAL:      10554       2336       1102       1071   Function call interrupts
TLB:        364        562        753        468   TLB shootdowns
ERR:          0
MIS:          0

Quoted 4 lines

# fdformat /dev/fd0u1440
Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
Formatting ... done
Verifying ... done

Hmmm.. Thats very interesting indeed.

That clearly says that HPET MSI interrupts somehow is causing some caching side effect in the chipset that results in this floppy dma failure.

Here's is what we have until now.
IRQ 0 is based on HPET legacy interrupt and HPET device is also capable of MSI on this platform. So we also have a percpu hpet (hpet2 tied to CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast in cases where LAPIC timer will stop working in deep C-state. As we have only one HPET channel free for percpu HPET, we only have hpet2 tied to CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with deep C-state.

One problem here is that percpu hpet should only get used when LAPIC cannot be used (that is when CPU enters deep C-state). Using hpet2 in place of LAPIC timer even when deep C-state is not supported is not right in terms of performance. We need some changes here to fix that [Problem 1].

But, that still does not explain why we are seeing this problem in the first place. I mean, using hpet2 is not optimal, but should not have functionality issues like this. Even fixing [Problem 1] above, we may see this problem on some other platform that supports deep C-state and so has hpet2 enabled for a valid reason.

Also, I am not sure whether the problem also happens if legacy HPET interrupts are used during run time in place of LAPIC timer (May be worth to try this with a simple test patch, let me think about it). In this case, legacy HPET interrupt rightly goes quiet after boot, giving priority to LAPIC timer.

With hpet MSI interrupts, we do a write followed by read of HPET memmapped register to set a HPET channel timeout + read of global HPET timer. This happens on every timer interrupt on CPU 0. And we also have MSI interrupt being delivered to CPU 0. I cannot think of any reason why this can break dma. We can probably try adding some dummy HPET read after dma write, to see if that flushes things properly.

Quoted 109 lines · Venkatesh

Quoted 71 lines

Quoted 21 lines

Can you try this one line patch either on .28 or .32 (with /proc/interrupts output).
This disables hpet2 and lapic timer should then be used on CPU 0. If things work with this test patch, we will know that the failure is somehow related to HPET usage in MSI mode.

Thanks,
Venki

Reduce the rating of percpu hpet timer

Signed-off-by Venkatesh Pallipadi <[email protected]>

---
 arch/x86/kernel/hpet.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

arch/x86/kernel/hpet.c +0 -0

index cafb1c6..f89d17a 100644

arch/x86/kernel/hpet.c static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu) · line 486

Quoted 8 lines

arch/x86/kernel/hpet.c +1 -1

@@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
 	hpet_setup_irq(hdev);
 	evt->irq = hdev->irq;
 
-	evt->rating = 110;
+	evt->rating = 40;
 	evt->features = CLOCK_EVT_FEAT_ONESHOT;
 	if (hdev->flags & HPET_DEV_PERI_CAP)
 		evt->features |= CLOCK_EVT_FEAT_PERIODIC;

That made it work. Used 2.6.32.2

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:         82          0          0          1   IO-APIC-edge      timer
  1:          0          0          0         67   IO-APIC-edge      i8042
  3:          0          0          0          6   IO-APIC-edge
  4:          0          0          0          4   IO-APIC-edge
  6:          0          0          0          4   IO-APIC-edge      floppy
  8:          0          0          0          8   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0         10       1519   IO-APIC-edge      i8042
 14:          0          0         39      10995   IO-APIC-edge

pata_atiixp
15: 0 0 3 391 IO-APIC-edge
pata_atiixp
16: 0 0 2 606 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
17: 0 0 0 3 IO-APIC-fasteoi
ehci_hcd:usb1, parport0, ni-pci-gpib
18: 0 0 10 2168 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
19: 0 0 0 130 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1

 22:          0          0          8       1151   IO-APIC-fasteoi   ahci
 24:          0          0          0          0  HPET_MSI-edge      hpet2
 29:          0          0          0         48   PCI-MSI-edge
sky2@pci:0000:04:00.0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      34842      30177      29672      29632   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring
interrupts
PND:          0          0          0          0   Performance pending work
RES:      17501      20449      16670      11224   Rescheduling interrupts
CAL:      10554       2336       1102       1071   Function call interrupts
TLB:        364        562        753        468   TLB shootdowns
ERR:          0
MIS:          0

Quoted 4 lines

# fdformat /dev/fd0u1440
Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
Formatting ... done
Verifying ... done

Hmmm.. Thats very interesting indeed.

That clearly says that HPET MSI interrupts somehow is causing some caching side effect in the chipset that results in this floppy dma failure.

Here's is what we have until now.
IRQ 0 is based on HPET legacy interrupt and HPET device is also capable of MSI on this platform. So we also have a percpu hpet (hpet2 tied to CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast in cases where LAPIC timer will stop working in deep C-state. As we have only one HPET channel free for percpu HPET, we only have hpet2 tied to CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with deep C-state.

One problem here is that percpu hpet should only get used when LAPIC cannot be used (that is when CPU enters deep C-state). Using hpet2 in place of LAPIC timer even when deep C-state is not supported is not right in terms of performance. We need some changes here to fix that [Problem 1].

But, that still does not explain why we are seeing this problem in the first place. I mean, using hpet2 is not optimal, but should not have functionality issues like this. Even fixing [Problem 1] above, we may see this problem on some other platform that supports deep C-state and so has hpet2 enabled for a valid reason.

Also, I am not sure whether the problem also happens if legacy HPET interrupts are used during run time in place of LAPIC timer (May be worth to try this with a simple test patch, let me think about it). In this case, legacy HPET interrupt rightly goes quiet after boot, giving priority to LAPIC timer.

Quoted 7 lines

With hpet MSI interrupts, we do a write followed by read of HPET memmapped register to set a HPET channel timeout + read of global HPET timer. This happens on every timer interrupt on CPU 0. And we also have MSI interrupt being delivered to CPU 0. I cannot think of any reason why this can break dma. We can probably try adding some dummy HPET read after dma write, to see if that flushes things properly.

Haven't seen any activity on this thread in a while. Just curious, are we still working this?
Is there anything else I can do to help?

Quoted 119 lines · Mark Hounschell

Quoted 109 lines · Venkatesh

Quoted 71 lines

Quoted 21 lines

Can you try this one line patch either on .28 or .32 (with /proc/interrupts output).
This disables hpet2 and lapic timer should then be used on CPU 0. If things work with this test patch, we will know that the failure is somehow related to HPET usage in MSI mode.

Thanks,
Venki

Reduce the rating of percpu hpet timer

Signed-off-by Venkatesh Pallipadi <[email protected]>

---
 arch/x86/kernel/hpet.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

arch/x86/kernel/hpet.c +0 -0

index cafb1c6..f89d17a 100644

arch/x86/kernel/hpet.c static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu) · line 486

Quoted 8 lines

arch/x86/kernel/hpet.c +1 -1

@@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
 	hpet_setup_irq(hdev);
 	evt->irq = hdev->irq;
 
-	evt->rating = 110;
+	evt->rating = 40;
 	evt->features = CLOCK_EVT_FEAT_ONESHOT;
 	if (hdev->flags & HPET_DEV_PERI_CAP)
 		evt->features |= CLOCK_EVT_FEAT_PERIODIC;

That made it work. Used 2.6.32.2

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:         82          0          0          1   IO-APIC-edge      timer
  1:          0          0          0         67   IO-APIC-edge      i8042
  3:          0          0          0          6   IO-APIC-edge
  4:          0          0          0          4   IO-APIC-edge
  6:          0          0          0          4   IO-APIC-edge      floppy
  8:          0          0          0          8   IO-APIC-edge      rtc0
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          0          0         10       1519   IO-APIC-edge      i8042
 14:          0          0         39      10995   IO-APIC-edge

pata_atiixp
15: 0 0 3 391 IO-APIC-edge
pata_atiixp
16: 0 0 2 606 IO-APIC-fasteoi
aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
17: 0 0 0 3 IO-APIC-fasteoi
ehci_hcd:usb1, parport0, ni-pci-gpib
18: 0 0 10 2168 IO-APIC-fasteoi
ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
19: 0 0 0 130 IO-APIC-fasteoi
aic7xxx, ehci_hcd:usb2, ttySLG0, eth1

 22:          0          0          8       1151   IO-APIC-fasteoi   ahci
 24:          0          0          0          0  HPET_MSI-edge      hpet2
 29:          0          0          0         48   PCI-MSI-edge
sky2@pci:0000:04:00.0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      34842      30177      29672      29632   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring
interrupts
PND:          0          0          0          0   Performance pending work
RES:      17501      20449      16670      11224   Rescheduling interrupts
CAL:      10554       2336       1102       1071   Function call interrupts
TLB:        364        562        753        468   TLB shootdowns
ERR:          0
MIS:          0

Quoted 4 lines

# fdformat /dev/fd0u1440
Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
Formatting ... done
Verifying ... done

Hmmm.. Thats very interesting indeed.

That clearly says that HPET MSI interrupts somehow is causing some caching side effect in the chipset that results in this floppy dma failure.

Here's is what we have until now.
IRQ 0 is based on HPET legacy interrupt and HPET device is also capable of MSI on this platform. So we also have a percpu hpet (hpet2 tied to CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast in cases where LAPIC timer will stop working in deep C-state. As we have only one HPET channel free for percpu HPET, we only have hpet2 tied to CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with deep C-state.

One problem here is that percpu hpet should only get used when LAPIC cannot be used (that is when CPU enters deep C-state). Using hpet2 in place of LAPIC timer even when deep C-state is not supported is not right in terms of performance. We need some changes here to fix that [Problem 1].

But, that still does not explain why we are seeing this problem in the first place. I mean, using hpet2 is not optimal, but should not have functionality issues like this. Even fixing [Problem 1] above, we may see this problem on some other platform that supports deep C-state and so has hpet2 enabled for a valid reason.

Also, I am not sure whether the problem also happens if legacy HPET interrupts are used during run time in place of LAPIC timer (May be worth to try this with a simple test patch, let me think about it). In this case, legacy HPET interrupt rightly goes quiet after boot, giving priority to LAPIC timer.

Quoted 7 lines

With hpet MSI interrupts, we do a write followed by read of HPET memmapped register to set a HPET channel timeout + read of global HPET timer. This happens on every timer interrupt on CPU 0. And we also have MSI interrupt being delivered to CPU 0. I cannot think of any reason why this can break dma. We can probably try adding some dummy HPET read after dma write, to see if that flushes things properly.

Quoted 3 lines

Haven't seen any activity on this thread in a while. Just curious, are we still working this?
Is there anything else I can do to help?

Sorry for not following up on this. We have narrowed this down to HPET MSI and floppy DMA. I still don't know how HPET MSI interrupts are breaking floppy DMA.

You are seeing the problem on two different systems. Correct? Do you have any system where this works with HPET MSI enabled?

Couple of options on how we can go about this one:
1) Change the HPET-MSI change to not get activated when there are no C-states with LAPIC stoppage involved. This will resolve the problem on the systems you reported as there are no deep C-states. But, I fear that with the actual problem unresolved, we may hit it in future with this or some other platform having same issue with CPUs that support deep C-state.
2) Try this testcase on few other platforms that support HPET-MSI and deep C-states and check how widespread the problem is and then add a whitelist-blacklist for HPET MSI usage.

I think, for 2.6.33 option 1 is better. Will work on that and send in patches for you test.

Quoted 5 lines · Venkatesh

Sorry for not following up on this. We have narrowed this down to HPET MSI and floppy DMA. I still don't know how HPET MSI interrupts are breaking floppy DMA.

Quoted 3 lines

You are seeing the problem on two different systems. Correct? Do you have any system where this works with HPET MSI enabled?

I see the problem on every system in which the HPET2 shows up in /proc/interrupts. The machines that work with HPET enabled don't show HPET at all in /proc/interrupts. I have some of each. All the boxes that fail here use the (AMD) 790x series chip sets.

Quoted 14 lines

Couple of options on how we can go about this one:
1) Change the HPET-MSI change to not get activated when there are no C-states with LAPIC stoppage involved. This will resolve the problem on the systems you reported as there are no deep C-states. But, I fear that with the actual problem unresolved, we may hit it in future with this or some other platform having same issue with CPUs that support deep C-state.
2) Try this testcase on few other platforms that support HPET-MSI and deep C-states and check how widespread the problem is and then add a whitelist-blacklist for HPET MSI usage.

I think, for 2.6.33 option 1 is better. Will work on that and send in patches for you test.

Quoted 6 lines · Mark Hounschell

Quoted 5 lines · Venkatesh

Sorry for not following up on this. We have narrowed this down to HPET MSI and floppy DMA. I still don't know how HPET MSI interrupts are breaking floppy DMA.

Quoted 24 lines

Quoted 3 lines

You are seeing the problem on two different systems. Correct? Do you have any system where this works with HPET MSI enabled?

I see the problem on every system in which the HPET2 shows up in /proc/interrupts. The machines that work with HPET enabled don't show HPET at all in /proc/interrupts. I have some of each. All the boxes that fail here use the (AMD) 790x series chip sets.

Quoted 14 lines

Couple of options on how we can go about this one:
1) Change the HPET-MSI change to not get activated when there are no C-states with LAPIC stoppage involved. This will resolve the problem on the systems you reported as there are no deep C-states. But, I fear that with the actual problem unresolved, we may hit it in future with this or some other platform having same issue with CPUs that support deep C-state.
2) Try this testcase on few other platforms that support HPET-MSI and deep C-states and check how widespread the problem is and then add a whitelist-blacklist for HPET MSI usage.

I think, for 2.6.33 option 1 is better. Will work on that and send in patches for you test.

Mark,

I just sent out a patchset that should workaround the problem here. Can you check and let me know whether thats the case.

We will still need a simpler/smaller workaround for .33. Will send a patch for that soon.

Also, are you testing this with usb floppy controller? I tried to test it on my end, but fdformat doesn't seem to like my usb floppy drive. I tried, 'ufiformat -f 1440 <dev>', with which I am not able to reproduce the failure on any of my boxes. Not sure whether that really means I don't hit this bug or that is going through totally different code path.

Quoted 6 lines · Mark Hounschell

Quoted 5 lines · Venkatesh

Sorry for not following up on this. We have narrowed this down to HPET MSI and floppy DMA. I still don't know how HPET MSI interrupts are breaking floppy DMA.

Quoted 24 lines

Quoted 3 lines

You are seeing the problem on two different systems. Correct? Do you have any system where this works with HPET MSI enabled?

I see the problem on every system in which the HPET2 shows up in /proc/interrupts. The machines that work with HPET enabled don't show HPET at all in /proc/interrupts. I have some of each. All the boxes that fail here use the (AMD) 790x series chip sets.

Quoted 14 lines

Couple of options on how we can go about this one:
1) Change the HPET-MSI change to not get activated when there are no C-states with LAPIC stoppage involved. This will resolve the problem on the systems you reported as there are no deep C-states. But, I fear that with the actual problem unresolved, we may hit it in future with this or some other platform having same issue with CPUs that support deep C-state.
2) Try this testcase on few other platforms that support HPET-MSI and deep C-states and check how widespread the problem is and then add a whitelist-blacklist for HPET MSI usage.

I think, for 2.6.33 option 1 is better. Will work on that and send in patches for you test.

Mark,

I just sent out a patchset that should workaround the problem here. Can you check and let me know whether thats the case.

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

I think somebody from AMD should review the situation.Clearly

Quoted 3 lines · Yuhong Bao

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Argh, will see what information I can find about this problem ...

Quoted 1 line

I think somebody from AMD should review the situation.Clearly

something is happening inside their southbridge.CCing Andreas Herrmann from AMD.

I have the feeling that this problem should rather be fixed with a DMI quirk instead of disabling HPET MSI for the entire chipset.

Was the latest available BIOS installed on the affected system?

Quoted 14 lines · Andreas Herrmann

Quoted 3 lines · Yuhong Bao

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Argh, will see what information I can find about this problem ...

Quoted 1 line

I think somebody from AMD should review the situation.Clearly

something is happening inside their southbridge.CCing Andreas Herrmann from AMD.

I have the feeling that this problem should rather be fixed with a DMI quirk instead of disabling HPET MSI for the entire chipset.

Quoted 2 lines

Was the latest available BIOS installed on the affected system?

You mean "systems" of different manufactures? I will check today. Due to mis configured filters I didn't see this until today. Sorry.

Mark

Quoted 18 lines · Mark Hounschell

Quoted 16 lines · Andreas Herrmann

Quoted 3 lines · Yuhong Bao

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Argh, will see what information I can find about this problem ...

Quoted 1 line

I think somebody from AMD should review the situation.Clearly

something is happening inside their southbridge.CCing Andreas Herrmann from AMD.

I have the feeling that this problem should rather be fixed with a DMI quirk instead of disabling HPET MSI for the entire chipset.

Was the latest available BIOS installed on the affected system?

Quoted 5 lines

You mean "systems" of different manufactures? I will check today. Due to mis configured filters I didn't see this until today. Sorry.

Mark

My BIOS were below rev on all my affected boards but updating did not help with the problem.

Quoted 19 lines · Mark Hounschell

Quoted 18 lines · Mark Hounschell

Quoted 16 lines · Andreas Herrmann

Quoted 3 lines · Yuhong Bao

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Argh, will see what information I can find about this problem ...

Quoted 1 line

I think somebody from AMD should review the situation.Clearly

something is happening inside their southbridge.CCing Andreas Herrmann from AMD.

I have the feeling that this problem should rather be fixed with a DMI quirk instead of disabling HPET MSI for the entire chipset.

Was the latest available BIOS installed on the affected system?

Quoted 8 lines

Quoted 5 lines

You mean "systems" of different manufactures? I will check today. Due to mis configured filters I didn't see this until today. Sorry.

Mark

My BIOS were below rev on all my affected boards but updating did not help with the problem.

Hi,

can you post the BIOS vendors of the boards along with the respective BIOS versions?

Thanks.

Quoted 3 lines · Yuhong Bao

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Argh, will see what information I can find about this problem ...

The suggested workaround for this is to disable HPET MSI if LPC
devices are used. I doubt that there is a convenient way for Linux to
find out whether LPC devices are used.And don't forget that the Super I/O chip in most motherboards is an LPC device!(In fact, that was what LPC was invented for) Thus I think the only solution
to safely avoid the problem is the currently implemented quirk to
disable HPET MSI on this chipset.

FYI. I've tried to trigger the publication of errata information for that chipset. Finally this has happened.

The discussed problem is indeed due to an erratum. See erratum #27 in
http://support.amd.com/us/Embedded_TechDocs/46837.pdf

The suggested workaround for this is to disable HPET MSI if LPC devices are used. I doubt that there is a convenient way for Linux to find out whether LPC devices are used. Thus I think the only solution to safely avoid the problem is the currently implemented quirk to disable HPET MSI on this chipset.

Quoted 4 lines · Andreas Herrmann

FYI. I've tried to trigger the publication of errata information for that chipset. Finally this has happened.

Quoted 8 lines

The discussed problem is indeed due to an erratum. See erratum #27 in
http://support.amd.com/us/Embedded_TechDocs/46837.pdf

The suggested workaround for this is to disable HPET MSI if LPC devices are used. I doubt that there is a convenient way for Linux to find out whether LPC devices are used. Thus I think the only solution to safely avoid the problem is the currently implemented quirk to disable HPET MSI on this chipset.

Goodie. It would be good to point this out in the source too. Would you be willing to send in a patch that documents this quirk as a result of that erratum #27, so that we don't lose sight of why we're doing that odd MSI disable?

Quoted 6 lines · Andreas Herrmann

Quoted 3 lines · Yuhong Bao

Quoted 3 lines

HPET MSI on platforms with ATI SB700/SB800 as they seem to have some side-effects on floppy DMA. Do not use HPET MSI on such platforms.

Argh, will see what information I can find about this problem ...

FYI. I've tried to trigger the publication of errata information for that chipset. Finally this has happened.

The discussed problem is indeed due to an erratum. See erratum #27 in
http://support.amd.com/us/Embedded_TechDocs/46837.pdf

The suggested workaround for this is to disable HPET MSI if LPC devices are used. I doubt that there is a convenient way for Linux to find out whether LPC devices are used. Thus I think the only solution to safely avoid the problem is the currently implemented quirk to disable HPET MSI on this chipset.

If one wanted, you could disable HPET MSI on this chipset only when a driver requests an ISA DMA channel. Then if there's no floppy or other LPC DMA device present, it can stay enabled. I don't know if it's worth the trouble, though.

Quoted 4 lines · Robert Hancock

If one wanted, you could disable HPET MSI on this chipset only when a driver requests an ISA DMA channel. Then if there's no floppy or other LPC DMA device present, it can stay enabled. I don't know if it's worth the trouble, though.

Nope, that wouldn't work.

Imagine a driver that already loaded, and is already using MSI (say, network device). What happens now if you want to access the floppy and load the floppy module? Oh, you can't? Need to bring down the network interface, unload that module first? Not practical.

Sure, in theory we can do some crazy callback for "you now need to re-do your interrupt registration" for all devices. In practice, I can onyl say "not going to happen".

If one wanted, you could disable HPET MSI on this chipset only when a driver requests an ISA DMA channel. Then if there's no floppy or other LPC DMA device present, it can stay enabled. I don't know if it's worth the trouble, though.

Quoted 4 lines

If one wanted, you could disable HPET MSI on this chipset only when a driver requests an ISA DMA channel. Then if there's no floppy or other LPC DMA device present, it can stay enabled. I don't know if it's worth the trouble, though.

There can be LPC devices which are not visible to the kernel, but only used through ACPI or the BIOS. Think of fancy fan controllers and similar.