Skip to contentSkip to conversation

2.6.29 regression: ATA bus errors on resume

DISCUSSION Niel Lambrechts 7 participants 28 messages Mar 30, 2009 – Sep 30, 2009 22 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 84 lines · Jeff Garzik
Quoted 68 lines · Niel Lambrechts
Quoted 58 lines · Tejun Heo

Hello,

For some reason, I can't find the original thread, so replying here.

Quoted 10 lines · Niel Lambrechts
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

The ext4 errors are interleaved with hardware errors, and the ext4 errors are about I/O errors.

EXT4-fs error (device sda6): __ext4_get_inode_loc: unable to read inode block - inode=2346519
EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure

This looks more like a hibernation problem than an ext4 problem. Looks like the hard drive is being left in some inconsistent state after resuming from hibernation.

Yeap, ext4 is just the victim here.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

ata1.00: irq_stat 0x00400008, PHY RDY changed
ata1: SError: { PHYRdyChg CommWake }

Your SATA hardware flags a connect-or-disconnect event ("PHY RDY"), which requires us to abort a bunch of queued commands:

Quoted 3 lines

ata1.00: cmd 60/18:00:77:88:6f/00:00:0e:00:00/40 tag 0 ncq 12288 in

         res 50/00:30:07:b3:10/00:00:0c:00:00/40 Emask 0x10 (ATA

bus error)

[...]

...

Quoted 14 lines
Quoted 14 lines
Quoted 1 line

The SCSI subsystem aborts each of the queued commands.

No .. this is the SCSI subsystem receives an ABORTED COMMAND return in sense data for each of the outstanding I/Os

The only place these are generated is in ata_sense_to_error() which only
occurs if there's some type of ata error.

If I had to theorise, I'd say the system suspended with commands outstanding to the device. On resume, the device gets reset and returns
some type of ATA error which gets translated to ABORTED COMMAND which causes a failure.

Quoted 4 lines
Quoted 4 lines

In the mid layer, we translate ABORTED_COMMAND into a retry until the command runs out of them ... could it be there's a race readying the device and we run through the retries before it can accept the command?

When libata-eh thinks that the problem isn't worth retrying, it sets scmd->retries to scmd->allowed so that it gets aborted immediately. The code is in ata_eh_qc_complete().

Whether a command is to be retried or not is determined with ATA_QCFLAG_RETRY which is set in ata_eh_link_autopsy() for each failed command. Immediate-failure criteria is pretty strict - only driver software errors (AC_ERR_INVALID) and PC or other special commands which failed which got aborted by the device get the immediate pink slip. In this case, the commands are from FS and failed with AC_ERR_ATA_BUS, so it definitely doesn't fit into the criteria. Strange.

Quoted 2 lines

How reproducible is the problem? Are you interested in trying out some debug patches?

Hi Tejun,

I think I should be able to reproduce when actively using X with 2.6.29, and I have an external disk where I could backup to / boot from if the corruption became a problem.

Quoted 12 lines

These issues are keeping me from 2.6.29 so I'll gladly help where I can, if you can please provide me the patches and the .config settings that may be required?

Niel

Signature · 5 lines
To unsubscribe from this list: send the line "unsubscribe
linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Any chance you could use bisect to narrow down the problem commit?

Quoted 7 lines

http://kernel.org/pub/software/scm/git/docs/v1.4.4.4/howto/isolate-bugs-with-bisect.txt

This should identify which patch caused your problems, if you have a known good starting point (such as 2.6.28).

Jeff

Any idea of the volume of data would I need to download, git repository wise? I currently only have the 2.6.27 source and patches on top... and bandwidth is quite expensive in SA...

Niel

#10 lore raw
Quoted 3 lines · Niel Lambrechts

Any idea of the volume of data would I need to download, git repository wise? I currently only have the 2.6.27 source and patches on top... and bandwidth is quite expensive in SA...

Around 300MB for the repo, I think?

Jeff

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#24 lore raw
Quoted 5 lines · Niel Lambrechts
Quoted 3 lines · Tejun Heo

Hmm... none of the debugging messages got triggered, so there should be no noticeable timing change. It seems like you're being lucky for the time being. Can you please keep testing and report back?

Quoted 8 lines

Hi Tejun,

Just to keep you up informed - this issue is still happening, I've been testing the latest 2.6.30-git kernels the last couple of weeks and just checked my logs after having an unexpected fsck upon system boot - the freeze seems to trigger very infrequently though, since I use s2disk at least twice a day and the error only seems to have happened twice since May 2.

I notice that if I hibernate with a CD in the drive and then remove it as soon as I boot, I get a hard reset on the ata1 bus.

I'm not sure if this maybe plays a role in triggering the problem at times, perhaps it is entirely expected - but vim highlights it in red and to a "normal" user that is bad. :)

Maybe after applying your patch I should play around with this?

/var/log/warn:

May 23 11:28:44 linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
May 23 11:28:44 linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 23 11:28:44 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: configured for UDMA/133
May 23 11:28:44 linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
May 23 11:28:44 linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
May 23 11:28:44 linux-7vph kernel: ata1.00: configured for UDMA/133
May 23 11:28:44 linux-7vph kernel: ata1: EH complete
May 23 11:28:44 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata2.00: configured for UDMA/133
May 23 11:28:44 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900407, writing 0x900403)
May 23 11:28:44 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
May 23 11:28:44 linux-7vph kernel: Restarting tasks ... done.
May 23 11:28:44 linux-7vph kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
May 23 11:28:44 linux-7vph kernel: ata2: irq_stat 0x40000001
May 23 11:28:44 linux-7vph kernel: ata2.00: configured for UDMA/133
May 23 11:28:44 linux-7vph kernel: ata2: EH complete
May 23 11:28:44 linux-7vph kernel: ata1.00: exception Emask 0x10 SAct 0x1 SErr 0x10000 action 0xe frozen
May 23 11:28:44 linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed
May 23 11:28:44 linux-7vph kernel: ata1: SError: { PHYRdyChg }
May 23 11:28:44 linux-7vph kernel: ata1.00: cmd 61/08:00:37:5a:f4/00:00:09:00:00/40 tag 0 ncq 4096 out
May 23 11:28:44 linux-7vph kernel:          res 50/00:08:37:5a:f4/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 23 11:28:44 linux-7vph kernel: ata1.00: status: { DRDY }
May 23 11:28:44 linux-7vph kernel: ata1: hard resetting link
May 23 11:28:44 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 23 11:28:44 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 23 11:28:44 linux-7vph kernel: ata1.00: configured for UDMA/133
May 23 11:28:44 linux-7vph kernel: ata1.00: configured for UDMA/133
May 23 11:28:44 linux-7vph kernel: ata1: EH complete

Regards,
Niel

#25 lore raw
Quoted 8 lines · Niel Lambrechts

I notice that if I hibernate with a CD in the drive and then remove it as soon as I boot, I get a hard reset on the ata1 bus.

I'm not sure if this maybe plays a role in triggering the problem at times, perhaps it is entirely expected - but vim highlights it in red and to a "normal" user that is bad. :)

Maybe after applying your patch I should play around with this?

That isn't normal. You're getting PHY event on ata1 if you do something to the cdrom which is attached to ata2. Something very fishy is going on there. Sounds like an electric or some sort of interference problem but I'm not sure. :-(

Thanks.

Signature · 1 line
tejun
#26 lore raw
Quoted 3 lines

something to the cdrom which is attached to ata2. Something very fishy is going on there. Sounds like an electric or some sort of interference problem but I'm not sure. :-(

I'm just wondering what happens on a resume if a phy change event is seen by the phy before or during the time registers (such as the port mappings on an ICH) are being restored (or not restored even). That seems to be one way you could get an event on the wrong port.

#27 lore raw
Quoted 8 lines · Alan Cox
Quoted 4 lines

something to the cdrom which is attached to ata2. Something very fishy is going on there. Sounds like an electric or some sort of interference problem but I'm not sure. :-(

I'm just wondering what happens on a resume if a phy change event is seen by the phy before or during the time registers (such as the port mappings on an ICH) are being restored (or not restored even). That seems to be one way you could get an event on the wrong port

I've tested all of the kernels I have again since 2.6.29.4 also came out just recently. I did a hibernate/resume for each in the console, then repeated the same in X, then continued to the next kernel.

The 2.6.29.4 log is much larger, since some other badness happened there - there is a large kernel trace in there as my first X hibernation attempt failed and came back to X after a few seconds. The system seemed functional, it did not keep generating kernel messages - when I then retried a hibernate it worked, along with the resume. Another unrelated bug perhaps?

As for "hard resetting link" messages, they seemed to always happen under X the times I tried it.

Kernel       EXT4-errors?    Console:ata1 reset?    Console:ata2-reset?    X:ata1 reset?    X:ata2 reset?
2.6.28.10    No              no                     yes                

yes no
2.6.29.4* No no no
no no
2.6.29.4** No - -
yes no
2.6.30-rc6 Yes - -
yes no
2.6.30-rc6 No no no
yes no

* Xorg hibernation attempt failed.
* Xorg Second hibernation attempt (no extra reboot)

I also did a side by side comparison of the messages I have for 2.6.30-rc6, the one with EXT4 errors I reported on yesterday, and another one that worked just fine tonight. I stripped all time-stamps and some pulseaudio messages from the bad one and attached them here, and also saved the full messages for each kernel to http://bugzilla.kernel.org/show_bug.cgi?id=13017 .

Since analysing the code-path is still a bit beyond me, I'll leave you with a little summary of the differences I notice.

A = 2.6.30-rc6 (EXT4 clean)
B = 2.6.30-rc6 (EXT4 errors triggered)

# B first does an ata2 ACPI cmd, A starts with ata1. Only a slight sequence difference, output is the same.
B: linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded

# The main difference appears:
A:linux-7vph kernel: Restarting tasks ... done.
B:linux-7vph kernel: Restarting tasks ... <3>ata1.00: exception Emask 0x10 SAct 0x1f SErr 0x50000 action 0xe frozen

# A first shows "done", then only followed by a frozen message, but with a different SAct value:
A:linux-7vph kernel: ata1.00: exception Emask 0x10 SAct 0x1ff SErr 0x50000 action 0xe frozen

# Both then have:
linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed

# from there, A seems to have a little extra sequence:
linux-7vph kernel: ata1.00: cmd 60/08:00:ef:fc:48/00:00:0b:00:00/40 tag 0 ncq 4096 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:08:97:d5:69/00:00:0e:00:00/40 tag 1 ncq 4096 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/30:10:8f:5f:56/00:00:0f:00:00/40 tag 2 ncq 24576 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/10:18:c7:5f:56/00:00:0f:00:00/40 tag 3 ncq 8192 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/10:20:ff:5f:56/00:00:0f:00:00/40 tag 4 ncq 8192 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)

# B only then shows:
B:linux-7vph kernel: done.

# soon after, B spirals into errors:
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=2 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush finish cmd: f6838ec0
linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]

Hope this helps.

Niel

linux-7vph kernel: PM: Shrinking memory...  done (98443 pages freed)
linux-7vph kernel: PM: Freed 393772 kbytes in 3.23 seconds (121.91 MB/s)
linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
linux-7vph kernel: ACPI handle has no context!
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
linux-7vph kernel: ata2: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
linux-7vph kernel: PM: Saving platform NVS memory
linux-7vph kernel: Disabling non-boot CPUs ...
linux-7vph kernel: CPU 1 is now offline
linux-7vph kernel: SMP alternatives: switching to UP code
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 attaching NULL sched-domain.
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 is down
linux-7vph kernel: Extended CMOS year: 2000
linux-7vph kernel: PM: Creating hibernation image:
linux-7vph kernel: PM: Need to copy 124750 pages
linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
linux-7vph kernel: Intel machine check architecture supported.
linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
linux-7vph kernel: PM: Restoring platform NVS memory
linux-7vph kernel: Extended CMOS year: 2000
linux-7vph kernel: Enabling non-boot CPUs ...
linux-7vph kernel: SMP alternatives: switching to SMP code
linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
linux-7vph kernel: Initializing CPU#1
linux-7vph kernel: Calibrating delay using timer specific routine.. 5054.05 BogoMIPS (lpj=10108117)
linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
linux-7vph kernel: CPU: L2 cache: 6144K
linux-7vph kernel: CPU: Physical Processor ID: 0
linux-7vph kernel: CPU: Processor Core ID: 1
linux-7vph kernel: Intel machine check architecture supported.
linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: Switched to high resolution mode on CPU 1
linux-7vph kernel: CPU0 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 0 1
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: CPU1 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 1 0
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: microcode: failed to init CPU1
linux-7vph kernel: CPU1 is up
linux-7vph kernel: ACPI: Waking up from system sleep state S4
linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900007, writing 0x900403)
linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100506)
linux-7vph kernel: pci 0000:00:02.0: PME# disabled
linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
linux-7vph kernel: pci 0000:00:02.1: PME# disabled
linux-7vph kernel: pci 0000:00:03.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x16 frozen
linux-7vph kernel: ata2: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x16 frozen
linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
linux-7vph kernel: pci 0000:15:00.2: PME# disabled
linux-7vph kernel: pci 0000:15:00.3: PME# disabled
linux-7vph kernel: pci 0000:15:00.4: PME# disabled
linux-7vph kernel: pci 0000:15:00.5: PME# disabled
linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: EH complete
linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
linux-7vph kernel: ata2.00: configured for UDMA/133
linux-7vph kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
linux-7vph kernel: ata2: irq_stat 0x40000001
linux-7vph kernel: ata2.00: configured for UDMA/133
linux-7vph kernel: ata2: EH complete
linux-7vph kernel: Registered led device: iwl-phy0::radio
linux-7vph kernel: Registered led device: iwl-phy0::assoc
linux-7vph kernel: Registered led device: iwl-phy0::RX
linux-7vph kernel: Registered led device: iwl-phy0::TX
linux-7vph kernel: mac80211-phy0: failed to set key (0, 00:1d:92:1d:1e:8e) to hardware (-22)
linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900407, writing 0x900403)
linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
linux-7vph kernel: Restarting tasks ... <3>ata1.00: exception Emask 0x10 SAct 0x1f SErr 0x50000 action 0xe frozen
linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed
linux-7vph kernel: ata1: SError: { PHYRdyChg CommWake }
linux-7vph kernel: ata1.00: cmd 60/08:00:5f:89:67/01:00:0e:00:00/40 tag 0 ncq 135168 in
linux-7vph kernel:          res 50/00:08:5f:08:c8/00:00:0c:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:08:6f:88:6f/01:00:0e:00:00/40 tag 1 ncq 135168 in
linux-7vph kernel:          res 50/00:08:5f:08:c8/00:00:0c:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:10:5f:08:c8/00:00:0c:00:00/40 tag 2 ncq 4096 in
linux-7vph kernel:          res 50/00:08:5f:08:c8/00:00:0c:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:18:9f:98:6d/00:00:0e:00:00/40 tag 3 ncq 4096 in
linux-7vph kernel:          res 50/00:08:5f:08:c8/00:00:0c:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/e8:20:5f:8e:2b/00:00:0b:00:00/40 tag 4 ncq 118784 in
linux-7vph kernel:          res 50/00:08:5f:08:c8/00:00:0c:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1: hard resetting link
linux-7vph kernel: done.
linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 t4
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=2 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush finish cmd: f6838ec0
linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
linux-7vph kernel:         0c c8 08 5f
linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
linux-7vph kernel: end_request: I/O error, dev sda, sector 241666399
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=2 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush finish cmd: f6838680
linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command <2>EXT4-fs error (device sda6): __ext4_get_inode_loc: unable to read inode block - inode=2330983, block=9338933
linux-7vph kernel: [current] [descriptor]
linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
linux-7vph kernel:         0c c8 08 5f
linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
linux-7vph kernel: end_request: I/O error, dev sda, sector 242190447
linux-7vph kernel: EXT4-fs error (device sda6): __ext4_get_inode_loc: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: unable to read inode block - inode=2346524, block=9404418
linux-7vph kernel: scsi_eh_0: flush retry cmd: f6838800
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f403de00
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=2 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush finish cmd: f403d200
linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
linux-7vph kernel:         0c c8 08 5f
linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
linux-7vph kernel: end_request: I/O error, dev sda, sector 187403871
linux-7vph kernel: ata1: EH complete
linux-7vph kernel: EXT4-fs error (device sda6): __ext4_get_inode_loc: unable to read inode block - inode=641187, block=2556122
linux-7vph kernel: EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure
linux-7vph kernel: EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure
linux-7vph kernel: EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x10
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 attaching NULL sched-domain.
linux-7vph kernel: CPU0 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 0 1
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: CPU1 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 1 0
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048) 
linux-7vph kernel: Syncing filesystems ... done.
linux-7vph kernel: Freezing user space processes ... (elapsed 0.00 seconds) done.
linux-7vph kernel: Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
linux-7vph kernel: PM: Shrinking memory... done (0 pages freed)
linux-7vph kernel: PM: Freed 0 kbytes in 0.14 seconds (0.00 MB/s)
linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
linux-7vph kernel: ACPI handle has no context!
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
linux-7vph kernel: ata2: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
linux-7vph kernel: PM: Saving platform NVS memory
linux-7vph kernel: Disabling non-boot CPUs ...
linux-7vph kernel: CPU 1 is now offline
linux-7vph kernel: SMP alternatives: switching to UP code
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 attaching NULL sched-domain.
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 is down
linux-7vph kernel: Extended CMOS year: 2000
linux-7vph kernel: PM: Creating hibernation image: 
linux-7vph kernel: PM: Need to copy 36801 pages
linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
linux-7vph kernel: Intel machine check architecture supported.
linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
linux-7vph kernel: PM: Restoring platform NVS memory
linux-7vph kernel: Extended CMOS year: 2000
linux-7vph kernel: Enabling non-boot CPUs ...
linux-7vph kernel: SMP alternatives: switching to SMP code
linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
linux-7vph kernel: Initializing CPU#1
linux-7vph kernel: Calibrating delay using timer specific routine.. 5054.12 BogoMIPS (lpj=10108243)
linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
linux-7vph kernel: CPU: L2 cache: 6144K
linux-7vph kernel: CPU: Physical Processor ID: 0
linux-7vph kernel: CPU: Processor Core ID: 1
linux-7vph kernel: Intel machine check architecture supported.
linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: Switched to high resolution mode on CPU 1
linux-7vph kernel: CPU0 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 0 1
linux-7vph kernel: CPU1 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 1 0
linux-7vph kernel: microcode: failed to init CPU1
linux-7vph kernel: CPU1 is up
linux-7vph kernel: ACPI: Waking up from system sleep state S4
linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
linux-7vph kernel: pci 0000:00:02.0: PME# disabled
linux-7vph kernel: pci 0000:00:02.1: PME# disabled
linux-7vph kernel: pci 0000:00:03.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x16 frozen
linux-7vph kernel: ata2: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x16 frozen
linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100506, writing 0x100106)
linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
linux-7vph kernel: pci 0000:15:00.2: PME# disabled
linux-7vph kernel: pci 0000:15:00.3: PME# disabled
linux-7vph kernel: pci 0000:15:00.4: PME# disabled
linux-7vph kernel: pci 0000:15:00.5: PME# disabled
linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: EH complete
linux-7vph kernel: ata2.00: configured for UDMA/133
linux-7vph kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
linux-7vph kernel: ata2: irq_stat 0x40000001
linux-7vph kernel: ata2.00: configured for UDMA/133
linux-7vph kernel: ata2: EH complete
linux-7vph kernel: Registered led device: iwl-phy0::radio
linux-7vph kernel: Registered led device: iwl-phy0::assoc
linux-7vph kernel: Registered led device: iwl-phy0::RX
linux-7vph kernel: Registered led device: iwl-phy0::TX
linux-7vph kernel: mac80211-phy0: failed to set key (0, 00:1d:92:1d:1e:8e) to hardware (-22)
linux-7vph kernel: Restarting tasks ... done.
linux-7vph init: Switching to runlevel: 5
linux-7vph kernel: Kernel logging (proc) stopped.
linux-7vph kernel: Kernel log daemon terminating.

linux-7vph syslog-ng[2164]: Termination requested via signal, terminating;
linux-7vph syslog-ng[2164]: syslog-ng shutting down; version='2.0.9'
linux-7vph syslog-ng[3311]: syslog-ng starting up; version='2.0.9'
linux-7vph kdm_config[3350]: Multiple occurrences of key 'UseTheme' in section [X-*-Greeter] of /usr/share/kde4/config/kdm/kdmrc

linux-7vph kernel: klogd 1.4.1, log source = /proc/kmsg started.
linux-7vph kernel: [drm] Initialized drm 1.1.0 20060810
linux-7vph kernel: pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
linux-7vph kernel: pci 0000:00:02.0: irq 31 for MSI/MSI-X
linux-7vph kernel: acpi device:03: registered as cooling_device2
linux-7vph kernel: input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/device:02/input/input12
linux-7vph kernel: ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
linux-7vph kernel: [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x10
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 attaching NULL sched-domain.
linux-7vph kernel: CPU0 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 0 1
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: CPU1 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 1 0
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph dhclient: Internet Systems Consortium DHCP Client V3.1.1
linux-7vph dhclient: Copyright 2004-2008 Internet Systems Consortium.
linux-7vph dhclient: All rights reserved.
linux-7vph dhclient: For info, please visit http://www.isc.org/sw/dhcp/
linux-7vph dhclient: 
linux-7vph dhclient: Listening on LPF/eth0/00:1c:25:95:93:ec
linux-7vph dhclient: Sending on   LPF/eth0/00:1c:25:95:93:ec
linux-7vph dhclient: Sending on   Socket/fallback
linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
linux-7vph python: hp-systray(init)[3661]: warning: No hp: or hpfax: devices found in any installed CUPS queue. Exiting.
linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 18
linux-7vph kernel: CE: hpet increasing min_delta_ns to 15000 nsec
linux-7vph kernel: CE: hpet increasing min_delta_ns to 22500 nsec
linux-7vph sudo:   niella : TTY=pts/2 ; PWD=/home/niella ; USER=root ; COMMAND=/bin/su -
linux-7vph su: (to root) niella on /dev/pts/2
linux-7vph kernel: CE: hpet increasing min_delta_ns to 33750 nsec
linux-7vph kernel: CE: hpet increasing min_delta_ns to 50624 nsec
linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 16
linux-7vph dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched...
linux-7vph dns-resolver: You can find my version in /etc/resolv.conf.netconfig
linux-7vph kernel: [drm:gm45_get_vblank_counter] *ERROR* trying to get vblank count for disabled pipe 0
linux-7vph kernel: Syncing filesystems ... done.
linux-7vph kernel: Freezing user space processes ... (elapsed 0.00 seconds) done.
linux-7vph kernel: Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
linux-7vph kernel: PM: Shrinking memory... done (63060 pages freed)
linux-7vph kernel: PM: Freed 252240 kbytes in 1.77 seconds (142.50 MB/s)
linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
linux-7vph kernel: ACPI handle has no context!
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
linux-7vph kernel: ata2: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
linux-7vph kernel: PM: Saving platform NVS memory
linux-7vph kernel: Disabling non-boot CPUs ...
linux-7vph kernel: CPU 1 is now offline
linux-7vph kernel: SMP alternatives: switching to UP code
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 attaching NULL sched-domain.
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 is down
linux-7vph kernel: Extended CMOS year: 2000
linux-7vph kernel: PM: Creating hibernation image: 
linux-7vph kernel: PM: Need to copy 121000 pages
linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
linux-7vph kernel: Intel machine check architecture supported.
linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
linux-7vph kernel: PM: Restoring platform NVS memory
linux-7vph kernel: Extended CMOS year: 2000
linux-7vph kernel: Enabling non-boot CPUs ...
linux-7vph kernel: SMP alternatives: switching to SMP code
linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
linux-7vph kernel: Initializing CPU#1
linux-7vph kernel: Calibrating delay using timer specific routine.. 5054.04 BogoMIPS (lpj=10108092)
linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
linux-7vph kernel: CPU: L2 cache: 6144K
linux-7vph kernel: CPU: Physical Processor ID: 0
linux-7vph kernel: CPU: Processor Core ID: 1
linux-7vph kernel: Intel machine check architecture supported.
linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: Switched to high resolution mode on CPU 1
linux-7vph kernel: CPU0 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 0 1
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: CPU1 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 1 0
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: microcode: failed to init CPU1
linux-7vph kernel: CPU1 is up
linux-7vph kernel: ACPI: Waking up from system sleep state S4
linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900007, writing 0x900403)
linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100506)
linux-7vph kernel: pci 0000:00:02.0: PME# disabled
linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
linux-7vph kernel: pci 0000:00:02.1: PME# disabled
linux-7vph kernel: pci 0000:00:03.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x16 frozen
linux-7vph kernel: ata2: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x16 frozen
linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
linux-7vph kernel: pci 0000:15:00.2: PME# disabled
linux-7vph kernel: pci 0000:15:00.3: PME# disabled
linux-7vph kernel: pci 0000:15:00.4: PME# disabled
linux-7vph kernel: pci 0000:15:00.5: PME# disabled
linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: EH complete
linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
linux-7vph kernel: ata2.00: configured for UDMA/133
linux-7vph kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
linux-7vph kernel: ata2: irq_stat 0x40000001
linux-7vph kernel: ata2.00: configured for UDMA/133
linux-7vph kernel: ata2: EH complete
linux-7vph kernel: Registered led device: iwl-phy0::radio
linux-7vph kernel: Registered led device: iwl-phy0::assoc
linux-7vph kernel: Registered led device: iwl-phy0::RX
linux-7vph kernel: Registered led device: iwl-phy0::TX
linux-7vph kernel: mac80211-phy0: failed to set key (0, 00:1d:92:1d:1e:8e) to hardware (-22)
linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900407, writing 0x900403)
linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
linux-7vph kernel: Restarting tasks ... done.
linux-7vph kernel: ata1.00: exception Emask 0x10 SAct 0x1ff SErr 0x50000 action 0xe frozen
linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed
linux-7vph kernel: ata1: SError: { PHYRdyChg CommWake }
linux-7vph kernel: ata1.00: cmd 60/08:00:ef:fc:48/00:00:0b:00:00/40 tag 0 ncq 4096 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:08:97:d5:69/00:00:0e:00:00/40 tag 1 ncq 4096 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/30:10:8f:5f:56/00:00:0f:00:00/40 tag 2 ncq 24576 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/10:18:c7:5f:56/00:00:0f:00:00/40 tag 3 ncq 8192 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/10:20:ff:5f:56/00:00:0f:00:00/40 tag 4 ncq 8192 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:28:3f:e6:48/00:00:0c:00:00/40 tag 5 ncq 4096 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/28:30:37:7f:e6/00:00:0b:00:00/40 tag 6 ncq 20480 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/08:38:6f:7f:e6/00:00:0b:00:00/40 tag 7 ncq 4096 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1.00: cmd 60/38:40:97:7f:e6/00:00:0b:00:00/40 tag 8 ncq 28672 in
linux-7vph kernel:          res 50/00:38:97:7f:e6/00:00:0b:00:00/40 Emask 0x10 (ATA bus error)
linux-7vph kernel: ata1.00: status: { DRDY }
linux-7vph kernel: ata1: hard resetting link
linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 t4
linux-7vph kernel: ata1.00: configured for UDMA/133
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f680d2c0
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f136a9c0
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f428b140
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f64b3d40
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f479f900
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f428b2c0
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f680d200
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f680d8c0
linux-7vph kernel: XXX scsi_eh_flush_done_q: online=1(2) noretry=0 retries=0 allowed=5
linux-7vph kernel: scsi_eh_0: flush retry cmd: f680dc80
linux-7vph kernel: ata1: EH complete
linux-7vph kernel: ata1: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x10
linux-7vph kernel: CPU0 attaching NULL sched-domain.
linux-7vph kernel: CPU1 attaching NULL sched-domain.
linux-7vph kernel: CPU0 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 0 1
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
linux-7vph kernel: CPU1 attaching sched-domain:
linux-7vph kernel:  domain 0: span 0-1 level MC
linux-7vph kernel:   groups: 1 0
linux-7vph kernel:   domain 1: span 0-1 level CPU
linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
#28 lore raw

Hello, Niel.

Quoted 31 lines · Niel Lambrechts

I've tested all of the kernels I have again since 2.6.29.4 also came out just recently. I did a hibernate/resume for each in the console, then repeated the same in X, then continued to the next kernel.

The 2.6.29.4 log is much larger, since some other badness happened there
- there is a large kernel trace in there as my first X hibernation attempt failed and came back to X after a few seconds. The system seemed functional, it did not keep generating kernel messages - when I then retried a hibernate it worked, along with the resume. Another unrelated bug perhaps?

As for "hard resetting link" messages, they seemed to always happen under X the times I tried it.

Kernel       EXT4-errors?    Console:ata1 reset?   Console:ata2-reset?    X:ata1 reset?    X:ata2 reset?
2.6.28.10    No              no                     yes                   yes              no
2.6.29.4*    No              no                     no                    no               no
2.6.29.4**   No              -                      -                     yes              no
2.6.30-rc6   Yes             -                      -                     yes              no
2.6.30-rc6   No              no                     no                    yes              no

* Xorg hibernation attempt failed.
* Xorg Second hibernation attempt (no extra reboot)

I also did a side by side comparison of the messages I have for 2.6.30-rc6, the one with EXT4 errors I reported on yesterday, and another one that worked just fine tonight. I stripped all time-stamps and some pulseaudio messages from the bad one and attached them here, and also saved the full messages for each kernel to
http://bugzilla.kernel.org/show_bug.cgi?id=13017 .

Quoted 5 lines

Since analysing the code-path is still a bit beyond me, I'll leave you with a little summary of the differences I notice.

A = 2.6.30-rc6 (EXT4 clean)
B = 2.6.30-rc6 (EXT4 errors triggered)

Duplicate PHY events are likely to be dependent on timing and non-deterministic. The ext4 corrupting or not depends on whether a request with failfast set was in-flight at the time of the second PHY event, which again is dependent on timing. At any rate, this looks like a problem of ext4 (or something between ext4 and the driver). It either shouldn't issue failfast command or should take appropriate recovery action if it does. It would be really nice if you can give a shot at ext3.

Thanks.

Signature · 1 line
tejun
#29 lore raw
Quoted 41 lines · Tejun Heo

Hello, Niel.

Quoted 31 lines · Niel Lambrechts

I've tested all of the kernels I have again since 2.6.29.4 also came out just recently. I did a hibernate/resume for each in the console, then repeated the same in X, then continued to the next kernel.

The 2.6.29.4 log is much larger, since some other badness happened there
- there is a large kernel trace in there as my first X hibernation attempt failed and came back to X after a few seconds. The system seemed functional, it did not keep generating kernel messages - when I then retried a hibernate it worked, along with the resume. Another unrelated bug perhaps?

As for "hard resetting link" messages, they seemed to always happen under X the times I tried it.

Kernel       EXT4-errors?    Console:ata1 reset?   Console:ata2-reset?    X:ata1 reset?    X:ata2 reset?
2.6.28.10    No              no                     yes                   yes              no
2.6.29.4*    No              no                     no                    no               no
2.6.29.4**   No              -                      -                     yes              no
2.6.30-rc6   Yes             -                      -                     yes              no
2.6.30-rc6   No              no                     no                    yes              no

* Xorg hibernation attempt failed.
* Xorg Second hibernation attempt (no extra reboot)

I also did a side by side comparison of the messages I have for 2.6.30-rc6, the one with EXT4 errors I reported on yesterday, and another one that worked just fine tonight. I stripped all time-stamps and some pulseaudio messages from the bad one and attached them here, and also saved the full messages for each kernel to
http://bugzilla.kernel.org/show_bug.cgi?id=13017 .

Quoted 6 lines

Since analysing the code-path is still a bit beyond me, I'll leave you with a little summary of the differences I notice.

A = 2.6.30-rc6 (EXT4 clean)
B = 2.6.30-rc6 (EXT4 errors triggered)

Quoted 8 lines

Duplicate PHY events are likely to be dependent on timing and non-deterministic. The ext4 corrupting or not depends on whether a request with failfast set was in-flight at the time of the second PHY event, which again is dependent on timing. At any rate, this looks like a problem of ext4 (or something between ext4 and the driver). It either shouldn't issue failfast command or should take appropriate recovery action if it does. It would be really nice if you can give a shot at ext3.

Urgh. My root file-system is mounted with extents on, I would have to re-install entirely.

I'm wondering why no one else is complaining, or whether the problem is limited to ICH9M/M-E controllers with EXT4 or a certain type of hard-drive. The laptop is a Lenovo W500 (fairly similar to T500), so maybe not a lot of people with this type of controller is using EXT4 yet.

Anyhow, I think Theodore may have ruled this out as a EXT4 problem already (I first copied him) so I'm not sure what to do now, it will take some strong will (and even more time) for me to re-install EXT3. I just shouldn't have to, dammit. :-p

Regards,
Niel

#30 lore raw
Quoted 7 lines · Niel Lambrechts

Urgh. My root file-system is mounted with extents on, I would have to re-install entirely.

I'm wondering why no one else is complaining, or whether the problem is limited to ICH9M/M-E controllers with EXT4 or a certain type of hard-drive. The laptop is a Lenovo W500 (fairly similar to T500), so maybe not a lot of people with this type of controller is using EXT4 yet.

The double PHY events are not too rare but it's not too frequent either. The number of people using ext4 on machines which have such issues and are regularly suspending and resuming might not be too high.

Quoted 4 lines

Anyhow, I think Theodore may have ruled this out as a EXT4 problem already (I first copied him) so I'm not sure what to do now, it will take some strong will (and even more time) for me to re-install EXT3. I just shouldn't have to, dammit. :-p

Well, that is a pretty easy way to rule out many possibilities from my point of view. :-p If it's difficult, we can try to find out where the flag is being set with debug patches.

Thanks.

Signature · 1 line
tejun
#31 lore raw
Quoted 16 lines · Tejun Heo
Quoted 8 lines · Niel Lambrechts

Urgh. My root file-system is mounted with extents on, I would have to re-install entirely.

I'm wondering why no one else is complaining, or whether the problem is limited to ICH9M/M-E controllers with EXT4 or a certain type of hard-drive. The laptop is a Lenovo W500 (fairly similar to T500), so maybe not a lot of people with this type of controller is using EXT4 yet.

The double PHY events are not too rare but it's not too frequent either. The number of people using ext4 on machines which have such issues and are regularly suspending and resuming might not be too high.

Quoted 9 lines
Quoted 5 lines

Anyhow, I think Theodore may have ruled this out as a EXT4 problem already (I first copied him) so I'm not sure what to do now, it will take some strong will (and even more time) for me to re-install EXT3. I just shouldn't have to, dammit. :-p

Well, that is a pretty easy way to rule out many possibilities from my point of view. :-p If it's difficult, we can try to find out where the flag is being set with debug patches.

If you send some patches I'll make every effort to test, it beats having to re-install, my installation is just too customized. :)

Regards,
Niel

#32 lore raw
Quoted 2 lines · Niel Lambrechts

If you send some patches I'll make every effort to test, it beats having to re-install, my installation is just too customized. :)

First, let's make sure we aren't balking up the wrong tree. Can you please apply the attached patch and report the kernel log?

Thanks.

Signature · 1 line
tejun
drivers/scsi/scsi_error.c +15 -1
index 0c2c73b..d92f98c 100644
@@ -1223,19 +1223,31 @@ static void scsi_eh_offline_sdevs(struct list_head *work_q,
  */
 int scsi_noretry_cmd(struct scsi_cmnd *scmd)
 {
+	printk("XXX scmd->result=0x%x ff_t=%d ff_dev=%d ff_drv=%d\n",
+	       scmd->result, blk_failfast_transport(scmd->request),
+	       blk_failfast_dev(scmd->request),
+	       blk_failfast_driver(scmd->request));
+
 	switch (host_byte(scmd->result)) {
 	case DID_OK:
+		printk("XXX DID_OK\n");
 		break;
 	case DID_BUS_BUSY:
+		printk("XXX DID_BUS_BUSY, returning ff_t\n");
 		return blk_failfast_transport(scmd->request);
 	case DID_PARITY:
+		printk("XXX DID_PARITY, returning ff_dev\n");
 		return blk_failfast_dev(scmd->request);
 	case DID_ERROR:
+		printk("XXX DID_ERROR\n");
 		if (msg_byte(scmd->result) == COMMAND_COMPLETE &&
-		    status_byte(scmd->result) == RESERVATION_CONFLICT)
+		    status_byte(scmd->result) == RESERVATION_CONFLICT) {
+			printk("XXX RESERVATION_CONFLICT\n");
 			return 0;
+		}
 		/* fall through */
 	case DID_SOFT_ERROR:
+		printk("XXX DID_SOFT_ERROR, returning ff_drv\n");
 		return blk_failfast_driver(scmd->request);
 	}
 
@@ -1245,9 +1257,11 @@ int scsi_noretry_cmd(struct scsi_cmnd *scmd)
 		 * assume caller has checked sense and determinted
 		 * the check condition was retryable.
 		 */
+		printk("XXX CHECK_CONDITION, returning ff_dev\n");
 		return blk_failfast_dev(scmd->request);
 	}
 
+	printk("XXX returning 0\n");
 	return 0;
 }
 
#33 lore raw
Quoted 7 lines · Tejun Heo
Quoted 3 lines · Niel Lambrechts

If you send some patches I'll make every effort to test, it beats having to re-install, my installation is just too customized. :)

First, let's make sure we aren't balking up the wrong tree. Can you please apply the attached patch and report the kernel log?

Hi Tejun,

Okay it took 5 attempts, some of during which I played audio, did 'find /' etc. but I still do not have a clue whether the extra activity helped trigger it or not.

linux-7vph:/# dumpe2fs -h /dev/sda6
dumpe2fs 1.41.4 (27-Jan-2009)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID:          545370aa-9fb4-4efe-be74-0a64c98c9dad
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3016944
Block count:              12060082
Reserved block count:     361802
Free blocks:              4666750
Free inodes:              2471577
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1021
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8176
Inode blocks per group:   511
Filesystem created:       Tue Sep 30 22:03:02 2008
Last mount time:          Tue May 26 20:37:30 2009
Last write time:          Tue May 26 19:04:08 2009
Mount count:              5
Maximum mount count:      -1
Last checked:             Sat May 23 14:33:27 2009
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Journal inode:            8
First orphan inode:       2330902
Default directory hash:   tea
Directory Hash Seed:      a53491df-10b7-4f13-ad3e-279c67192359
Journal backup:           inode blocks
Journal size:             128M

Regards,
Niel

May 26 19:04:07 linux-7vph pulseaudio[3690]: alsa-sink.c: Increasing wakeup watermark to 50.00 ms
May 26 19:04:08 linux-7vph kernel: Syncing filesystems ... done.
May 26 19:04:09 linux-7vph kernel: Freezing user space processes ... (elapsed 0.00 seconds) done.
May 26 19:04:09 linux-7vph kernel: Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
May 26 19:04:09 linux-7vph kernel: PM: Shrinking memory... done (41460 pages freed)
May 26 19:04:09 linux-7vph kernel: PM: Freed 165840 kbytes in 3.62 seconds (45.81 MB/s)
May 26 19:04:09 linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
May 26 19:04:09 linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
May 26 19:04:09 linux-7vph kernel: ACPI handle has no context!
May 26 19:04:09 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
May 26 19:04:09 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
May 26 19:04:09 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
May 26 19:04:09 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
May 26 19:04:09 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
May 26 19:04:09 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
May 26 19:04:09 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
May 26 19:04:09 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
May 26 19:04:09 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
May 26 19:04:09 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
May 26 19:04:10 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
May 26 19:04:10 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
May 26 19:04:10 linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
May 26 19:04:10 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
May 26 19:04:10 linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
May 26 19:04:10 linux-7vph kernel: PM: Saving platform NVS memory
May 26 19:04:10 linux-7vph kernel: Disabling non-boot CPUs ...
May 26 19:04:10 linux-7vph kernel: CPU 1 is now offline
May 26 19:04:10 linux-7vph kernel: SMP alternatives: switching to UP code
May 26 19:04:10 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 26 19:04:10 linux-7vph kernel: CPU1 attaching NULL sched-domain.
May 26 19:04:10 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 26 19:04:10 linux-7vph kernel: CPU1 is down
May 26 19:04:10 linux-7vph kernel: Extended CMOS year: 2000
May 26 19:04:10 linux-7vph kernel: PM: Creating hibernation image: 
May 26 19:04:10 linux-7vph kernel: PM: Need to copy 124671 pages
May 26 19:04:10 linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
May 26 19:04:10 linux-7vph kernel: Intel machine check architecture supported.
May 26 19:04:10 linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
May 26 19:04:10 linux-7vph kernel: PM: Restoring platform NVS memory
May 26 19:04:10 linux-7vph kernel: Extended CMOS year: 2000
May 26 19:04:10 linux-7vph kernel: Enabling non-boot CPUs ...
May 26 19:04:10 linux-7vph kernel: SMP alternatives: switching to SMP code
May 26 19:04:10 linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
May 26 19:04:10 linux-7vph kernel: Initializing CPU#1
May 26 19:04:10 linux-7vph kernel: Calibrating delay using timer specific routine.. 5054.10 BogoMIPS (lpj=10108210)
May 26 19:04:10 linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 26 19:04:10 linux-7vph kernel: CPU: L2 cache: 6144K
May 26 19:04:10 linux-7vph kernel: CPU: Physical Processor ID: 0
May 26 19:04:10 linux-7vph kernel: CPU: Processor Core ID: 1
May 26 19:04:10 linux-7vph kernel: Intel machine check architecture supported.
May 26 19:04:10 linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
May 26 19:04:10 linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
May 26 19:04:10 linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
May 26 19:04:10 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 26 19:04:10 linux-7vph kernel: Switched to high resolution mode on CPU 1
May 26 19:04:10 linux-7vph kernel: CPU0 attaching sched-domain:
May 26 19:04:10 linux-7vph kernel:  domain 0: span 0-1 level MC
May 26 19:04:10 linux-7vph kernel:   groups: 0 1
May 26 19:04:10 linux-7vph kernel:   domain 1: span 0-1 level CPU
May 26 19:04:10 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
May 26 19:04:11 linux-7vph kernel: CPU1 attaching sched-domain:
May 26 19:04:11 linux-7vph kernel:  domain 0: span 0-1 level MC
May 26 19:04:11 linux-7vph kernel:   groups: 1 0
May 26 19:04:11 linux-7vph kernel:   domain 1: span 0-1 level CPU
May 26 19:04:11 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
May 26 19:04:11 linux-7vph kernel: microcode: failed to init CPU1
May 26 19:04:11 linux-7vph kernel: CPU1 is up
May 26 19:04:11 linux-7vph kernel: ACPI: Waking up from system sleep state S4
May 26 19:04:11 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900007, writing 0x900403)
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
May 26 19:04:11 linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
May 26 19:04:11 linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
May 26 19:04:11 linux-7vph kernel: pci 0000:00:02.0: PME# disabled
May 26 19:04:11 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
May 26 19:04:11 linux-7vph kernel: pci 0000:00:02.1: PME# disabled
May 26 19:04:11 linux-7vph kernel: pci 0000:00:03.0: PME# disabled
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
May 26 19:04:11 linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
May 26 19:04:11 linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
May 26 19:04:12 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
May 26 19:04:12 linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
May 26 19:04:12 linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
May 26 19:04:12 linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
May 26 19:04:12 linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
May 26 19:04:12 linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100506, writing 0x100106)
May 26 19:04:12 linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
May 26 19:04:12 linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
May 26 19:04:12 linux-7vph kernel: pci 0000:15:00.2: PME# disabled
May 26 19:04:12 linux-7vph kernel: pci 0000:15:00.3: PME# disabled
May 26 19:04:12 linux-7vph kernel: pci 0000:15:00.4: PME# disabled
May 26 19:04:12 linux-7vph kernel: pci 0000:15:00.5: PME# disabled
May 26 19:04:12 linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
May 26 19:04:12 linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 26 19:04:12 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 26 19:04:12 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 26 19:04:12 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata1.00: configured for UDMA/133
May 26 19:04:12 linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
May 26 19:04:12 linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
May 26 19:04:12 linux-7vph kernel: ata1.00: configured for UDMA/133
May 26 19:04:12 linux-7vph kernel: ata1: EH complete
May 26 19:04:12 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
May 26 19:04:12 linux-7vph kernel: ata2.00: configured for UDMA/133
May 26 19:04:12 linux-7vph kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
May 26 19:04:12 linux-7vph kernel: ata2: irq_stat 0x40000001
May 26 19:04:12 linux-7vph kernel: ata2.00: configured for UDMA/133
May 26 19:04:12 linux-7vph kernel: ata2: EH complete
May 26 19:04:12 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900407, writing 0x900403)
May 26 19:04:12 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
May 26 19:04:12 linux-7vph kernel: Registered led device: iwl-phy1::radio
May 26 19:04:12 linux-7vph kernel: Registered led device: iwl-phy1::assoc
May 26 19:04:12 linux-7vph kernel: Registered led device: iwl-phy1::RX
May 26 19:04:12 linux-7vph kernel: Registered led device: iwl-phy1::TX
May 26 19:04:12 linux-7vph kernel: mac80211-phy1: failed to set key (0, 00:1d:92:1d:1e:8e) to hardware (-22)
May 26 19:04:12 linux-7vph kernel: Restarting tasks ... done.
May 26 19:04:12 linux-7vph kernel: ata1.00: exception Emask 0x10 SAct 0x7fffffff SErr 0x50000 action 0xe frozen
May 26 19:04:12 linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed
May 26 19:04:12 linux-7vph kernel: ata1: SError: { PHYRdyChg CommWake }
May 26 19:04:12 linux-7vph kernel: ata1.00: cmd 60/40:00:d1:6e:b8/00:00:09:00:00/40 tag 0 ncq 32768 in
May 26 19:04:12 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:12 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:15 linux-7vph kernel: ata1.00: cmd 60/28:08:51:72:b8/00:00:09:00:00/40 tag 1 ncq 20480 in
May 26 19:04:15 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:15 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:15 linux-7vph kernel: ata1.00: cmd 60/08:10:67:88:70/00:00:0e:00:00/40 tag 2 ncq 4096 in
May 26 19:04:15 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:15 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:15 linux-7vph kernel: ata1.00: cmd 60/08:18:6f:88:6f/01:00:0e:00:00/40 tag 3 ncq 135168 in
May 26 19:04:16 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:16 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:16 linux-7vph kernel: ata1.00: cmd 60/40:20:91:55:be/00:00:09:00:00/40 tag 4 ncq 32768 in
May 26 19:04:16 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:16 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:16 linux-7vph kernel: ata1.00: cmd 60/40:28:d1:55:be/00:00:09:00:00/40 tag 5 ncq 32768 in
May 26 19:04:16 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:16 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:16 linux-7vph kernel: ata1.00: cmd 60/40:30:91:56:be/00:00:09:00:00/40 tag 6 ncq 32768 in
May 26 19:04:16 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:38:51:56:be/00:00:09:00:00/40 tag 7 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:40:11:57:be/00:00:09:00:00/40 tag 8 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:48:51:57:be/00:00:09:00:00/40 tag 9 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/08:50:91:56:b7/00:00:09:00:00/40 tag 10 ncq 4096 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/08:58:5f:08:c8/00:00:0c:00:00/40 tag 11 ncq 4096 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/10:60:91:5f:b8/00:00:09:00:00/40 tag 12 ncq 8192 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:68:91:76:b8/00:00:09:00:00/40 tag 13 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:70:11:7c:b8/00:00:09:00:00/40 tag 14 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:78:11:7d:b8/00:00:09:00:00/40 tag 15 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:80:11:58:be/00:00:09:00:00/40 tag 16 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:88:d1:57:be/00:00:09:00:00/40 tag 17 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:90:51:6f:b8/00:00:09:00:00/40 tag 18 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:98:91:6f:b8/00:00:09:00:00/40 tag 19 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/28:a0:d1:6f:b8/00:00:09:00:00/40 tag 20 ncq 20480 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/10:a8:c1:6e:b8/00:00:09:00:00/40 tag 21 ncq 8192 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/38:b0:19:6a:b8/00:00:09:00:00/40 tag 22 ncq 28672 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:b8:11:70:b8/00:00:09:00:00/40 tag 23 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:c0:51:70:b8/00:00:09:00:00/40 tag 24 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:c8:91:70:b8/00:00:09:00:00/40 tag 25 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:d0:d1:70:b8/00:00:09:00:00/40 tag 26 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:d8:11:71:b8/00:00:09:00:00/40 tag 27 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:e0:51:71:b8/00:00:09:00:00/40 tag 28 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:e8:91:71:b8/00:00:09:00:00/40 tag 29 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1.00: cmd 60/40:f0:d1:71:b8/00:00:09:00:00/40 tag 30 ncq 32768 in
May 26 19:04:17 linux-7vph kernel:          res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)
May 26 19:04:17 linux-7vph kernel: ata1.00: status: { DRDY }
May 26 19:04:17 linux-7vph kernel: ata1: hard resetting link
May 26 19:04:17 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 26 19:04:17 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 26 19:04:17 linux-7vph kernel: ata1.00: configured for UDMA/133
May 26 19:04:17 linux-7vph kernel: ata1.00: configured for UDMA/133
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=4 ff_dev=2 ff_drv=8
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 26 19:04:17 linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
May 26 19:04:17 linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
May 26 19:04:17 linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
May 26 19:04:17 linux-7vph kernel:         09 b8 71 d1 
May 26 19:04:17 linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
May 26 19:04:17 linux-7vph kernel: end_request: I/O error, dev sda, sector 242190447
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: XXX scmd->result=0x8000002 ff_t=0 ff_dev=0 ff_drv=0
May 26 19:04:17 linux-7vph kernel: XXX DID_OK
May 26 19:04:17 linux-7vph kernel: XXX CHECK_CONDITION, returning ff_dev
May 26 19:04:17 linux-7vph kernel: ata1: EH complete
May 26 19:04:17 linux-7vph kernel: EXT4-fs error (device sda6): __ext4_get_inode_loc: unable to read inode block - inode=2346519, block=9404418
May 26 19:04:17 linux-7vph kernel: EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure
May 26 19:04:17 linux-7vph kernel: wlan0: no probe response from AP 00:1d:92:1d:1e:8e - disassociating
May 26 19:04:18 linux-7vph kernel: wlan0: authenticate with AP 00:1d:92:1d:1e:8e
May 26 19:04:18 linux-7vph kernel: wlan0: authenticated
May 26 19:04:18 linux-7vph kernel: wlan0: associate with AP 00:1d:92:1d:1e:8e
May 26 19:04:18 linux-7vph kernel: wlan0: RX ReassocResp from 00:1d:92:1d:1e:8e (capab=0x411 status=0 aid=1)
May 26 19:04:18 linux-7vph kernel: wlan0: associated
May 26 19:04:20 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 26 19:04:20 linux-7vph kernel: CPU1 attaching NULL sched-domain.
May 26 19:04:20 linux-7vph kernel: CPU0 attaching sched-domain:
May 26 19:04:20 linux-7vph kernel:  domain 0: span 0-1 level MC
May 26 19:04:20 linux-7vph kernel:   groups: 0 1
May 26 19:04:20 linux-7vph kernel:   domain 1: span 0-1 level CPU
May 26 19:04:20 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
May 26 19:04:20 linux-7vph kernel: CPU1 attaching sched-domain:
May 26 19:04:20 linux-7vph kernel:  domain 0: span 0-1 level MC
May 26 19:04:20 linux-7vph kernel:   groups: 1 0
May 26 19:04:20 linux-7vph kernel:   domain 1: span 0-1 level CPU
May 26 19:04:20 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
#34 lore raw
Quoted 6 lines · Niel Lambrechts
Quoted 5 lines · Tejun Heo
Quoted 3 lines · Niel Lambrechts

If you send some patches I'll make every effort to test, it beats having to re-install, my installation is just too customized. :)

Quoted 8 lines
Quoted 2 lines

First, let's make sure we aren't balking up the wrong tree. Can you please apply the attached patch and report the kernel log?

Hi Tejun,

Okay it took 5 attempts, some of during which I played audio, did 'find /' etc. but I still do not have a clue whether the extra activity helped trigger it or not.

Thanks for testing.

XXX scmd->result=0x8000002 ff_t=4 ff_dev=2 ff_drv=8
XXX DID_OK
XXX CHECK_CONDITION, returning ff_dev
sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor] Descriptor sense data with sense descriptors (in hex):

	 72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
	 09 b8 71 d1 

sd 0:0:0:0: [sda] Add. Sense: No additional sense information
end_request: I/O error, dev sda, sector 242190447

The above is the offending failure and all three failfast bits are set. This corresponds to the following ATA exception.

ata1.00: cmd 60/08:18:6f:88:6f/01:00:0e:00:00/40 tag 3 ncq 135168 in

	  res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)

It's 33 page long read command. Looking at the code the only way all three fastfail bits can be set seems to be if the request is readahead
- the first part of block/blk-core.c::init_request_from_bio(). Now, the failure of a readahead request isn't supposed to cause any problem. If it fails, well, it fails and things should go on as if nothing happened.

Can you please try the attached patch? It takes suspend/resume cycle out of the equation and simply induces artificial failure to readahead requests. It's currently set to fail every 40th readahead. Feel free to adjust the frequency as you see fit. catting files into /dev/null would trigger readahead to kick in. Can you reproduce filesystem failure with this alone?

Thanks.

Signature · 1 line
tejun
block/blk-core.c +11 -1
index c89883b..9b11aea 100644
@@ -163,8 +163,18 @@ static void req_bio_endio(struct request *rq, struct bio *bio,
 		if (bio_integrity(bio))
 			bio_integrity_advance(bio, nbytes);
 
-		if (bio->bi_size == 0)
+		if (bio->bi_size == 0) {
+			static unsigned cnt;
+			if (bio_rw_ahead(bio) && !error && !(++cnt % 40)) {
+				printk("XXX %s: failing readahead bio, "
+				       "sec=%llu f=0x%lx rw=0x%lx\n",
+				       rq->rq_disk ? rq->rq_disk->disk_name : "?",
+				       (unsigned long long)bio->bi_sector,
+				       bio->bi_flags, bio->bi_rw);
+				error = -EIO;
+			}
 			bio_endio(bio, error);
+		}
 	} else {
 
 		/*
#35 lore raw
Quoted 13 lines · Tejun Heo

The above is the offending failure and all three failfast bits are set. This corresponds to the following ATA exception.

ata1.00: cmd 60/08:18:6f:88:6f/01:00:0e:00:00/40 tag 3 ncq 135168 in

	  res 50/00:40:d1:71:b8/00:00:09:00:00/40 Emask 0x10 (ATA bus error)

It's 33 page long read command. Looking at the code the only way all three fastfail bits can be set seems to be if the request is readahead
- the first part of block/blk-core.c::init_request_from_bio(). Now, the failure of a readahead request isn't supposed to cause any problem. If it fails, well, it fails and things should go on as if nothing happened.

Quoted 7 lines

Can you please try the attached patch? It takes suspend/resume cycle out of the equation and simply induces artificial failure to readahead requests. It's currently set to fail every 40th readahead. Feel free to adjust the frequency as you see fit. catting files into /dev/null would trigger readahead to kick in. Can you reproduce filesystem failure with this alone?

No corruption, I tried cat on an entire directory of mp3s, then also started X, but there the messages debug output got fairly drastic so I didn't care to continue for an entire day.

It did trigger the "XXX ...failing readahead" message nearly 300 times, and I also did a s2disk and resume cycle in there - so I hope this is enough for us to conclude that it is not the cause.

Regards,
Niel

May 27 08:51:11 linux-7vph syslog-ng[2175]: syslog-ng starting up; version='2.0.9'
May 27 08:51:12 linux-7vph auditd[2262]: Started dispatcher: /sbin/audispd pid: 2269
May 27 08:51:12 linux-7vph audispd: priority_boost_parser called with: 4
May 27 08:51:12 linux-7vph audispd: af_unix plugin initialized
May 27 08:51:12 linux-7vph audispd: audispd initialized with q_depth=80 and 1 active plugins
May 27 08:51:12 linux-7vph auditd[2262]: Init complete, auditd 1.7.7 listening for events (startup state disable)
May 27 08:51:13 linux-7vph network: Starting the NetworkManager
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Found user 'avahi' (UID 102) and group 'avahi' (GID 104).
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Successfully dropped root privileges.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: avahi-daemon 0.6.25 starting up.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Loading service file /etc/avahi/services/sftp-ssh.service.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Loading service file /etc/avahi/services/ssh.service.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Network interface enumeration completed.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Registering HINFO record with values 'I686'/'LINUX'.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Server startup complete. Host name is linux-7vph.local. Local service cookie is 2123976226.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Service "linux-7vph" (/etc/avahi/services/ssh.service) successfully established.
May 27 08:51:14 linux-7vph avahi-daemon[2481]: Service "SFTP File Transfer on linux-7vph" (/etc/avahi/services/sftp-ssh.service) successfully established.
May 27 08:51:15 linux-7vph dbus-daemon: Rejected send message, 11 matched rules; type="error", sender=":1.7" (uid=0 pid=2506 comm="/usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wp") interface="(unset)" member="(unset)" error name="fi.epitest.hostap.WPASupplicant.InvalidInterface" requested_reply=0 destination=":1.6" (uid=0 pid=2500 comm="/usr/sbin/NetworkManager "))
May 27 08:51:15 linux-7vph avahi-dnsconfd[2504]: Successfully connected to Avahi daemon.
May 27 08:51:15 linux-7vph smartd[2464]: smartd 5.39 2008-10-24 22:33 [i686-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce Allen, http://smartmontools.sourceforge.net
May 27 08:51:15 linux-7vph smartd[2464]: Opened configuration file /etc/smartd.conf
May 27 08:51:15 linux-7vph modem-manager: Loaded plugin MBM
May 27 08:51:15 linux-7vph smartd[2464]: Drive: DEVICESCAN, implied '-a' Directive on line 26 of file /etc/smartd.conf
May 27 08:51:15 linux-7vph smartd[2464]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
May 27 08:51:15 linux-7vph modem-manager: Loaded plugin Sierra
May 27 08:51:15 linux-7vph smartd[2464]: Device: /dev/sda, type changed from 'scsi' to 'sat'
May 27 08:51:15 linux-7vph smartd[2464]: Device: /dev/sda [SAT], opened
May 27 08:51:15 linux-7vph smartd[2464]: Device: /dev/sda [SAT], not found in smartd database.
May 27 08:51:15 linux-7vph modem-manager: Loaded plugin HSO
May 27 08:51:15 linux-7vph modem-manager: Loaded plugin Option
May 27 08:51:15 linux-7vph modem-manager: Loaded plugin Novatel
May 27 08:51:16 linux-7vph modem-manager: Loaded plugin Huawei
May 27 08:51:16 linux-7vph kernel: klogd 1.4.1, log source = /proc/kmsg started.
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=239569119 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=239569679 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=241928783 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=169323911 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=169577527 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=169577607 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=174296263 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=261327887 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=171413327 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=169839239 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=171149535 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=176130599 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=260541031 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=175606431 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=169839103 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=261852487 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=260540559 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=172986079 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=176655903 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=169839719 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph smartd[2464]: Device: /dev/sda [SAT], is SMART capable. Adding to "monitor" list.
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=261066303 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=175605903 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=173771455 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=174296079 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=175082583 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=175868063 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=261589215 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=242453519 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph /usr/sbin/cron[2583]: (CRON) STARTUP (V5.0)
May 27 08:51:17 linux-7vph smartd[2464]: Device: /dev/sda [SAT], state read from /var/lib/smartmontools/smartd.HITACHI_HTS722020K9SA00__FDE-080726DP0470DTGMUNPC.ata.state
May 27 08:51:17 linux-7vph smartd[2464]: Monitoring 1 ATA and 0 SCSI devices
May 27 08:51:16 linux-7vph kernel: XXX sda: failing readahead bio, sec=171411823 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=242190503 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=241667031 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=242715415 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=242714935 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=242190719 f=0xf0000009 rw=0x2
May 27 08:51:16 linux-7vph sshd[2532]: Server listening on 0.0.0.0 port 22.
May 27 08:51:17 linux-7vph sshd[2532]: Server listening on :: port 22.
May 27 08:51:17 linux-7vph smartd[2464]: Device: /dev/sda [SAT], SMART Prefailure Attribute: 3 Spin_Up_Time changed from 181 to 180
May 27 08:51:17 linux-7vph nm-system-settings: Loaded plugin ifcfg-suse: (C) 2008 Novell, Inc.	To report bugs please use the NetworkManager mailing list.
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=242453183 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=218859799 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: platform microcode: firmware: requesting intel-ucode/06-17-06
May 27 08:51:17 linux-7vph kernel: platform microcode: firmware: requesting intel-ucode/06-17-06
May 27 08:51:17 linux-7vph kernel: Microcode Update Driver: v2.00 <[email protected]>, Peter Oruba
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=252679759 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph smartd[2464]: Device: /dev/sda [SAT], state written to /var/lib/smartmontools/smartd.HITACHI_HTS722020K9SA00__FDE-080726DP0470DTGMUNPC.ata.state
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=252938895 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=252939223 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: NET: Registered protocol family 10
May 27 08:51:17 linux-7vph kernel: lo: Disabled Privacy Extensions
May 27 08:51:17 linux-7vph kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=208374063 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=181110927 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=253989599 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 08:51:17 linux-7vph smartd[2587]: smartd has fork()ed into background mode. New PID=2587.
May 27 08:51:17 linux-7vph nm-system-settings: Loaded plugin keyfile: (c) 2007 - 2008 Red Hat, Inc.  To report bugs please use the NetworkManager mailing list.
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=253990687 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 08:51:17 linux-7vph kernel: CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
May 27 08:51:17 linux-7vph kernel: nf_conntrack.acct=1 kernel paramater, acct=1 nf_conntrack module option or
May 27 08:51:17 linux-7vph kernel: sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=253991015 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=253462951 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=253465063 f=0xf0000009 rw=0x2
May 27 08:51:17 linux-7vph kernel: XXX sda: failing readahead bio, sec=253464871 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253462647 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=188451031 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253463551 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253463375 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253464479 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=180062463 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=250579207 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=242192183 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=252676999 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=217024663 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=255299119 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=212568191 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=240355647 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=167751023 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253726895 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=219384823 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253726783 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=168798599 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=168799687 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=258182415 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=258182215 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=256879783 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=256879343 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=223054143 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=181112695 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=253725679 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=177441095 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph SuSEfirewall2: Setting up rules from /etc/sysconfig/SuSEfirewall2 ...
May 27 08:51:18 linux-7vph SuSEfirewall2: using default zone 'ext' for interface wlan0
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=254249287 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=255823055 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph SuSEfirewall2: using default zone 'ext' for interface wmaster0
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=177441543 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph kernel: XXX sda: failing readahead bio, sec=252679247 f=0xf0000009 rw=0x2
May 27 08:51:18 linux-7vph SuSEfirewall2: batch committing...
May 27 08:51:18 linux-7vph SuSEfirewall2: Firewall rules successfully set
May 27 08:51:19 linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
May 27 08:51:19 linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
May 27 08:51:19 linux-7vph kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 27 08:51:19 linux-7vph kernel: iwlagn 0000:03:00.0: firmware: requesting iwlwifi-5000-2.ucode
May 27 08:51:19 linux-7vph kernel: iwlagn 0000:03:00.0: iwlwifi-5000-2.ucode firmware file req failed: -2
May 27 08:51:19 linux-7vph kernel: iwlagn 0000:03:00.0: firmware: requesting iwlwifi-5000-1.ucode
May 27 08:51:19 linux-7vph firmware.sh[2701]: Cannot find  firmware file 'iwlwifi-5000-2.ucode'
May 27 08:51:19 linux-7vph kernel: iwlagn 0000:03:00.0: Loaded firmware iwlwifi-5000-1.ucode, which is deprecated. Please use API v2 instead.
May 27 08:51:19 linux-7vph kernel: iwlagn 0000:03:00.0: Firmware has old API version. Expected v2, got v1. New firmware can be obtained from http://www.intellinuxwireless.org.
May 27 08:51:19 linux-7vph kernel: iwlagn 0000:03:00.0: loaded firmware version 5.4.1.16
May 27 08:51:19 linux-7vph kernel: Registered led device: iwl-phy0::radio
May 27 08:51:19 linux-7vph kernel: Registered led device: iwl-phy0::assoc
May 27 08:51:19 linux-7vph kernel: Registered led device: iwl-phy0::RX
May 27 08:51:19 linux-7vph kernel: Registered led device: iwl-phy0::TX
May 27 08:51:19 linux-7vph kernel: ADDRCONF(NETDEV_UP): wlan0: link is not ready
May 27 08:51:19 linux-7vph kernel: NET: Registered protocol family 17
May 27 08:51:21 linux-7vph nm-system-settings: Adding default connection 'Auto eth0' for /org/freedesktop/Hal/devices/net_00_1c_25_95_93_ec
May 27 08:51:22 linux-7vph dhclient: Internet Systems Consortium DHCP Client V3.1.1
May 27 08:51:22 linux-7vph dhclient: Copyright 2004-2008 Internet Systems Consortium.
May 27 08:51:22 linux-7vph dhclient: All rights reserved.
May 27 08:51:22 linux-7vph dhclient: For info, please visit http://www.isc.org/sw/dhcp/
May 27 08:51:22 linux-7vph dhclient: 
May 27 08:51:22 linux-7vph dhclient: Listening on LPF/eth0/00:1c:25:95:93:ec
May 27 08:51:22 linux-7vph dhclient: Sending on   LPF/eth0/00:1c:25:95:93:ec
May 27 08:51:22 linux-7vph dhclient: Sending on   Socket/fallback
May 27 08:51:22 linux-7vph kernel: XXX sda: failing readahead bio, sec=240094111 f=0xf0000009 rw=0x2
May 27 08:51:22 linux-7vph kernel: XXX sda: failing readahead bio, sec=256608487 f=0xf0000009 rw=0x2
May 27 08:51:22 linux-7vph kernel: XXX sda: failing readahead bio, sec=256609567 f=0xf0000009 rw=0x2
May 27 08:51:22 linux-7vph kernel: XXX sda: failing readahead bio, sec=257133663 f=0xf0000009 rw=0x2
May 27 08:51:23 linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5
May 27 08:51:23 linux-7vph kernel: XXX sda: failing readahead bio, sec=241142967 f=0xf0000009 rw=0x2
May 27 08:51:23 linux-7vph kernel: XXX sda: failing readahead bio, sec=195529471 f=0xf0000009 rw=0x2
May 27 08:51:23 linux-7vph kernel: XXX sda: failing readahead bio, sec=240356671 f=0xf0000009 rw=0x2
May 27 08:51:23 linux-7vph kernel: XXX sda: failing readahead bio, sec=252679303 f=0xf0000009 rw=0x2
May 27 08:51:23 linux-7vph kernel: XXX sda: failing readahead bio, sec=252678871 f=0xf0000009 rw=0x2
May 27 08:51:24 linux-7vph kernel: XXX sda: failing readahead bio, sec=258968615 f=0xf0000009 rw=0x2
May 27 08:51:24 linux-7vph kernel: XXX sda: failing readahead bio, sec=258967927 f=0xf0000009 rw=0x2
May 27 08:51:24 linux-7vph kernel: XXX sda: failing readahead bio, sec=258967991 f=0xf0000009 rw=0x2
May 27 08:51:24 linux-7vph kernel: XXX sda: failing readahead bio, sec=252678407 f=0xf0000009 rw=0x2
May 27 08:51:24 linux-7vph kernel: wlan0: authenticate with AP 00:1d:92:1d:1e:8e
May 27 08:51:24 linux-7vph kernel: wlan0: authenticated
May 27 08:51:24 linux-7vph kernel: wlan0: associate with AP 00:1d:92:1d:1e:8e
May 27 08:51:24 linux-7vph kernel: wlan0: RX AssocResp from 00:1d:92:1d:1e:8e (capab=0x411 status=0 aid=1)
May 27 08:51:24 linux-7vph kernel: wlan0: associated
May 27 08:51:24 linux-7vph kernel: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
May 27 08:51:24 linux-7vph kernel: wlan0: authenticate with AP 00:1d:92:1d:1e:8e
May 27 08:51:24 linux-7vph kernel: wlan0: authenticate with AP 00:1d:92:1d:1e:8e
May 27 08:51:24 linux-7vph kernel: XXX sda: failing readahead bio, sec=241666911 f=0xf0000009 rw=0x2
May 27 08:51:24 linux-7vph kernel: wlan0: authenticated
May 27 08:51:24 linux-7vph kernel: wlan0: associate with AP 00:1d:92:1d:1e:8e
May 27 08:51:24 linux-7vph kernel: wlan0: RX ReassocResp from 00:1d:92:1d:1e:8e (capab=0x411 status=0 aid=1)
May 27 08:51:24 linux-7vph kernel: wlan0: associated
May 27 08:51:25 linux-7vph dhclient: Internet Systems Consortium DHCP Client V3.1.1
May 27 08:51:25 linux-7vph dhclient: Copyright 2004-2008 Internet Systems Consortium.
May 27 08:51:25 linux-7vph dhclient: All rights reserved.
May 27 08:51:25 linux-7vph dhclient: For info, please visit http://www.isc.org/sw/dhcp/
May 27 08:51:25 linux-7vph dhclient: 
May 27 08:51:25 linux-7vph dhclient: Listening on LPF/wlan0/00:16:ea:b2:33:a0
May 27 08:51:25 linux-7vph dhclient: Sending on   LPF/wlan0/00:16:ea:b2:33:a0
May 27 08:51:25 linux-7vph dhclient: Sending on   Socket/fallback
May 27 08:51:26 linux-7vph avahi-daemon[2481]: Registering new address record for fe80::216:eaff:feb2:33a0 on wlan0.*.
May 27 08:51:27 linux-7vph dhclient: DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 2
May 27 08:51:28 linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13
May 27 08:51:28 linux-7vph dhclient: DHCPOFFER from 192.168.1.254
May 27 08:51:32 linux-7vph dhclient: DHCPREQUEST on wlan0 to 255.255.255.255 port 67
May 27 08:51:32 linux-7vph dhclient: DHCPACK from 192.168.1.254
May 27 08:51:32 linux-7vph dhclient: bound to 192.168.1.100 -- renewal in 1373 seconds.
May 27 08:51:32 linux-7vph avahi-daemon[2481]: Joining mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.100.
May 27 08:51:32 linux-7vph avahi-daemon[2481]: New relevant interface wlan0.IPv4 for mDNS.
May 27 08:51:32 linux-7vph avahi-daemon[2481]: Registering new address record for 192.168.1.100 on wlan0.IPv4.
May 27 08:51:33 linux-7vph kernel: XXX sda: failing readahead bio, sec=254512783 f=0xf0000009 rw=0x2
May 27 08:51:33 linux-7vph dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched...
May 27 08:51:33 linux-7vph dns-resolver: You can find my version in /etc/resolv.conf.netconfig
May 27 08:51:33 linux-7vph nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/autofs' exited with error status 1.
May 27 08:51:33 linux-7vph kernel: XXX sda: failing readahead bio, sec=258183911 f=0xf0000009 rw=0x2
May 27 08:51:33 linux-7vph SuSEfirewall2: Setting up rules from /etc/sysconfig/SuSEfirewall2 ...
May 27 08:51:33 linux-7vph SuSEfirewall2: using default zone 'ext' for interface wlan0
May 27 08:51:33 linux-7vph SuSEfirewall2: using default zone 'ext' for interface wmaster0
May 27 08:51:34 linux-7vph SuSEfirewall2: batch committing...
May 27 08:51:34 linux-7vph SuSEfirewall2: Firewall rules successfully set
May 27 08:51:34 linux-7vph kernel: XXX sda: failing readahead bio, sec=255297879 f=0xf0000009 rw=0x2
May 27 08:51:34 linux-7vph sm-notify[3027]: Already notifying clients; Exiting!
May 27 08:51:34 linux-7vph nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/nfs' exited with error status 6.
May 27 08:51:35 linux-7vph kernel: wlan0: no IPv6 routers present
May 27 08:51:38 linux-7vph dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched...
May 27 08:51:38 linux-7vph dns-resolver: You can find my version in /etc/resolv.conf.netconfig
May 27 08:51:41 linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10
May 27 08:51:51 linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13
May 27 08:52:04 linux-7vph dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9
May 27 08:52:07 linux-7vph dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched...
May 27 08:52:07 linux-7vph dns-resolver: You can find my version in /etc/resolv.conf.netconfig
May 27 08:54:45 linux-7vph kernel: XXX sda: failing readahead bio, sec=241666431 f=0xf0000009 rw=0x2
May 27 08:54:45 linux-7vph kernel: XXX sda: failing readahead bio, sec=250579159 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=253201407 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=250579287 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=252680063 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=219384023 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=252677383 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=252677191 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=252678039 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=252677839 f=0xf0000009 rw=0x2
May 27 08:56:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=214404639 f=0xf0000009 rw=0x2
May 27 08:56:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=214403455 f=0xf0000009 rw=0x2
May 27 08:56:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=214403783 f=0xf0000009 rw=0x2
May 27 08:56:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=253987343 f=0xf0000009 rw=0x2
May 27 08:56:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=253987919 f=0xf0000009 rw=0x2
May 27 09:51:28 linux-7vph -- MARK --
May 27 09:51:28 linux-7vph kernel: Syncing filesystems ... done.
May 27 09:51:28 linux-7vph syslog-ng[2175]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=752', processed='center(received)=394', processed='destination(newsnotice)=0', processed='destination(acpid)=1', processed='destination(firewall)=11', processed='destination(null)=10', processed='destination(mail)=2', processed='destination(mailinfo)=2', processed='destination(console)=118', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=246', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=124', processed='destination(mailerr)=0', processed='destination(xconsole)=118', processed='destination(warn)=120', processed='source(src)=394'
May 27 09:51:28 linux-7vph kernel: Freezing user space processes ... (elapsed 0.00 seconds) done.
May 27 09:51:28 linux-7vph kernel: Freezing remaining freezable tasks ... (elapsed 0.09 seconds) done.
May 27 09:51:28 linux-7vph kernel: PM: Shrinking memory... done (0 pages freed)
May 27 09:51:28 linux-7vph kernel: PM: Freed 0 kbytes in 0.04 seconds (0.00 MB/s)
May 27 09:51:28 linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
May 27 09:51:28 linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
May 27 09:51:28 linux-7vph kernel: ACPI handle has no context!
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
May 27 09:51:28 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
May 27 09:51:28 linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
May 27 09:51:28 linux-7vph kernel: PM: Saving platform NVS memory
May 27 09:51:28 linux-7vph kernel: Disabling non-boot CPUs ...
May 27 09:51:28 linux-7vph kernel: CPU 1 is now offline
May 27 09:51:28 linux-7vph kernel: SMP alternatives: switching to UP code
May 27 09:51:28 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 27 09:51:28 linux-7vph kernel: CPU1 attaching NULL sched-domain.
May 27 09:51:28 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 27 09:51:28 linux-7vph kernel: CPU1 is down
May 27 09:51:28 linux-7vph kernel: Extended CMOS year: 2000
May 27 09:51:28 linux-7vph kernel: PM: Creating hibernation image: 
May 27 09:51:28 linux-7vph kernel: PM: Need to copy 111037 pages
May 27 09:51:28 linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
May 27 09:51:28 linux-7vph kernel: Intel machine check architecture supported.
May 27 09:51:28 linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
May 27 09:51:28 linux-7vph kernel: PM: Restoring platform NVS memory
May 27 09:51:28 linux-7vph kernel: Extended CMOS year: 2000
May 27 09:51:28 linux-7vph kernel: Enabling non-boot CPUs ...
May 27 09:51:28 linux-7vph kernel: SMP alternatives: switching to SMP code
May 27 09:51:28 linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
May 27 09:51:28 linux-7vph kernel: Initializing CPU#1
May 27 09:51:28 linux-7vph kernel: Calibrating delay using timer specific routine.. 5054.04 BogoMIPS (lpj=10108094)
May 27 09:51:28 linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
May 27 09:51:28 linux-7vph kernel: CPU: L2 cache: 6144K
May 27 09:51:28 linux-7vph kernel: CPU: Physical Processor ID: 0
May 27 09:51:28 linux-7vph kernel: CPU: Processor Core ID: 1
May 27 09:51:28 linux-7vph kernel: Intel machine check architecture supported.
May 27 09:51:28 linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
May 27 09:51:28 linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
May 27 09:51:28 linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
May 27 09:51:28 linux-7vph kernel: CPU0 attaching NULL sched-domain.
May 27 09:51:28 linux-7vph kernel: Switched to high resolution mode on CPU 1
May 27 09:51:28 linux-7vph kernel: CPU0 attaching sched-domain:
May 27 09:51:28 linux-7vph kernel:  domain 0: span 0-1 level MC
May 27 09:51:28 linux-7vph kernel:   groups: 0 1
May 27 09:51:28 linux-7vph kernel: CPU1 attaching sched-domain:
May 27 09:51:28 linux-7vph kernel:  domain 0: span 0-1 level MC
May 27 09:51:28 linux-7vph kernel:   groups: 1 0
May 27 09:51:28 linux-7vph kernel: microcode: failed to init CPU1
May 27 09:51:28 linux-7vph kernel: CPU1 is up
May 27 09:51:28 linux-7vph kernel: ACPI: Waking up from system sleep state S4
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
May 27 09:51:28 linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
May 27 09:51:28 linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
May 27 09:51:28 linux-7vph kernel: pci 0000:00:02.0: PME# disabled
May 27 09:51:28 linux-7vph kernel: pci 0000:00:02.1: PME# disabled
May 27 09:51:28 linux-7vph kernel: pci 0000:00:03.0: PME# disabled
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
May 27 09:51:28 linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
May 27 09:51:28 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
May 27 09:51:28 linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
May 27 09:51:28 linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
May 27 09:51:28 linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
May 27 09:51:28 linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100506, writing 0x100106)
May 27 09:51:28 linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
May 27 09:51:28 linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
May 27 09:51:28 linux-7vph kernel: pci 0000:15:00.2: PME# disabled
May 27 09:51:28 linux-7vph kernel: pci 0000:15:00.3: PME# disabled
May 27 09:51:28 linux-7vph kernel: pci 0000:15:00.4: PME# disabled
May 27 09:51:28 linux-7vph kernel: pci 0000:15:00.5: PME# disabled
May 27 09:51:28 linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
May 27 09:51:28 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 27 09:51:28 linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
May 27 09:51:28 linux-7vph kernel: ata1.00: configured for UDMA/133
May 27 09:51:28 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
May 27 09:51:28 linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
May 27 09:51:28 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
May 27 09:51:28 linux-7vph kernel: ata1.00: configured for UDMA/133
May 27 09:51:28 linux-7vph kernel: ata1: EH complete
May 27 09:51:28 linux-7vph kernel: ata2.00: configured for UDMA/133
May 27 09:51:28 linux-7vph kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
May 27 09:51:28 linux-7vph kernel: ata2: irq_stat 0x40000001
May 27 09:51:28 linux-7vph kernel: ata2.00: configured for UDMA/133
May 27 09:51:28 linux-7vph kernel: ata2: EH complete
May 27 09:51:28 linux-7vph kernel: Registered led device: iwl-phy0::radio
May 27 09:51:28 linux-7vph kernel: Registered led device: iwl-phy0::assoc
May 27 09:51:28 linux-7vph kernel: Registered led device: iwl-phy0::RX
May 27 09:51:28 linux-7vph kernel: Registered led device: iwl-phy0::TX
May 27 09:51:29 linux-7vph kernel: mac80211-phy0: failed to set key (0, 00:1d:92:1d:1e:8e) to hardware (-22)
May 27 09:51:29 linux-7vph kernel: Restarting tasks ... done.
May 27 09:51:29 linux-7vph kernel: XXX sda: failing readahead bio, sec=241666439 f=0xf0000009 rw=0x2
May 27 09:51:29 linux-7vph kernel: XXX sda: failing readahead bio, sec=241667015 f=0xf0000009 rw=0x2
May 27 09:51:29 linux-7vph kernel: wlan0: no probe response from AP 00:1d:92:1d:1e:8e - disassociating
May 27 09:51:29 linux-7vph kernel: e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
May 27 09:51:29 linux-7vph kernel: 0000:00:19.0: eth0: 10/100 speed: disabling TSO
May 27 09:51:29 linux-7vph kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 27 09:51:31 linux-7vph avahi-daemon[2481]: Registering new address record for fe80::21c:25ff:fe95:93ec on eth0.*.
May 27 09:51:37 linux-7vph kernel: martian source 255.255.255.255 from 10.59.202.26, on dev eth0
May 27 09:51:37 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:21:70:b6:08:ad:08:00
May 27 09:51:40 linux-7vph kernel: eth0: no IPv6 routers present
May 27 09:51:44 linux-7vph kernel: XXX sda: failing readahead bio, sec=241666335 f=0xf0000009 rw=0x2
May 27 09:51:44 linux-7vph kernel: XXX sda: failing readahead bio, sec=253991015 f=0xf0000009 rw=0x2
May 27 09:51:44 linux-7vph avahi-daemon[2481]: Withdrawing address record for 192.168.1.100 on wlan0.
May 27 09:51:44 linux-7vph avahi-daemon[2481]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 192.168.1.100.
May 27 09:51:44 linux-7vph avahi-daemon[2481]: Interface wlan0.IPv4 no longer relevant for mDNS.
May 27 09:51:45 linux-7vph kernel: XXX sda: failing readahead bio, sec=242452631 f=0xf0000009 rw=0x2
May 27 09:51:45 linux-7vph kernel: XXX sda: failing readahead bio, sec=214404287 f=0xf0000009 rw=0x2
May 27 09:51:45 linux-7vph dns-resolver: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched...
May 27 09:51:45 linux-7vph dns-resolver: You can find my version in /etc/resolv.conf.netconfig
May 27 09:51:46 linux-7vph nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/autofs' exited with error status 1.
May 27 09:51:46 linux-7vph kernel: XXX sda: failing readahead bio, sec=167479583 f=0xf0000009 rw=0x2
May 27 09:52:39 linux-7vph kernel: martian source 255.255.255.255 from 10.59.202.16, on dev eth0
May 27 09:52:39 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:16:d4:bd:af:8b:08:00
May 27 09:52:40 linux-7vph kernel: martian source 255.255.255.255 from 10.59.202.26, on dev eth0
May 27 09:52:40 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:21:70:b6:08:ad:08:00
May 27 09:52:41 linux-7vph init: Switching to runlevel: 5
May 27 09:52:42 linux-7vph kernel: Kernel logging (proc) stopped.
May 27 09:52:42 linux-7vph kernel: Kernel log daemon terminating.
May 27 09:52:42 linux-7vph syslog-ng[2175]: Termination requested via signal, terminating;
May 27 09:52:42 linux-7vph syslog-ng[2175]: syslog-ng shutting down; version='2.0.9'
May 27 09:52:42 linux-7vph syslog-ng[3583]: syslog-ng starting up; version='2.0.9'
May 27 09:52:46 linux-7vph kdm_config[3622]: Multiple occurrences of key 'UseTheme' in section [X-*-Greeter] of /usr/share/kde4/config/kdm/kdmrc
May 27 09:52:47 linux-7vph kernel: klogd 1.4.1, log source = /proc/kmsg started.
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=242190863 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=205752679 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=257919135 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=174042583 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=174041391 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=174041943 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253724831 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=185567367 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=195266711 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=195267271 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=195791127 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=200247639 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=260016535 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=204703983 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=208111919 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=218859647 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=219384271 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=219384847 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=219384663 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=226461871 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=229083311 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=239569135 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=242190655 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=249530559 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=252676839 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=195267143 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=242453655 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=252678527 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253200831 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253462799 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253463127 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253463951 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253465039 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253725455 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253726295 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253989527 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253990103 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=253990687 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=254250087 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=254511287 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=256084711 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=256085047 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=260278399 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=260278727 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=260279055 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=241928791 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=260016799 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=245860599 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=239569759 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=247171231 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=252679559 f=0xf0000009 rw=0x2
May 27 09:52:47 linux-7vph kernel: XXX sda: failing readahead bio, sec=252679367 f=0xf0000009 rw=0x2
May 27 09:52:48 linux-7vph kernel: XXX sda: failing readahead bio, sec=195267615 f=0xf0000009 rw=0x2
May 27 09:52:48 linux-7vph kernel: [drm] Initialized drm 1.1.0 20060810
May 27 09:52:48 linux-7vph kernel: pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
May 27 09:52:48 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
May 27 09:52:48 linux-7vph kernel: pci 0000:00:02.0: irq 31 for MSI/MSI-X
May 27 09:52:48 linux-7vph kernel: acpi device:03: registered as cooling_device2
May 27 09:52:48 linux-7vph kernel: input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/device:02/input/input12
May 27 09:52:48 linux-7vph kernel: ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
May 27 09:52:48 linux-7vph kernel: [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
May 27 09:52:48 linux-7vph kernel: XXX sda: failing readahead bio, sec=242192223 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=241928871 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=241142175 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=257657079 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=257658663 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=179538047 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=179539639 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=180848975 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=260279423 f=0xf0000009 rw=0x2
May 27 09:52:50 linux-7vph kernel: XXX sda: failing readahead bio, sec=260281031 f=0xf0000009 rw=0x2
May 27 09:52:51 linux-7vph kernel: XXX sda: failing readahead bio, sec=253990367 f=0xf0000009 rw=0x2
May 27 09:52:51 linux-7vph kernel: XXX sda: failing readahead bio, sec=253988391 f=0xf0000009 rw=0x2
May 27 09:52:51 linux-7vph kernel: XXX sda: failing readahead bio, sec=175606135 f=0xf0000009 rw=0x2
May 27 09:52:51 linux-7vph kernel: XXX sda: failing readahead bio, sec=254249351 f=0xf0000009 rw=0x2
May 27 09:52:52 linux-7vph kernel: XXX sda: failing readahead bio, sec=239569919 f=0xf0000009 rw=0x2
May 27 09:52:52 linux-7vph kernel: XXX sda: failing readahead bio, sec=253463879 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=187402399 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=169839055 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=169838871 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=170101111 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=254249159 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=170364439 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=258706519 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=171935903 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=174042351 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=174041687 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=174295975 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=177965303 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=253464879 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=253464607 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=199985527 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=208636079 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=257133647 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=214665367 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=214665695 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=240093751 f=0xf0000009 rw=0x2
May 27 09:52:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=240093823 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=239570119 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=239570447 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=239570775 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240355751 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240617711 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=239831615 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=239831951 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=239832279 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240093343 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240355583 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240094287 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240094623 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240095191 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240095527 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=240356711 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=241404199 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254252119 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=253725871 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=253987071 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=253987671 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254250919 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254251247 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254251575 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254511511 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254511847 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254512175 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254512503 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254773455 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254773783 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=254774871 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=255298231 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=255298807 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=255299135 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=255821959 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=255822295 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=255822623 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=256085351 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=256346487 f=0xf0000009 rw=0x2
May 27 09:52:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=256879535 f=0xf0000009 rw=0x2
May 27 09:52:55 linux-7vph kernel: XXX sda: failing readahead bio, sec=256879855 f=0xf0000009 rw=0x2
May 27 09:52:55 linux-7vph kernel: XXX sda: failing readahead bio, sec=257920311 f=0xf0000009 rw=0x2
May 27 09:52:55 linux-7vph kernel: XXX sda: failing readahead bio, sec=258967695 f=0xf0000009 rw=0x2
May 27 09:52:55 linux-7vph kernel: XXX sda: failing readahead bio, sec=258968031 f=0xf0000009 rw=0x2
May 27 09:52:55 linux-7vph kernel: XXX sda: failing readahead bio, sec=259493407 f=0xf0000009 rw=0x2
May 27 09:52:55 linux-7vph kernel: XXX sda: failing readahead bio, sec=260541279 f=0xf0000009 rw=0x2
May 27 09:53:16 linux-7vph kernel: martian source 255.255.255.255 from 10.59.202.46, on dev eth0
May 27 09:53:16 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:14:38:15:0c:7a:08:00
May 27 09:53:24 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.63, on dev eth0
May 27 09:53:24 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:07:e7:08:00
May 27 09:53:29 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.63, on dev eth0
May 27 09:53:29 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:07:e7:08:00
May 27 09:53:35 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.69, on dev eth0
May 27 09:53:35 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:ed:c1:53:08:00
May 27 09:53:39 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.69, on dev eth0
May 27 09:53:39 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:ed:c1:53:08:00
May 27 09:53:40 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.170, on dev eth0
May 27 09:53:40 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:1d:72:4b:ab:e3:08:00
May 27 09:53:42 linux-7vph kernel: martian source 255.255.255.255 from 10.59.202.16, on dev eth0
May 27 09:53:42 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:16:d4:bd:af:8b:08:00
May 27 09:53:50 linux-7vph kernel: [drm:gm45_get_vblank_counter] *ERROR* trying to get vblank count for disabled pipe 0
May 27 09:54:05 linux-7vph init: Switching to runlevel: 3
May 27 09:54:16 linux-7vph kernel: martian source 255.255.255.255 from 10.59.202.46, on dev eth0
May 27 09:54:16 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:14:38:15:0c:7a:08:00
May 27 09:55:44 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.63, on dev eth0
May 27 09:55:44 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:07:e7:08:00
May 27 09:55:47 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.63, on dev eth0
May 27 09:55:47 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:07:e7:08:00
May 27 09:57:23 linux-7vph kernel: martian source 255.255.255.255 from 10.59.200.42, on dev eth0
May 27 09:57:23 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:03:be:08:00
May 27 09:57:26 linux-7vph kernel: martian source 255.255.255.255 from 10.59.200.42, on dev eth0
May 27 09:57:26 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:03:be:08:00
May 27 09:58:02 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.63, on dev eth0
May 27 09:58:02 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:07:e7:08:00
May 27 09:58:06 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.63, on dev eth0
May 27 09:58:06 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:17:a4:fc:07:e7:08:00
May 27 09:59:08 linux-7vph kernel: XXX sda: failing readahead bio, sec=192645295 f=0xf0000009 rw=0x2
May 27 09:59:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=197364447 f=0xf0000009 rw=0x2
May 27 09:59:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=241143319 f=0xf0000009 rw=0x2
May 27 09:59:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=197364319 f=0xf0000009 rw=0x2
May 27 09:59:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=177441527 f=0xf0000009 rw=0x2
May 27 09:59:09 linux-7vph kernel: XXX sda: failing readahead bio, sec=197364039 f=0xf0000009 rw=0x2
May 27 09:59:14 linux-7vph kernel: XXX sda: failing readahead bio, sec=241142927 f=0xf0000009 rw=0x2
May 27 09:59:51 linux-7vph kernel: XXX sda: failing readahead bio, sec=222529767 f=0xf0000009 rw=0x2
May 27 09:59:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=173771631 f=0xf0000009 rw=0x2
May 27 09:59:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=255560879 f=0xf0000009 rw=0x2
May 27 09:59:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=255560439 f=0xf0000009 rw=0x2
May 27 09:59:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=177441847 f=0xf0000009 rw=0x2
May 27 09:59:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=260280439 f=0xf0000009 rw=0x2
May 27 09:59:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=260279231 f=0xf0000009 rw=0x2
May 27 09:59:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=260280327 f=0xf0000009 rw=0x2
May 27 10:00:12 linux-7vph kernel: XXX sda: failing readahead bio, sec=187403095 f=0xf0000009 rw=0x2
May 27 10:00:23 linux-7vph dbus-daemon: Rejected send message, 3 matched rules; type="method_call", sender=":1.20" (uid=1000 pid=4896 comm="nm-online ") interface="org.freedesktop.NetworkManager" member="state" error name="(unset)" requested_reply=0 destination="org.freedesktop.NetworkManager" (uid=0 pid=2500 comm="/usr/sbin/NetworkManager "))
May 27 10:02:35 linux-7vph kernel: XXX sda: failing readahead bio, sec=200772087 f=0xf0000009 rw=0x2
May 27 10:02:35 linux-7vph kernel: XXX sda: failing readahead bio, sec=204179583 f=0xf0000009 rw=0x2
May 27 10:02:35 linux-7vph kernel: XXX sda: failing readahead bio, sec=195792047 f=0xf0000009 rw=0x2
May 27 10:02:35 linux-7vph kernel: XXX sda: failing readahead bio, sec=195005823 f=0xf0000009 rw=0x2
May 27 10:02:39 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.157, on dev eth0
May 27 10:02:39 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:16:d3:38:7d:af:08:00
May 27 10:02:42 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.157, on dev eth0
May 27 10:02:42 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:16:d3:38:7d:af:08:00
May 27 10:02:43 linux-7vph kernel: XXX sda: failing readahead bio, sec=195268031 f=0xf0000009 rw=0x2
May 27 10:02:44 linux-7vph kernel: XXX sda: failing readahead bio, sec=195267847 f=0xf0000009 rw=0x2
May 27 10:03:11 linux-7vph kernel: XXX sda: failing readahead bio, sec=195791703 f=0xf0000009 rw=0x2
May 27 10:03:11 linux-7vph kernel: XXX sda: failing readahead bio, sec=196840855 f=0xf0000009 rw=0x2
May 27 10:03:11 linux-7vph kernel: XXX sda: failing readahead bio, sec=200771823 f=0xf0000009 rw=0x2
May 27 10:03:41 linux-7vph kernel: XXX sda: failing readahead bio, sec=167750847 f=0xf0000009 rw=0x2
May 27 10:03:42 linux-7vph kernel: XXX sda: failing readahead bio, sec=253728511 f=0xf0000009 rw=0x2
May 27 10:03:43 linux-7vph kernel: XXX sda: failing readahead bio, sec=194742879 f=0xf0000009 rw=0x2
May 27 10:03:52 linux-7vph kernel: XXX sda: failing readahead bio, sec=252679087 f=0xf0000009 rw=0x2
May 27 10:03:52 linux-7vph kernel: XXX sda: failing readahead bio, sec=245074183 f=0xf0000009 rw=0x2
May 27 10:03:52 linux-7vph kernel: XXX sda: failing readahead bio, sec=252938623 f=0xf0000009 rw=0x2
May 27 10:03:57 linux-7vph kernel: XXX sda: failing readahead bio, sec=261327807 f=0xf0000009 rw=0x2
May 27 10:04:04 linux-7vph kernel: martian source 255.255.255.255 from 10.59.200.118, on dev eth0
May 27 10:04:04 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:21:70:a5:83:6a:08:00
May 27 10:04:49 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.15, on dev eth0
May 27 10:04:49 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:c5:11:93:fe:08:00
May 27 10:04:52 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.15, on dev eth0
May 27 10:04:52 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:15:c5:11:93:fe:08:00
May 27 10:04:53 linux-7vph kernel: XXX sda: failing readahead bio, sec=167482119 f=0xf0000009 rw=0x2
May 27 10:04:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=171674951 f=0xf0000009 rw=0x2
May 27 10:04:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=175868287 f=0xf0000009 rw=0x2
May 27 10:04:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=174295511 f=0xf0000009 rw=0x2
May 27 10:04:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=171149511 f=0xf0000009 rw=0x2
May 27 10:04:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=173771503 f=0xf0000009 rw=0x2
May 27 10:04:54 linux-7vph kernel: XXX sda: failing readahead bio, sec=174295879 f=0xf0000009 rw=0x2
May 27 10:05:00 linux-7vph kernel: martian source 255.255.255.255 from 10.59.201.170, on dev eth0
May 27 10:05:00 linux-7vph kernel: ll header: ff:ff:ff:ff:ff:ff:00:1d:72:4b:ab:e3:08:00
May 27 10:05:06 linux-7vph shutdown[15848]: shutting down for system reboot
#36 lore raw
Quoted 3 lines · Niel Lambrechts
Quoted 2 lines · Tejun Heo

The above is the offending failure and all three failfast bits are set. This corresponds to the following ATA exception.

<snip>

Quoted 8 lines
Quoted 7 lines

Can you please try the attached patch? It takes suspend/resume cycle out of the equation and simply induces artificial failure to readahead requests. It's currently set to fail every 40th readahead. Feel free to adjust the frequency as you see fit. catting files into /dev/null would trigger readahead to kick in. Can you reproduce filesystem failure with this alone?

Quoted 7 lines

No corruption, I tried cat on an entire directory of mp3s, then also started X, but there the messages debug output got fairly drastic so I didn't care to continue for an entire day.

It did trigger the "XXX ...failing readahead" message nearly 300 times, and I also did a s2disk and resume cycle in there - so I hope this is enough for us to conclude that it is not the cause.

Hi Tejun,

Did you perhaps have any time to look into my feedback around the readahead patch?

Quoted 1 line

From my side, I tried on Saturday to bisect this problem again, doing

5-8 hibernates per each bisect from 2.6.28. I stopped at 2.6.30-rc2 due time (or fatigue), and did not manage to replicate the problem at all which is strange since I was playing audio, doing finds and even doing an entire dd of the root partition. I saved the bisect logs so perhaps I can continue to see if the problem becomes more prevalent in later versions - the first time it ever happened to me was somewhere in 2.6.29 originally.

The other interesting thing was to see that "hard resetting link" messages seem to first start appearing at v2.6.29-rc7 or perhaps rc8. Is it worth trying to track down the commit that lead to this?

Do you have any other debug patches to try, or should I try to delve deeper into finding commits that can be reverted? I'm running out of ideas, I even tried to find later firmware for my drive, but I seem to be on the latest level.

Thanks for the help so far!

Niel

#37 lore raw

Hello, Niel.

Quoted 2 lines · Niel Lambrechts

Did you perhaps have any time to look into my feedback around the readahead patch?

Yeah, I've been thinking about it and am a bit out of ideas, so no immediate follow-up.

Quoted 9 lines
Quoted 1 line

From my side, I tried on Saturday to bisect this problem again, doing

5-8 hibernates per each bisect from 2.6.28. I stopped at 2.6.30-rc2 due time (or fatigue), and did not manage to replicate the problem at all which is strange since I was playing audio, doing finds and even doing an entire dd of the root partition. I saved the bisect logs so perhaps I can continue to see if the problem becomes more prevalent in later versions - the first time it ever happened to me was somewhere in 2.6.29 originally.

Quoted 8 lines

The other interesting thing was to see that "hard resetting link" messages seem to first start appearing at v2.6.29-rc7 or perhaps rc8. Is it worth trying to track down the commit that lead to this?

Do you have any other debug patches to try, or should I try to delve deeper into finding commits that can be reverted? I'm running out of ideas, I even tried to find later firmware for my drive, but I seem to be on the latest level.

Given the non-deterministic nature of the failure, I think bisection would be quite difficult. I think the best way to diagnose the problem is to track down the owner of the failed request. It has FAILFAST set and yet the issuer fails to deal with an error condition which is expected when FAILFAST is set. I'll prep a patch to track request / bio failure.

Thanks.

Signature · 1 line
tejun
#38 lore raw

Hello,

Can you please do the followings?

1. Apply the attached patch, build & boot

2. Trigger the problem and record dmesg

3. On failed IO, the kernel will print the address of bi_endio. Run "nm -n" on the vmlinux in the kernel build root and look up which function it is and post the dmesg and function name.

Thanks.

Signature · 1 line
tejun
block/blk-core.c +6 -1
index c89883b..6c03f54 100644
@@ -163,8 +163,13 @@ static void req_bio_endio(struct request *rq, struct bio *bio,
 		if (bio_integrity(bio))
 			bio_integrity_advance(bio, nbytes);
 
-		if (bio->bi_size == 0)
+		if (bio->bi_size == 0) {
+			if (error)
+				printk("XXX %s: failing bio %p with %d\n",
+				       rq->rq_disk ? rq->rq_disk->disk_name : "?",
+				       bio, error);
 			bio_endio(bio, error);
+		}
 	} else {
 
 		/*
fs/bio.c +13 -2
index 9871164..c26364a 100644
@@ -1391,13 +1391,24 @@ void bio_check_pages_dirty(struct bio *bio)
  **/
 void bio_endio(struct bio *bio, int error)
 {
+	char name[BDEVNAME_SIZE] = "?";
+
+	if (bio->bi_bdev)
+		bdevname(bio->bi_bdev, name);
+
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
-	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
+	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags)) {
+		printk("XXX %s: !uptodate on bio %p\n", name, bio);
 		error = -EIO;
+	}
 
-	if (bio->bi_end_io)
+	if (bio->bi_end_io) {
+		if (error)
+			printk("XXX %s: bio=%p error=%d bi_end_io=%p\n",
+			       name, bio, error, bio->bi_end_io);
 		bio->bi_end_io(bio, error);
+	}
 }
 
 void bio_pair_release(struct bio_pair *bp)
#39 lore raw

Morning Tejun,

Quoted 6 lines · Tejun Heo

Hello,

Can you please do the followings?

1. Apply the attached patch, build & boot

I chose 2.6.30-rc7...

Quoted 2 lines

2. Trigger the problem and record dmesg

It took 3 days and quite a few hibernate attempts ... :-)

Quoted 3 lines

3. On failed IO, the kernel will print the address of bi_endio. Run "nm -n" on the vmlinux in the kernel build root and look up which function it is and post the dmesg and function name.

I did not have that specific vmlinux.o file any more, but /boot/System.map-2.6.30-rc7-pae shows:
c01a49fd t end_bio_bh_io_sync

Hope this is sufficient to help you. Sorry if this is silly - being so inexperienced with the kernel - but I wondered if or why a dump_stack() in that debug patch would not be helpful?

Regards,
Niel

Jun  5 09:26:12 linux-7vph -- MARK --
Jun  5 09:26:12 linux-7vph syslog-ng[24615]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=3777', processed='center(received)=3142', processed='destination(newsnotice)=0', processed='destination(acpid)=13', processed='destination(firewall)=18', processed='destination(null)=208', processed='destination(mail)=0', processed='destination(mailinfo)=0', processed='destination(console)=200', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=1909', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=994', processed='destination(mailerr)=0', processed='destination(xconsole)=200', processed='destination(warn)=235', processed='source(src)=3142'
Jun  5 09:26:11 linux-7vph kernel: Syncing filesystems ... done.
Jun  5 09:26:12 linux-7vph kernel: Freezing user space processes ... (elapsed 0.00 seconds) done.
Jun  5 09:26:12 linux-7vph kernel: Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
Jun  5 09:26:12 linux-7vph kernel: PM: Shrinking memory... done (149507 pages freed)
Jun  5 09:26:12 linux-7vph kernel: PM: Freed 598028 kbytes in 6.49 seconds (92.14 MB/s)
Jun  5 09:26:12 linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
Jun  5 09:26:12 linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
Jun  5 09:26:12 linux-7vph kernel: ACPI handle has no context!
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
Jun  5 09:26:12 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
Jun  5 09:26:13 linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
Jun  5 09:26:13 linux-7vph kernel: PM: Saving platform NVS memory
Jun  5 09:26:13 linux-7vph kernel: Disabling non-boot CPUs ...
Jun  5 09:26:13 linux-7vph kernel: CPU 1 is now offline
Jun  5 09:26:13 linux-7vph kernel: SMP alternatives: switching to UP code
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: CPU1 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: CPU1 is down
Jun  5 09:26:13 linux-7vph kernel: Extended CMOS year: 2000
Jun  5 09:26:13 linux-7vph kernel: PM: Creating hibernation image: 
Jun  5 09:26:13 linux-7vph kernel: PM: Need to copy 223773 pages
Jun  5 09:26:13 linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jun  5 09:26:13 linux-7vph kernel: Intel machine check architecture supported.
Jun  5 09:26:13 linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
Jun  5 09:26:13 linux-7vph kernel: PM: Restoring platform NVS memory
Jun  5 09:26:13 linux-7vph kernel: Extended CMOS year: 2000
Jun  5 09:26:13 linux-7vph kernel: Enabling non-boot CPUs ...
Jun  5 09:26:13 linux-7vph kernel: SMP alternatives: switching to SMP code
Jun  5 09:26:13 linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
Jun  5 09:26:13 linux-7vph kernel: Initializing CPU#1
Jun  5 09:26:13 linux-7vph kernel: Calibrating delay using timer specific routine.. 5134.67 BogoMIPS (lpj=10269343)
Jun  5 09:26:13 linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jun  5 09:26:13 linux-7vph kernel: CPU: L2 cache: 6144K
Jun  5 09:26:13 linux-7vph kernel: CPU: Physical Processor ID: 0
Jun  5 09:26:13 linux-7vph kernel: CPU: Processor Core ID: 1
Jun  5 09:26:13 linux-7vph kernel: Intel machine check architecture supported.
Jun  5 09:26:13 linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
Jun  5 09:26:13 linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
Jun  5 09:26:13 linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: Switched to high resolution mode on CPU 1
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching sched-domain:
Jun  5 09:26:13 linux-7vph kernel:  domain 0: span 0-1 level MC
Jun  5 09:26:13 linux-7vph kernel:   groups: 0 1
Jun  5 09:26:13 linux-7vph kernel:   domain 1: span 0-1 level CPU
Jun  5 09:26:13 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
Jun  5 09:26:13 linux-7vph kernel: CPU1 attaching sched-domain:
Jun  5 09:26:13 linux-7vph kernel:  domain 0: span 0-1 level MC
Jun  5 09:26:13 linux-7vph kernel:   groups: 1 0
Jun  5 09:26:13 linux-7vph kernel:   domain 1: span 0-1 level CPU
Jun  5 09:26:13 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
Jun  5 09:26:13 linux-7vph kernel: microcode: failed to init CPU1
Jun  5 09:26:13 linux-7vph kernel: CPU1 is up
Jun  5 09:26:13 linux-7vph kernel: ACPI: Waking up from system sleep state S4
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900007, writing 0x900403)
Jun  5 09:26:13 linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
Jun  5 09:26:13 linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
Jun  5 09:26:13 linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
Jun  5 09:26:13 linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
Jun  5 09:26:13 linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
Jun  5 09:26:13 linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100506)
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.0: PME# disabled
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.1: PME# disabled
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:03.0: PME# disabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Jun  5 09:26:14 linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jun  5 09:26:14 linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.2: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.3: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.4: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.5: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
Jun  5 09:26:14 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun  5 09:26:14 linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:14 linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
Jun  5 09:26:14 linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
Jun  5 09:26:14 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:14 linux-7vph kernel: ata1: EH complete
Jun  5 09:26:14 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900407, writing 0x900403)
Jun  5 09:26:14 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: configured for UDMA/133
Jun  5 09:26:14 linux-7vph kernel: Restarting tasks ... <3>ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
Jun  5 09:26:14 linux-7vph kernel: ata2: irq_stat 0x40000001
Jun  5 09:26:14 linux-7vph kernel: ata1.00: exception Emask 0x10 SAct 0x1ff SErr 0x50000 action 0xe frozen
Jun  5 09:26:14 linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed
Jun  5 09:26:14 linux-7vph kernel: ata1: SError: { PHYRdyChg CommWake }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:00:5f:06:44/00:00:0c:00:00/40 tag 0 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/58:08:6f:06:44/00:00:0c:00:00/40 tag 1 ncq 45056 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:10:d7:06:44/00:00:0c:00:00/40 tag 2 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:18:07:07:44/00:00:0c:00:00/40 tag 3 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/18:20:17:07:44/00:00:0c:00:00/40 tag 4 ncq 12288 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:28:67:88:70/00:00:0e:00:00/40 tag 5 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:30:17:70:14/00:00:0d:00:00/40 tag 6 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/d8:38:5f:89:6f/00:00:0e:00:00/40 tag 7 ncq 110592 in
Jun  5 09:26:15 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:15 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:15 linux-7vph kernel: ata1.00: cmd 60/28:40:3f:8a:6f/00:00:0e:00:00/40 tag 8 ncq 20480 in
Jun  5 09:26:15 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:15 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:15 linux-7vph kernel: ata1: hard resetting link
Jun  5 09:26:15 linux-7vph kernel: ata2.00: configured for UDMA/133
Jun  5 09:26:15 linux-7vph kernel: ata2: EH complete
Jun  5 09:26:15 linux-7vph kernel: done.
Jun  5 09:26:15 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:16 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:16 linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 09:26:16 linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
Jun  5 09:26:16 linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
Jun  5 09:26:16 linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jun  5 09:26:16 linux-7vph kernel:         0e 6f 8a 3f 
Jun  5 09:26:16 linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
Jun  5 09:26:16 linux-7vph kernel: end_request: I/O error, dev sda, sector 242190687
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio e1223140 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=e1223140 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa0c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa0c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa140 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa140 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa1c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa1c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa240 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa240 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa2c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa2c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa340 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa340 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa3c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa3c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa440 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa440 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa4c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa4c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa540 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa540 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa5c0 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa5c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa640 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa640 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa6c0 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa6c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa740 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa740 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa7c0 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa7c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa840 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa840 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caa8c0 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa8c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caa940 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa940 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caa9c0 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa9c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caaa40 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caaa40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caaac0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caaac0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caab40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caab40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caabc0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caabc0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caac40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caac40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caacc0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caacc0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caad40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caad40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 09:26:22 linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
Jun  5 09:26:22 linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
Jun  5 09:26:22 linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jun  5 09:26:22 linux-7vph kernel:         0e 6f 8a 3f 
Jun  5 09:26:22 linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
Jun  5 09:26:22 linux-7vph kernel: end_request: I/O error, dev sda, sector 242190911
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caadc0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caadc0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caae40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caae40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caaec0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caaec0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caaf40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caaf40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio e12230c0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=e12230c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: ata1: EH complete
Jun  5 09:26:22 linux-7vph kernel: EXT4-fs error (device sda6): __ext4_get_inode_loc: unable to read inode block - inode=2346994, block=9404448
Jun  5 09:26:22 linux-7vph kernel: EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure
#40 lore raw
Quoted 17 lines · Niel Lambrechts

Morning Tejun,

Quoted 6 lines · Tejun Heo

Hello,

Can you please do the followings?

1. Apply the attached patch, build & boot

I chose 2.6.30-rc7...

Quoted 2 lines

2. Trigger the problem and record dmesg

It took 3 days and quite a few hibernate attempts ... :-)

Quoted 3 lines

3. On failed IO, the kernel will print the address of bi_endio. Run "nm -n" on the vmlinux in the kernel build root and look up which function it is and post the dmesg and function name.

Quoted 7 lines

I did not have that specific vmlinux.o file any more, but /boot/System.map-2.6.30-rc7-pae shows:
c01a49fd t end_bio_bh_io_sync

Hope this is sufficient to help you. Sorry if this is silly - being so inexperienced with the kernel - but I wondered if or why a dump_stack() in that debug patch would not be helpful?

I guess dump_stack would be very helpful :-).

Quoted 275 lines

Regards,
Niel

Jun  5 09:26:12 linux-7vph -- MARK --
Jun  5 09:26:12 linux-7vph syslog-ng[24615]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=3777', processed='center(received)=3142', processed='destination(newsnotice)=0', processed='destination(acpid)=13', processed='destination(firewall)=18', processed='destination(null)=208', processed='destination(mail)=0', processed='destination(mailinfo)=0', processed='destination(console)=200', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=1909', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=994', processed='destination(mailerr)=0', processed='destination(xconsole)=200', processed='destination(warn)=235', processed='source(src)=3142'
Jun  5 09:26:11 linux-7vph kernel: Syncing filesystems ... done.
Jun  5 09:26:12 linux-7vph kernel: Freezing user space processes ... (elapsed 0.00 seconds) done.
Jun  5 09:26:12 linux-7vph kernel: Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
Jun  5 09:26:12 linux-7vph kernel: PM: Shrinking memory... done (149507 pages freed)
Jun  5 09:26:12 linux-7vph kernel: PM: Freed 598028 kbytes in 6.49 seconds (92.14 MB/s)
Jun  5 09:26:12 linux-7vph kernel: Suspending console(s) (use no_console_suspend to debug)
Jun  5 09:26:12 linux-7vph kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
Jun  5 09:26:12 linux-7vph kernel: ACPI handle has no context!
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D disabled
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A disabled
Jun  5 09:26:12 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B disabled
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D disabled
Jun  5 09:26:12 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B disabled
Jun  5 09:26:12 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A disabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A disabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: PME# enabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability enabled by ACPI
Jun  5 09:26:13 linux-7vph kernel: ACPI: Preparing to enter system sleep state S4
Jun  5 09:26:13 linux-7vph kernel: PM: Saving platform NVS memory
Jun  5 09:26:13 linux-7vph kernel: Disabling non-boot CPUs ...
Jun  5 09:26:13 linux-7vph kernel: CPU 1 is now offline
Jun  5 09:26:13 linux-7vph kernel: SMP alternatives: switching to UP code
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: CPU1 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: CPU1 is down
Jun  5 09:26:13 linux-7vph kernel: Extended CMOS year: 2000
Jun  5 09:26:13 linux-7vph kernel: PM: Creating hibernation image: 
Jun  5 09:26:13 linux-7vph kernel: PM: Need to copy 223773 pages
Jun  5 09:26:13 linux-7vph kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Jun  5 09:26:13 linux-7vph kernel: Intel machine check architecture supported.
Jun  5 09:26:13 linux-7vph kernel: Intel machine check reporting enabled on CPU#0.
Jun  5 09:26:13 linux-7vph kernel: PM: Restoring platform NVS memory
Jun  5 09:26:13 linux-7vph kernel: Extended CMOS year: 2000
Jun  5 09:26:13 linux-7vph kernel: Enabling non-boot CPUs ...
Jun  5 09:26:13 linux-7vph kernel: SMP alternatives: switching to SMP code
Jun  5 09:26:13 linux-7vph kernel: Booting processor 1 APIC 0x1 ip 0x6000
Jun  5 09:26:13 linux-7vph kernel: Initializing CPU#1
Jun  5 09:26:13 linux-7vph kernel: Calibrating delay using timer specific routine.. 5134.67 BogoMIPS (lpj=10269343)
Jun  5 09:26:13 linux-7vph kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jun  5 09:26:13 linux-7vph kernel: CPU: L2 cache: 6144K
Jun  5 09:26:13 linux-7vph kernel: CPU: Physical Processor ID: 0
Jun  5 09:26:13 linux-7vph kernel: CPU: Processor Core ID: 1
Jun  5 09:26:13 linux-7vph kernel: Intel machine check architecture supported.
Jun  5 09:26:13 linux-7vph kernel: Intel machine check reporting enabled on CPU#1.
Jun  5 09:26:13 linux-7vph kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
Jun  5 09:26:13 linux-7vph kernel: CPU1: Intel(R) Core(TM)2 Duo CPU     T9400  @ 2.53GHz stepping 06
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching NULL sched-domain.
Jun  5 09:26:13 linux-7vph kernel: Switched to high resolution mode on CPU 1
Jun  5 09:26:13 linux-7vph kernel: CPU0 attaching sched-domain:
Jun  5 09:26:13 linux-7vph kernel:  domain 0: span 0-1 level MC
Jun  5 09:26:13 linux-7vph kernel:   groups: 0 1
Jun  5 09:26:13 linux-7vph kernel:   domain 1: span 0-1 level CPU
Jun  5 09:26:13 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
Jun  5 09:26:13 linux-7vph kernel: CPU1 attaching sched-domain:
Jun  5 09:26:13 linux-7vph kernel:  domain 0: span 0-1 level MC
Jun  5 09:26:13 linux-7vph kernel:   groups: 1 0
Jun  5 09:26:13 linux-7vph kernel:   domain 1: span 0-1 level CPU
Jun  5 09:26:13 linux-7vph kernel:    groups: 0-1 (__cpu_power = 2048)
Jun  5 09:26:13 linux-7vph kernel: microcode: failed to init CPU1
Jun  5 09:26:13 linux-7vph kernel: CPU1 is up
Jun  5 09:26:13 linux-7vph kernel: ACPI: Waking up from system sleep state S4
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900007, writing 0x900403)
Jun  5 09:26:13 linux-7vph kernel: uhci_hcd 0000:00:1a.0: power state changed by ACPI to D0
Jun  5 09:26:13 linux-7vph kernel: uhci_hcd 0000:00:1a.2: power state changed by ACPI to D0
Jun  5 09:26:13 linux-7vph kernel: HDA Intel 0000:00:1b.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100102)
Jun  5 09:26:13 linux-7vph kernel: uhci_hcd 0000:00:1d.0: power state changed by ACPI to D0
Jun  5 09:26:13 linux-7vph kernel: ahci 0000:00:1f.2: restoring config space at offset 0x1 (was 0x2b00403, writing 0x2b00407)
Jun  5 09:26:13 linux-7vph kernel: iwlagn 0000:03:00.0: restoring config space at offset 0x1 (was 0x100106, writing 0x100506)
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.0: PME# disabled
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:02.1: PME# disabled
Jun  5 09:26:13 linux-7vph kernel: pci 0000:00:03.0: PME# disabled
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: pci_enable_pcie_error_reporting failed 0xfffffffb
Jun  5 09:26:13 linux-7vph kernel: e1000e 0000:00:19.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: wake-up capability disabled by ACPI
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.1: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.2: PCI INT C -> GSI 22 (level, low) -> IRQ 22
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1a.2: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PCI INT D -> GSI 23 (level, low) -> IRQ 23
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1a.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: HDA Intel 0000:00:1b.0: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Jun  5 09:26:14 linux-7vph kernel: HDA Intel 0000:00:1b.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.1: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
Jun  5 09:26:14 linux-7vph kernel: uhci_hcd 0000:00:1d.2: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ehci_hcd 0000:00:1d.7: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:00:1e.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ahci 0000:00:1f.2: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Jun  5 09:26:14 linux-7vph kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[17]  MMIO=[f4801000-f48017ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.2: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.3: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.4: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: pci 0000:15:00.5: PME# disabled
Jun  5 09:26:14 linux-7vph kernel: sd 0:0:0:0: [sda] Starting disk
Jun  5 09:26:14 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun  5 09:26:14 linux-7vph kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:14 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:14 linux-7vph kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
Jun  5 09:26:14 linux-7vph kernel: ata1: irq_stat 0x00400040, connection status changed
Jun  5 09:26:14 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:14 linux-7vph kernel: ata1: EH complete
Jun  5 09:26:14 linux-7vph kernel: pci 0000:00:02.0: restoring config space at offset 0x1 (was 0x900407, writing 0x900403)
Jun  5 09:26:14 linux-7vph kernel: pci 0000:00:02.0: setting latency timer to 64
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:1f:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: ACPI cmd e3/00:02:00:00:00:a0 succeeded
Jun  5 09:26:14 linux-7vph kernel: ata2.00: configured for UDMA/133
Jun  5 09:26:14 linux-7vph kernel: Restarting tasks ... <3>ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x9 t4
Jun  5 09:26:14 linux-7vph kernel: ata2: irq_stat 0x40000001
Jun  5 09:26:14 linux-7vph kernel: ata1.00: exception Emask 0x10 SAct 0x1ff SErr 0x50000 action 0xe frozen
Jun  5 09:26:14 linux-7vph kernel: ata1.00: irq_stat 0x00400008, PHY RDY changed
Jun  5 09:26:14 linux-7vph kernel: ata1: SError: { PHYRdyChg CommWake }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:00:5f:06:44/00:00:0c:00:00/40 tag 0 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/58:08:6f:06:44/00:00:0c:00:00/40 tag 1 ncq 45056 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:10:d7:06:44/00:00:0c:00:00/40 tag 2 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:18:07:07:44/00:00:0c:00:00/40 tag 3 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/18:20:17:07:44/00:00:0c:00:00/40 tag 4 ncq 12288 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:28:67:88:70/00:00:0e:00:00/40 tag 5 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/08:30:17:70:14/00:00:0d:00:00/40 tag 6 ncq 4096 in
Jun  5 09:26:14 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:14 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:14 linux-7vph kernel: ata1.00: cmd 60/d8:38:5f:89:6f/00:00:0e:00:00/40 tag 7 ncq 110592 in
Jun  5 09:26:15 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:15 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:15 linux-7vph kernel: ata1.00: cmd 60/28:40:3f:8a:6f/00:00:0e:00:00/40 tag 8 ncq 20480 in
Jun  5 09:26:15 linux-7vph kernel:          res 50/00:28:3f:8a:6f/00:00:0e:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 09:26:15 linux-7vph kernel: ata1.00: status: { DRDY }
Jun  5 09:26:15 linux-7vph kernel: ata1: hard resetting link
Jun  5 09:26:15 linux-7vph kernel: ata2.00: configured for UDMA/133
Jun  5 09:26:15 linux-7vph kernel: ata2: EH complete
Jun  5 09:26:15 linux-7vph kernel: done.
Jun  5 09:26:15 linux-7vph kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/02:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/5f:00:00:00:00:a0 succeeded
Jun  5 09:26:16 linux-7vph kernel: ata1.00: ACPI cmd ef/10:03:00:00:00:a0 filtered out
Jun  5 09:26:16 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:16 linux-7vph kernel: ata1.00: configured for UDMA/133
Jun  5 09:26:16 linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 09:26:16 linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
Jun  5 09:26:16 linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
Jun  5 09:26:16 linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jun  5 09:26:16 linux-7vph kernel:         0e 6f 8a 3f 
Jun  5 09:26:16 linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
Jun  5 09:26:16 linux-7vph kernel: end_request: I/O error, dev sda, sector 242190687
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio e1223140 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=e1223140 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa0c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa0c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa140 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa140 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa1c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa1c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa240 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa240 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa2c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa2c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa340 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa340 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa3c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa3c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa440 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa440 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa4c0 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa4c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa540 with -5
Jun  5 09:26:16 linux-7vph kernel: XXX sda: bio=f5caa540 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:16 linux-7vph kernel: XXX sda: failing bio f5caa5c0 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa5c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa640 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa640 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa6c0 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa6c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa740 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa740 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa7c0 with -5
Jun  5 09:26:19 linux-7vph kernel: XXX sda: bio=f5caa7c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:19 linux-7vph kernel: XXX sda: failing bio f5caa840 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa840 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caa8c0 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa8c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caa940 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa940 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caa9c0 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caa9c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caaa40 with -5
Jun  5 09:26:21 linux-7vph kernel: XXX sda: bio=f5caaa40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:21 linux-7vph kernel: XXX sda: failing bio f5caaac0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caaac0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caab40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caab40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caabc0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caabc0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caac40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caac40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caacc0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caacc0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caad40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caad40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 09:26:22 linux-7vph kernel: sd 0:0:0:0: [sda] Sense Key : Aborted Command [current] [descriptor]
Jun  5 09:26:22 linux-7vph kernel: Descriptor sense data with sense descriptors (in hex):
Jun  5 09:26:22 linux-7vph kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
Jun  5 09:26:22 linux-7vph kernel:         0e 6f 8a 3f 
Jun  5 09:26:22 linux-7vph kernel: sd 0:0:0:0: [sda] Add. Sense: No additional sense information
Jun  5 09:26:22 linux-7vph kernel: end_request: I/O error, dev sda, sector 242190911
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caadc0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caadc0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caae40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caae40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caaec0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caaec0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio f5caaf40 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=f5caaf40 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: XXX sda: failing bio e12230c0 with -5
Jun  5 09:26:22 linux-7vph kernel: XXX sda: bio=e12230c0 error=-5 bi_end_io=c01a49fd
Jun  5 09:26:22 linux-7vph kernel: ata1: EH complete
Jun  5 09:26:22 linux-7vph kernel: EXT4-fs error (device sda6): __ext4_get_inode_loc: unable to read inode block - inode=2346994, block=9404448
Jun  5 09:26:22 linux-7vph kernel: EXT4-fs error (device sda6) in ext4_reserve_inode_write: IO failure
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#41 lore raw
In reply to Niel Lambrechts

Sorry about the long delay.

Quoted 12 lines · Niel Lambrechts

Morning Tejun,

Quoted 6 lines · Tejun Heo

Hello,

Can you please do the followings?

1. Apply the attached patch, build & boot

I chose 2.6.30-rc7...

Quoted 2 lines

2. Trigger the problem and record dmesg

Quoted 8 lines

It took 3 days and quite a few hibernate attempts ... :-)

Quoted 3 lines

3. On failed IO, the kernel will print the address of bi_endio. Run "nm -n" on the vmlinux in the kernel build root and look up which function it is and post the dmesg and function name.

I did not have that specific vmlinux.o file any more, but /boot/System.map-2.6.30-rc7-pae shows:
c01a49fd t end_bio_bh_io_sync

So, it's coming from submit_bh()

Quoted 3 lines

Hope this is sufficient to help you. Sorry if this is silly - being so inexperienced with the kernel - but I wondered if or why a dump_stack() in that debug patch would not be helpful?

The result is perfectly good and yeah dump_stack() on the issue path would help but the problem is that block IO requests are processed asynchronously so by the time we find out which request fail, the requester stack is long gone. We can either record the stack trace with each request or trace it back one step at a time by chasing down the completion callbacks. The first requires more coding, so... :-)

Looks like the request gotta be coming from __breadahead(). The only place this is used in ext4 is in __ext4_get_inode_loc(). Ah.. it also contains the matching error message. I still don't see how the READA buffer reads can affect the synchronous path. They're doing proper exclusion via buffer lock. Maybe they're getting merged? Yeap, looks like block code is merging READAs and regular READs.

Can you please try the attached patch and reproduce the problem and report the kernel log? Hopefully, this will be the last debug run.

Thanks.

Signature · 1 line
tejun
block/blk-core.c +6 -1
index b06cf5c..c8b3a6f 100644
@@ -155,8 +155,13 @@ static void req_bio_endio(struct request *rq, struct bio *bio,
 		if (bio_integrity(bio))
 			bio_integrity_advance(bio, nbytes);
 
-		if (bio->bi_size == 0)
+		if (bio->bi_size == 0) {
+			if (error)
+				printk("XXX %s: failing bio %p bi_rw=0x%lx with %d\n",
+				       rq->rq_disk ? rq->rq_disk->disk_name : "?",
+				       bio, bio->bi_rw, error);
 			bio_endio(bio, error);
+		}
 	} else {
 
 		/*
fs/bio.c +13 -2
index 24c9140..007edb9 100644
@@ -1390,13 +1390,24 @@ void bio_check_pages_dirty(struct bio *bio)
  **/
 void bio_endio(struct bio *bio, int error)
 {
+	char name[BDEVNAME_SIZE] = "?";
+
+	if (bio->bi_bdev)
+		bdevname(bio->bi_bdev, name);
+
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
-	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
+	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags)) {
+		printk("XXX %s: !uptodate on bio %p\n", name, bio);
 		error = -EIO;
+	}
 
-	if (bio->bi_end_io)
+	if (bio->bi_end_io) {
+		if (error)
+			printk("XXX %s: bio=%p error=%d bi_end_io=%p\n",
+			       name, bio, error, bio->bi_end_io);
 		bio->bi_end_io(bio, error);
+	}
 }
 
 void bio_pair_release(struct bio_pair *bp)
#42 lore raw
Quoted 16 lines · Tejun Heo

Sorry about the long delay.

The result is perfectly good and yeah dump_stack() on the issue path would help but the problem is that block IO requests are processed asynchronously so by the time we find out which request fail, the requester stack is long gone. We can either record the stack trace with each request or trace it back one step at a time by chasing down the completion callbacks. The first requires more coding, so... :-)

Looks like the request gotta be coming from __breadahead(). The only place this is used in ext4 is in __ext4_get_inode_loc(). Ah.. it also contains the matching error message. I still don't see how the READA buffer reads can affect the synchronous path. They're doing proper exclusion via buffer lock. Maybe they're getting merged? Yeap, looks like block code is merging READAs and regular READs.

Quoted 3 lines

Can you please try the attached patch and reproduce the problem and report the kernel log? Hopefully, this will be the last debug run.

Hi Tejun,

I've recently switched my root partition from OpenSUSE 11.1 to Fedora 11 and since then I've not again seen the issue. I'm still using vanilla 2.6.30 generated with the same .config and EXT4 as before, so I have no idea why I cannot reproduce the issue. I still use hibernate + sleep frequently, and I just checked - I have 5 days uptime with a mount count of 20 and the file-system is still clean.

The one big difference is that my original partition was a EXT2 -> EXT3 -> EXT4 upgrade job over a long period of time, and some of the EXT4 parameters now used by Fedora 11 on the reformatted root partition are different from what I had then. Here is a summary of the differences in case it matters at all:

Current settings:

Default mount options:    user_xattr acl
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4

Previous settings:

Default mount options:    (none)
Inodes per group:         8176
Inode blocks per group:   511
Default directory hash:   tea

If I do notice any such errors again I'll apply the debug patch and let you know, but it does seem as if the upgrade made this issue disappear...

Regards,
Niel

#43 lore raw

Hello, Niel.

Quoted 22 lines · Niel Lambrechts

I've recently switched my root partition from OpenSUSE 11.1 to Fedora 11 and since then I've not again seen the issue. I'm still using vanilla 2.6.30 generated with the same .config and EXT4 as before, so I have no idea why I cannot reproduce the issue. I still use hibernate + sleep frequently, and I just checked - I have 5 days uptime with a mount count of 20 and the file-system is still clean.

The one big difference is that my original partition was a EXT2 -> EXT3 -> EXT4 upgrade job over a long period of time, and some of the EXT4 parameters now used by Fedora 11 on the reformatted root partition are different from what I had then. Here is a summary of the differences in case it matters at all:

Current settings:

Default mount options:    user_xattr acl
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4
Quoted 8 lines

Previous settings:

Default mount options:    (none)
Inodes per group:         8176
Inode blocks per group:   511
Default directory hash:   tea

If I do notice any such errors again I'll apply the debug patch and let you know, but it does seem as if the upgrade made this issue disappear...

Eh... if my analysis is correct the bug will be dependent on some non-deterministic things including timing and layout of blocks of the file which the filesystem touchs during suspend/resume. I'm fairly sure the problem is still there. Oh well, it seems I'll have to construct a synthetic case to test it. Anyways, please let me know if you encounter the problem again. I'll keep you in loop with further patches.

Thanks.

Signature · 1 line
tejun
#44 lore raw
Quoted 17 lines · Tejun Heo

Hello, Niel.

Quoted 13 lines · Niel Lambrechts

I've recently switched my root partition from OpenSUSE 11.1 to Fedora 11 and since then I've not again seen the issue. I'm still using vanilla 2.6.30 generated with the same .config and EXT4 as before, so I have no idea why I cannot reproduce the issue. I still use hibernate + sleep frequently, and I just checked - I have 5 days uptime with a mount count of 20 and the file-system is still clean.

The one big difference is that my original partition was a EXT2 -> EXT3 -> EXT4 upgrade job over a long period of time, and some of the EXT4 parameters now used by Fedora 11 on the reformatted root partition are different from what I had then. Here is a summary of the differences in case it matters at all:

<snip>

Quoted 8 lines

Eh... if my analysis is correct the bug will be dependent on some non-deterministic things including timing and layout of blocks of the file which the filesystem touchs during suspend/resume. I'm fairly sure the problem is still there. Oh well, it seems I'll have to construct a synthetic case to test it. Anyways, please let me know if you encounter the problem again. I'll keep you in loop with further patches.

Understood, what I mean to imply is that I might (at least temporarily) be shielded from the bug because reformatting the root partition could possibly have removed some fragmentation and improved the initial disk layout. With the previous partition layout there would also have been a mixture of extents and non-extents based files since that system was upgraded from EXT2 to EXT3 to EXT4.

I'll try to boot from 2.6.29 again as well, and put it through some suspend/resume cycles just in case that made any difference.

Regards,
Niel

#45 lore raw

Am Freitag 26 Juni 2009 schrieb Niel Lambrechts:

Quoted 8 lines · Tejun Heo
Quoted 7 lines

Eh... if my analysis is correct the bug will be dependent on some non-deterministic things including timing and layout of blocks of the file which the filesystem touchs during suspend/resume. I'm fairly sure the problem is still there. Oh well, it seems I'll have to construct a synthetic case to test it. Anyways, please let me know if you encounter the problem again. I'll keep you in loop with further patches.

I am not quite sure if I am at the right place here, however, I get very similar problems with a totally different setup. What I do is the following:

I added a fourth disk to a software raid5 array and did setup the raid completely from scratch (the same disks have been running for about 1.5 years without any problems before, the difference is that previously one of the disks was setup as hot spare).

After the software raid was in sync, I started to copy my data back to the raid, and after a less than 5 minutes time, one of the disks failed (from /var/log/warn):

Sep 18 22:04:02 Bacchus kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
Sep 18 22:04:02 Bacchus kernel: ata3.00: irq_stat 0x00400000, PHY RDY changed
Sep 18 22:04:02 Bacchus kernel: ata3: SError: { PHYRdyChg }
Sep 18 22:04:02 Bacchus kernel: ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Sep 18 22:04:02 Bacchus kernel:          res 40/00:0c:3f:6c:d5/00:00:15:00:00/40 Emask 0x10 (ATA bus error)
Sep 18 22:04:02 Bacchus kernel: ata3.00: status: { DRDY }
Sep 18 22:04:02 Bacchus kernel: ata3: hard resetting link
Sep 18 22:04:07 Bacchus kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 18 22:04:12 Bacchus kernel: ata3.00: qc timeout (cmd 0xec)
Sep 18 22:04:12 Bacchus kernel: ata3.00: failed to IDENTIFY (I/O error, err_mask=0x5)
Sep 18 22:04:12 Bacchus kernel: ata3.00: revalidation failed (errno=-5)
Sep 18 22:04:12 Bacchus kernel: ata3: hard resetting link
Sep 18 22:04:12 Bacchus kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 18 22:04:12 Bacchus kernel: ata3.00: configured for UDMA/133
Sep 18 22:04:12 Bacchus kernel: ata3: EH complete
Sep 18 22:04:12 Bacchus kernel: end_request: I/O error, dev sdc, sector 1953519853
Sep 18 22:04:12 Bacchus kernel: md: super_written gets error=-5, uptodate=0
Sep 18 22:04:12 Bacchus kernel: raid5: Disk failure on sdc1, disabling device.
Sep 18 22:04:12 Bacchus kernel: raid5: Operation continuing on 3 devices.
Sep 18 22:04:13 Bacchus kernel: RAID5 conf printout:
Sep 18 22:04:13 Bacchus kernel:  --- rd:4 wd:3
Sep 18 22:04:13 Bacchus kernel:  disk 0, o:1, dev:sda1
Sep 18 22:04:13 Bacchus kernel:  disk 1, o:1, dev:sdb1
Sep 18 22:04:13 Bacchus kernel:  disk 2, o:0, dev:sdc1
Sep 18 22:04:13 Bacchus kernel:  disk 3, o:1, dev:sdd1
Sep 18 22:04:13 Bacchus kernel: RAID5 conf printout:
Sep 18 22:04:13 Bacchus kernel:  --- rd:4 wd:3
Sep 18 22:04:13 Bacchus kernel:  disk 0, o:1, dev:sda1
Sep 18 22:04:13 Bacchus kernel:  disk 1, o:1, dev:sdb1
Sep 18 22:04:13 Bacchus kernel:  disk 3, o:1, dev:sdd1

The filesystem on /dev/md0 was XFS and there was no suspend/resume applied.
The raid is built with four 1TB disks, attached to a Intel ICH7 Controller:
00:1f.2 SATA controller: Intel Corporation 82801GR/GH (ICH7 Family) SATA AHCI Controller (rev 01)

The system is a openSUSE11.1 with all patches applied. When I first encountered this problem I had the original openSUSE kernel (2.6.27.29) running, but I thought it was a good idea to go with a current upstream kernel instead. The difference was, that with the openSUSE kernel, I could copy several 100 GB before I lost the disk, while with the upstream kernel (2.6.31 from ftp.kernel.org) I lost the disk after less than 4 GB. The result however is the same.

smart does not tell me any problems with the disks, and I did not encounter any problems during the build of the raid5 which took about 6 hours on this machine. Thus I don't believe that there is a hardware problem.

Maybe you have an idea what I can do to provide more information.

Thanks in advance

Berthold

#46 lore raw

Hello, Berthold.

Quoted 12 lines · Berthold Gunreben

I am not quite sure if I am at the right place here, however, I get very similar problems with a totally different setup. What I do is the following:

I added a fourth disk to a software raid5 array and did setup the raid completely from scratch (the same disks have been running for about 1.5 years without any problems before, the difference is that previously one of the disks was setup as hot spare).

After the software raid was in sync, I started to copy my data back to the raid, and after a less than 5 minutes time, one of the disks failed (from /var/log/warn):

Quoted 6 lines
Sep 18 22:04:02 Bacchus kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
Sep 18 22:04:02 Bacchus kernel: ata3.00: irq_stat 0x00400000, PHY RDY changed
Sep 18 22:04:02 Bacchus kernel: ata3: SError: { PHYRdyChg }
Sep 18 22:04:02 Bacchus kernel: ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Sep 18 22:04:02 Bacchus kernel:          res 40/00:0c:3f:6c:d5/00:00:15:00:00/40 Emask 0x10 (ATA bus error)
Sep 18 22:04:02 Bacchus kernel: ata3.00: status: { DRDY }

The disk is most likely losing power briefly. After boot, run "smartctl -a" on the device and record the output. After triggering the problem, do it again. See if Start_Stop_Count, Power_Cycle_Count or Power-Off_Retract_Count has increased. If so, take out your PSU, bury it half-deep in your backyard, apply some gasoline, light it up and enjoy the sight of perishing evil with a can of beer.

Thanks.

Signature · 1 line
tejun
#47 lore raw

Hi Tejun,

thanks a lot for your reply.

Am Freitag 25 September 2009 schrieb Tejun Heo:

Quoted 8 lines

Hello, Berthold.

The disk is most likely losing power briefly. After boot, run "smartctl -a" on the device and record the output. After triggering the problem, do it again. See if Start_Stop_Count, Power_Cycle_Count or Power-Off_Retract_Count has increased. If so, take out your PSU, bury it half-deep in your backyard, apply some gasoline, light it up and enjoy the sight of perishing evil with a can of beer.

You might be right. However, I cannot reproduce the problem anymore, since I switched to the totally unsupported JFS as filesystem.

In the meantime, I was able to copy 1.5TB of data back to the array, and the system also survived artificially generated high load. If the problem is a race (which I do not know), it might still be there. Obviously, it does not show up as often again.

It could still be the power supply of course, but I don't understand why a new kernel would trigger power outages so often (current kernels triggered the problem latest after 5 minutes). Maybe it has something to do with the chipset (ICH7R) which is capable of hot remove/add disks. Or it is related to the hotswap harddisk slots in the case
(http://www.chenbro.eu/corporatesite/products_detail.php?sku=79 ). I have no idea....

Thanks

Berthold

#48 lore raw

Hello,

Quoted 3 lines · Berthold Gunreben

It could still be the power supply of course, but I don't understand why a new kernel would trigger power outages so often (current kernels triggered the problem latest after 5 minutes).

Probably because older kernels didn't handle barrier properly for raid. Changing fs can definitely affect things but if I were you I wouldn't be trusting any valuable data on that configuration.

Signature · 1 line
tejun
#49 lore raw
In reply to Alan Cox
Quoted 8 lines · Alan Cox
Quoted 3 lines

something to the cdrom which is attached to ata2. Something very fishy is going on there. Sounds like an electric or some sort of interference problem but I'm not sure. :-(

I'm just wondering what happens on a resume if a phy change event is seen by the phy before or during the time registers (such as the port mappings on an ICH) are being restored (or not restored even). That seems to be one way you could get an event on the wrong port.

It doesn't really matter. Before the ports are resumed, all such events are ignored.

Thanks.

Signature · 1 line
tejun