ASUS M2A-VM - AMD690G - SB600 Non-Raid-5 SATA - problem

I had a significant breakthrough: when I removed two out of four 1 GB modules (just a stab in the dark to see if the system stability problem I reported to LKML ( http://marc.info/?l=linux-kernel&m=117813911431765&w=2 ; a few sample oops messages attached for reference) goes away, I stuck a gold mine: now the SATA controller detects both the drives :-)).

So, it'd seem the SATA controller wouldn't work fine when one has a lot of RAM on this board. The manual/manufacture's website claim it supports 8 GB of DDR2 800, while I only have 4, yet there are a few severe problems already: like this SATA drive detection problem & the system instability.

I now got things working very reliably (even without pci=nomsi) with 2 GB of RAM (3 times longer kernel compile torture time with 2 GB, without kernel issues, with both pairs of RAM individually, compared with all of 4 GB where the system is plagued with instability). I intent to check how far below 4095M I could use mem= parameter with, without destablising the system, assuming if at all mem= parameters going to help.

For the record, here's the complete dmesg of the current working system as an attachment (dmesg-2.6.21.1-2gigs):

Thank you for your insight into the problem. If there are any tricks involved in making use of all 4 GB of RAM with SATA drives & with system stability, I'll be very happy to learn them :-)).

Srihari Vijayaraghavan wrote:

Quoted 1 line

Thank you for your insight into the problem. If there are any tricks

involved

Quoted 2 lines

in making use of all 4 GB of RAM with SATA drives & with system stability, I'll be very happy to learn them :-)).

It might be that you just have a bad ram module. Does the machine work properly if you exchange the installed two modules with the other two? memtet86 time, I guess.

Even now, when 4 GB of RAM is installed, by default, the SATA drives aren't recongnised in Linux, except when I pass mem=4095MB. If I pass mem=4095M, alas only 3.6 of RAM is detected under Linux!

Here's the relevant section from dmesg:
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000dfee0000 (usable)
BIOS-e820: 00000000dfee0000 - 00000000dfee3000 (ACPI NVS)
BIOS-e820: 00000000dfee3000 - 00000000dfef0000 (ACPI data)
BIOS-e820: 00000000dfef0000 - 00000000dff00000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
Entering add_active_range(0, 0, 159) 0 entries of 3200 used Entering add_active_range(0, 256, 917216) 1 entries of 3200 used end_pfn_map = 1048576
...
Memory: 3570504k/3668864k available (2465k kernel code, 97972k reserved, 1445k data, 332k init)
...

Surprisingly, this usable ~3.6 GB equals to the initial two e820 sections only. I believe, when mem= parameter is used, the third 'usable' section (between 0x100000000 - 0x120000000) isn't used at all by Linux.

I wonder if there is a trick involved in using mem= or memmap= needs to used also (if so, I don't know yet how to construct memmap= parameter).

To recap:
System is very stable with the new case, power supply & fans SATA drives aren't by default detected when 4 GB of RAM is used SATA drives are detected by default when only 2 GB of RAM is physically
installed (ie, with both pairs of 2 GB of DIMMs individually)
SATA drives are detected when mem=4095M (the reported memory size in the
BIOS) is used, but only ~3.6 GB of RAM is used (suspected that the final e820 usable section, though detected, is not used).

Thank you for your help so far. I really appreciated that.

Hello,

It just seems sb600 claims that it can do DMA above 4G while it actually can't. Can you test the attached patch?

Thanks.

Signature · 1 line

tejun

Quoted 1 line

---

 drivers/ata/ahci.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

Index: tree0/drivers/ata/ahci.c
===================================================================

drivers/ata/ahci.c +3 -1

@@ -170,6 +170,7 @@ enum {
 	AHCI_FLAG_IGN_IRQ_IF_ERR	= (1 << 25), /* ignore IRQ_IF_ERR */
 	AHCI_FLAG_HONOR_PI		= (1 << 26), /* honor PORTS_IMPL */
 	AHCI_FLAG_IGN_SERR_INTERNAL	= (1 << 27), /* ignore SERR_INTERNAL */
+	AHCI_FLAG_32BIT_ONLY		= (1 << 28), /* force 32bit */
 };
 
 struct ahci_cmd_hdr {

@@ -370,7 +371,8 @@ static const struct ata_port_info ahci_p
 		.flags		= ATA_FLAG_SATA | ATA_FLAG_NO_LEGACY |
 				  ATA_FLAG_MMIO | ATA_FLAG_PIO_DMA |
 				  ATA_FLAG_SKIP_D2H_BSY |
-				  AHCI_FLAG_IGN_SERR_INTERNAL,
+				  AHCI_FLAG_IGN_SERR_INTERNAL |
+				  AHCI_FLAG_32BIT_ONLY,
 		.pio_mask	= 0x1f, /* pio0-4 */
 		.udma_mask	= 0x7f, /* udma0-6 ; FIXME */
 		.port_ops	= &ahci_ops,

@@ -1579,6 +1581,12 @@ static int ahci_host_init(struct ata_pro
 		probe_ent->n_ports = cap_n_ports;
 

Quoted 2 lines · --- Tejun Heo

It just seems sb600 claims that it can do DMA above 4G while it actually can't. Can you test the attached patch?

Makes sense now. Gladly check the patch. Checking ...

Great news: everything just works without any parameters :-)). True genious.

With your patch, with 4 GB of RAM, all SATA drives are detected & working normally. For the record dmesg is attached.

There's always a catch isn't it? Now no DMA over 32 bit memory space for SATA, right? CPU will have to work harder, I presume. Perhaps putting more pressure. While with 4 GB that might not be a problem, perhaps when I upgrade it to over 4 GB, it might be?? I see, even now, because a chunk of 512 MB is mapped over 4 GB (starting at 0x100000000), those are no good for the SATA's DMA access. If so, too sad :-(.

(I'll leave the patch for the poor souls stumble upon these emails some far future :-))

PS: if this turns out to a hardware limitation, then perhaps may I request this patch in better form (as not to impact on other ahci users) find its way to mainstream? Even to -stable if I'm lucky, such that Fedora 7 etc. might work out of the box for poor souls (not counting me; I'm used to quirky things) trying out Linux on this chipset.

Yeap, I'll forward it upstream and probably to -stable too. The problem here is that I'm not sure whether the problem is in the ahci controller itself or some bridge chip above it. It would be best if Conke Hu can verify this but you can also test it by buying a ATA controller which can do 64bit and whether it works properly on the machine. :-)

Just in case, can you post the result of 'lspci -nntv'?

Hello,

Quoted 3 lines · Srihari Vijayaraghavan

Quoted 2 lines · --- Tejun Heo

It just seems sb600 claims that it can do DMA above 4G while it actually can't. Can you test the attached patch?

Quoted 2 lines

With your patch, with 4 GB of RAM, all SATA drives are detected & working normally. For the record dmesg is attached.

Thanks for testing.

Oh well, that's the price you have to pay when you 1. have a device which can't access memory above 4G but 2. don't have IOMMU to do it for the device. If performance becomes problem, you can always get a not-so-expensive external controller which can do 64.

Yeap, I'll forward it upstream and probably to -stable too. The problem here is that I'm not sure whether the problem is in the ahci controller itself or some bridge chip above it.

It would be best if Conke Hu can
verify this but you can also test it by buying a ATA controller which can do 64bit and whether it works properly on the machine. :-)

Quoted 4 lines

Oh well, that's the price you have to pay when you 1. have a device which can't access memory above 4G but 2. don't have IOMMU to do it for the device. If performance becomes problem, you can always get a not-so-expensive external controller which can do 64.

Well, I think my AM2 chip is capable of IOMMU (well that's one of the deciding factor even since my original AMD64 3200+ :-)). BTW is there an easier way to prove whether IOMMU is in use?

Quoted 3 lines

It would be best if Conke Hu can
verify this but you can also test it by buying a ATA controller which can do 64bit and whether it works properly on the machine. :-)

Let's hope he does.

Buying an (S)ATA controller now is a lofty goal (for the lack of money more than anything :-)). But doesn't my current on-board IDE doesn't do 64bit DMA?

Here is its lspci -vvv:
00:14.1 IDE interface: ATI Technologies Inc SB600 IDE (prog-if 82 [Master PriP])
Subsystem: ASUSTeK Computer Inc. Unknown device 81ef
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Step
ping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort
- <MAbort- >SERR- <PERR-

        Latency: 64
        Interrupt: pin A routed to IRQ 16
        Region 0: I/O ports at 01f0 [size=8]
        Region 1: I/O ports at 03f4 [size=1]
        Region 2: I/O ports at 0170 [size=8]
        Region 3: I/O ports at 0374 [size=1]
        Region 4: I/O ports at f900 [size=16]
        Capabilities: [70] Message Signalled Interrupts: Mask- 64bit-
Queue=0/0 
Enable-
                Address: 00000000  Data: 0000

I'm not sure if this IDE controller is operating in 64 bit DMA?? (well if it didn't my IDE peripherals might not have worked properly without mem=4095M, like the SATA controller, no? They always worked fine.) I thought all PCI-E based devices were capable, no?

(snip)
Conke Hu, please respond.
(snip)

Quoted 3 lines · Tejun Heo

(snip)
Conke Hu, please respond.
(snip)

i have left AMD and i doubt if it's still good for me to discuss any questions regarding AMD chipset :(

what's more, now i am busy hunting for my next job, but "I WILL BE BACK" when i get my new job :)

Quoted 4 lines · Srihari Vijayaraghavan

Quoted 4 lines

Oh well, that's the price you have to pay when you 1. have a device which can't access memory above 4G but 2. don't have IOMMU to do it for the device. If performance becomes problem, you can always get a not-so-expensive external controller which can do 64.

Quoted 5 lines · --- Tejun Heo

Quoted 4 lines · Srihari Vijayaraghavan

Quoted 4 lines

Oh well, that's the price you have to pay when you 1. have a device which can't access memory above 4G but 2. don't have IOMMU to do it for the device. If performance becomes problem, you can always get a not-so-expensive external controller which can do 64.

Can the Skymaster S3132, based on Sil3132,
(http://umart.net/au/product_info.php?cPath=21_211&products_id=106108) do 64 bit DMA on Linux?

If so, I'm happy to buy that to isolate whether the current problem is limited to SB600 SATA controller alone on AMD690G chipset.

It'd be nice if somebody from AMD/ATI could confirm the 64 bit DMA capability of SB600 SATA controller. (I observe a few AMD/ATI folks have been submitting a few patches on LKML lately, so who knows.)

PS: FWIW, SB600 with your ahci fix has been working fine for the past week. Thank you for that.

Quoted 1 line · Srihari Vijayaraghavan

--- Tejun Heo <[email protected]> wrote:

Quoted 2 lines

Can the Skymaster S3132, based on Sil3132,
(http://umart.net/au/product_info.php?cPath=21_211&products_id=106108) do

64

Quoted 1 line

bit DMA on Linux?

Yeah, sil3124/32 can do 64bit alright.

Maybe you can cc the ATI person here if you know the address.

Quoted 1 line

PS: FWIW, SB600 with your ahci fix has been working fine for the past

week.

Quoted 1 line

Thank you for that.

Great.

Quoted 4 lines

Quoted 2 lines

Can the Skymaster S3132, based on Sil3132,
(http://umart.net/au/product_info.php?cPath=21_211&products_id=106108) do

64

Quoted 1 line

bit DMA on Linux?

Quoted 1 line

Yeah, sil3124/32 can do 64bit alright.

Sorry, I couldn't quite convince myself wasting a little bit of money on that. Temptation is hard to resist though.

Quoted 1 line

Maybe you can cc the ATI person here if you know the address.

Will do. Need to write a good story :-).

SB600 claims it can do 64bit DMA but it can't. Disable it.

DON'T APPLY YET
---

Srihari, can you test this against 2.6.22-rc1-git7 just in case? I'll send this upstream after you ack.

Thanks.

 ahci.c |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

drivers/ata/ahci.c +3 -1

index e00e1b9..bb94047 100644
@@ -170,6 +170,7 @@ enum {
 	AHCI_FLAG_IGN_IRQ_IF_ERR	= (1 << 25), /* ignore IRQ_IF_ERR */
 	AHCI_FLAG_HONOR_PI		= (1 << 26), /* honor PORTS_IMPL */
 	AHCI_FLAG_IGN_SERR_INTERNAL	= (1 << 27), /* ignore SERR_INTERNAL */
+	AHCI_FLAG_32BIT_ONLY		= (1 << 28), /* force 32bit */
 
 	AHCI_FLAG_COMMON		= ATA_FLAG_SATA | ATA_FLAG_NO_LEGACY |
 					  ATA_FLAG_MMIO | ATA_FLAG_PIO_DMA |

@@ -354,7 +355,8 @@ static const struct ata_port_info ahci_p
 	/* board_ahci_sb600 */
 	{
 		.flags		= AHCI_FLAG_COMMON |
-				  AHCI_FLAG_IGN_SERR_INTERNAL,
+				  AHCI_FLAG_IGN_SERR_INTERNAL |
+				  AHCI_FLAG_32BIT_ONLY,
 		.pio_mask	= 0x1f, /* pio0-4 */
 		.udma_mask	= 0x7f, /* udma0-6 ; FIXME */
 		.port_ops	= &ahci_ops,

@@ -492,6 +494,13 @@ static void ahci_save_initial_config(str
 	hpriv->saved_cap = cap = readl(mmio + HOST_CAP);
 	hpriv->saved_port_map = port_map = readl(mmio + HOST_PORTS_IMPL);
 

SB600 claims it can do 64bit DMA but it can't. Disable it.

Signed-off-by Tejun Heo <[email protected]>

---
Jeff, this one is for #upstream-fixes.

 ahci.c |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)