Skip to contentSkip to conversation

PATA IDE is slower in newer versions of kernel

DISCUSSION Joshua Hintze 6 participants 11 messages Sep 16, 2010 – Sep 20, 2010 1 message not in the archive libata

lore
Share & tools
lore

Joshua Hintze

author
#0 lore raw

Hello,

I've been digging into this for a couple weeks now. I have an embedded board that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA connection. Data is transferred using UDMA4 speeds and I get a nice sustained write speed of about 20 MBps which is adequate for us.

I have recently taken the plunge to update to a newer version of the kernel starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for sustained throughput.

I decided to read up on the ATA specification and I connected a logic analyzer to the PATA bus and here is what is happening...after a large chunk of data is written to the device, instead of the device pausing the transfer by asserting DDMARDY it actual initiates a device data-out termination by pulling DMARQ low.

The old kernel 2.6.10 responds to this by strobing the STOP line 8 times within 800uS and the device releases DMARQ shortly afterwards. On the newer kernels the strobing of the STOP line takes near 8mS of time. Since the hard drive only release DMARQ after this 8 pulse strobe I believe this is the cause of the slower write speeds on the newer kernels. This whole process happens thousands of times when writing megs of data. So those extra 7mS begin to add up fast.

My problem is I've been digging through the ide.c/ide-dma.c and other code but I'm not exactly sure where the code would jump to upon a device initiated termination on a data-out dma transfer.

My guess is ide_dma_intr(...). Is this correct?

Any other areas I could look at?

Thanks in advance.

#1 lore raw

El Thu, 16 Sep 2010 17:30:51 -0600
"Joshua Hintze" <[email protected]> escribió:

[ Just CC linux-ide ]

Quoted 42 lines

Hello,

I've been digging into this for a couple weeks now. I have an embedded board that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA connection. Data is transferred using UDMA4 speeds and I get a nice sustained write speed of about 20 MBps which is adequate for us.

I have recently taken the plunge to update to a newer version of the kernel starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for sustained throughput.

I decided to read up on the ATA specification and I connected a logic analyzer to the PATA bus and here is what is happening...after a large chunk of data is written to the device, instead of the device pausing the transfer by asserting DDMARDY it actual initiates a device data-out termination by pulling DMARQ low.

The old kernel 2.6.10 responds to this by strobing the STOP line 8 times within 800uS and the device releases DMARQ shortly afterwards. On the newer kernels the strobing of the STOP line takes near 8mS of time. Since the hard drive only release DMARQ after this 8 pulse strobe I believe this is the cause of the slower write speeds on the newer kernels. This whole process happens thousands of times when writing megs of data. So those extra 7mS begin to add up fast.

My problem is I've been digging through the ide.c/ide-dma.c and other code but I'm not exactly sure where the code would jump to upon a device initiated termination on a data-out dma transfer.

My guess is ide_dma_intr(...). Is this correct?

Any other areas I could look at?

Thanks in advance.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#2 lore raw

Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.

http://i56.tinypic.com/oqhrug.png

Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.

http://i51.tinypic.com/a9wiur.png

In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Thanks!

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alejandro Riveira Fernández
Sent: Friday, September 17, 2010 7:48 AM
To: Joshua Hintze
Cc: [email protected]; [email protected]
Subject: Re: PATA IDE is slower in newer versions of kernel

El Thu, 16 Sep 2010 17:30:51 -0600
"Joshua Hintze" <[email protected]> escribió:

[ Just CC linux-ide ]

Quoted 34 lines

Hello,

I've been digging into this for a couple weeks now. I have an embedded board that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA connection. Data is transferred using UDMA4 speeds and I get a nice sustained write speed of about 20 MBps which is adequate for us.

I have recently taken the plunge to update to a newer version of the kernel starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for sustained throughput.

I decided to read up on the ATA specification and I connected a logic analyzer to the PATA bus and here is what is happening...after a large chunk of data is written to the device, instead of the device pausing the transfer by asserting DDMARDY it actual initiates a device data-out termination by pulling DMARQ low.

The old kernel 2.6.10 responds to this by strobing the STOP line 8 times within 800uS and the device releases DMARQ shortly afterwards. On the newer kernels the strobing of the STOP line takes near 8mS of time. Since the hard drive only release DMARQ after this 8 pulse strobe I believe this is the cause of the slower write speeds on the newer kernels. This whole process happens thousands of times when writing megs of data. So those extra 7mS begin to add up fast.

My problem is I've been digging through the ide.c/ide-dma.c and other code but I'm not exactly sure where the code would jump to upon a device initiated termination on a data-out dma transfer.

My guess is ide_dma_intr(...). Is this correct?

Any other areas I could look at?

Quoted 8 lines

Thanks in advance.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3132 - Release Date: 09/16/10 12:34:00

#3 lore raw
Quoted 6 lines · Joshua Hintze

Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.

http://i56.tinypic.com/oqhrug.png

Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.

Quoted 8 lines

http://i51.tinypic.com/a9wiur.png

In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Thanks!

Maybe you can just bisect this?
Just a general advice, I don't know ide subsystem at all.

Best regards,
Maxim Levitsky

author
#4 lore raw

Thanks,

I'm not sure what you mean by bisect, other than just the definition of the word. I am breaking this problem down adding printks etc. I think I'm finally beginng to understand how this works :)

Josh

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Maxim Levitsky
Sent: Saturday, September 18, 2010 1:23 PM
To: Joshua Hintze
Cc: [email protected]; [email protected]
Subject: RE: PATA IDE is slower in newer versions of kernel

Quoted 6 lines · Joshua Hintze

Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.

http://i56.tinypic.com/oqhrug.png

Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.

Quoted 8 lines

http://i51.tinypic.com/a9wiur.png

In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Thanks!

Maybe you can just bisect this?
Just a general advice, I don't know ide subsystem at all.

Best regards,
Maxim Levitsky

--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3132 - Release Date: 09/18/10 00:34:00

#5 lore raw
Quoted 4 lines · On Sat, 18 Sep 2010 13:43:40 -0600 Joshua Hintze wrote:

Thanks,

I'm not sure what you mean by bisect, other than just the definition of the word. I am breaking this problem down adding printks etc. I think I'm finally beginng to understand how this works :)

http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#using-bisect

Quoted 22 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Maxim Levitsky
Sent: Saturday, September 18, 2010 1:23 PM
To: Joshua Hintze
Cc: [email protected]; [email protected]
Subject: RE: PATA IDE is slower in newer versions of kernel

Quoted 12 lines · Joshua Hintze

Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.

http://i56.tinypic.com/oqhrug.png

Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.

http://i51.tinypic.com/a9wiur.png

In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Quoted 7 lines
Quoted 2 lines

Thanks!

Maybe you can just bisect this?
Just a general advice, I don't know ide subsystem at all.

Best regards,
Maxim Levitsky

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

author
#6 lore raw

I got you.

Thanks.

-----Original Message-----
From: Randy Dunlap [mailto:[email protected]]
Sent: Saturday, September 18, 2010 1:48 PM
To: Joshua Hintze
Cc: 'Maxim Levitsky'; [email protected]; [email protected]
Subject: Re: PATA IDE is slower in newer versions of kernel

Quoted 3 lines · On Sat, 18 Sep 2010 13:43:40 -0600 Joshua Hintze wrote:

Thanks,

I'm not sure what you mean by bisect, other than just the definition of

the word. I am breaking this problem down adding printks etc. I think I'm finally beginng to understand how this works :)

Quoted 1 line

http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#using-bisec
t

Quoted 4 lines

-----Original Message-----
From: [email protected]

[mailto:[email protected]] On Behalf Of Maxim Levitsky

Quoted 7 lines

Sent: Saturday, September 18, 2010 1:23 PM
To: Joshua Hintze
Cc: [email protected]; [email protected]
Subject: RE: PATA IDE is slower in newer versions of kernel

Quoted 1 line · Joshua Hintze

Some updated information. I created a screen capture of what I am

seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.

Quoted 4 lines
Quoted 4 lines

http://i56.tinypic.com/oqhrug.png

Notice in the above link how there is a lot of activating and then large

gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.

Quoted 4 lines
Quoted 4 lines

http://i51.tinypic.com/a9wiur.png

In the above image notice how the wait time is considerable less 100's

of microseconds before the stop lines get strobed.

Quoted 2 lines
Quoted 2 lines

I'm dedicated to continue working on the problem, I just need to advice

from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Quoted 8 lines
Quoted 3 lines

Thanks!

Maybe you can just bisect this?
Just a general advice, I don't know ide subsystem at all.

Best regards,
Maxim Levitsky

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3132 - Release Date: 09/18/10 00:34:00

#7 lore raw
In reply to Maxim Levitsky
Quoted 9 lines · On Saturday 18 September 2010 21:22:54 Maxim Levitsky wrote:
Quoted 7 lines · Joshua Hintze

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Thanks!

Maybe you can just bisect this?

That would probably help to pinpoint the problem. A regression somewhere between 2.6.10 and 2.6.32 is not that interesting when people are working on stabilizing 2.6.36 and adding features to 2.6.37.

One important question would be if this happens with the libata drivers in 2.6.36-rc4 or linux-next as well. Note that the IDE subsystem has been replaced with libata a few years ago, and that is what all development happens on. If libata is good enough, there is probably no point in chasing down ide bugs.

Arnd

#8 lore raw
In reply to Joshua Hintze
Quoted 1 line

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

drivers/ide is pretty much obsolete - so what you are seeing is doubly odd. Most stuff now is using the newer drivers/ata code which can support SATA, NCQ etc unlike the old codebase.

One thing to check would be whether using the drivers/ata code shows the same problem.

Alan

author
#9 lore raw

Thank you for the input. The reason why I'm using the old drivers/ide is because I'm using a TI Davinci ARM processor in an embedded system. They wrote a PalmChip driver for the older IDE interface and I don't think they have anything yet for the drivers/ata (out of curiousity is the newer drivers/ata the reason why PATA devices show up as /dev/sda in newer kernels?)

After spending most of the day today adding printks and learning about the old ide driver I'm pretty much convinced that the IDE driver is working fine. The long delays are actually between block driver request queues. In the 2.6.10 kernel the time between block writes is less than 1 ms, where as the 2.6.32 that TI has blessed takes 8+ ms between blocks that are pushed to the IDE interface.

Like I mentioned earlier I tried changing the I/O scheduler to Noop and as for the elevator. Anything else I should look to so I get blocks sent out faster?

Thanks

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Alan Cox
Sent: Saturday, September 18, 2010 6:59 PM
To: Joshua Hintze
Cc: [email protected]; [email protected]
Subject: Re: PATA IDE is slower in newer versions of kernel

Quoted 1 line

I'm dedicated to continue working on the problem, I just need to advice

from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

drivers/ide is pretty much obsolete - so what you are seeing is doubly odd. Most stuff now is using the newer drivers/ata code which can support SATA, NCQ etc unlike the old codebase.

One thing to check would be whether using the drivers/ata code shows the same problem.

Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3143 - Release Date: 09/18/10 00:34:00

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#11 lore raw

Thank you,

The biggest effect I saw playing around with these is that when dirty_expire_centisecs is set at 0 writes speeds drastically decrease, but they are more constant.

I compared all the settings for these registers versus the 2.6.10 kernel and most of them are the same except for dirty_ratio. I made all of the registers that were available (since the newer kernels have more) but I didn't see any drastic improvement.

I'll continue playing with them. Thanks

-----Original Message-----
From: Jeffrey Hundstad [mailto:[email protected]]
Sent: Monday, September 20, 2010 12:50 PM
To: Joshua Hintze
Subject: Re: PATA IDE is slower in newer versions of kernel

Can you try the following:

echo 0 > /proc/sys/vm/dirty_writeback_centisecs

and/or

echo 0 > /proc/sys/vm/dirty_expire_centisecs

You can find info. about these at:
http://www.linuxinsight.com/proc_sys_vm_hierarchy.html
http://www.westnet.com/~gsmith/content/linux-pdflush.htm

Quoted 3 lines · Joshua Hintze

Hello,

I've been digging into this for a couple weeks now. I have an embedded

board

Quoted 5 lines

that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA connection. Data is transferred using UDMA4 speeds and I get a nice sustained write speed of about 20 MBps which is adequate for us.

I have recently taken the plunge to update to a newer version of the

kernel

Quoted 6 lines

starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for sustained throughput.

I decided to read up on the ATA specification and I connected a logic analyzer to the PATA bus and here is what is happening...after a large

chunk

Quoted 1 line

of data is written to the device, instead of the device pausing the

transfer

Quoted 5 lines

by asserting DDMARDY it actual initiates a device data-out termination by pulling DMARQ low.

The old kernel 2.6.10 responds to this by strobing the STOP line 8 times within 800uS and the device releases DMARQ shortly afterwards. On the

newer

Quoted 1 line

kernels the strobing of the STOP line takes near 8mS of time. Since the

hard

Quoted 13 lines

drive only release DMARQ after this 8 pulse strobe I believe this is the cause of the slower write speeds on the newer kernels. This whole process happens thousands of times when writing megs of data. So those extra 7mS begin to add up fast.

My problem is I've been digging through the ide.c/ide-dma.c and other code but I'm not exactly sure where the code would jump to upon a device initiated termination on a data-out dma transfer.

My guess is ide_dma_intr(...). Is this correct?

Any other areas I could look at?

Quoted 9 lines

Thanks in advance.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3143 - Release Date: 09/20/10 00:35:00