Skip to contentSkip to conversation

Serious performance degradation on a RAID with

Gelato-technical David Mosberger 5 participants 15 messages Apr 5, 2005 – Apr 6, 2005

lore
Share & tools
lore
#0 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius

Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).

Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.

	--david

[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html

#1 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

I've recently had discussions with several vendors who have mentioned similar magnitudes of disk performance degradation due to the coupling of vibration between cooling fans and disks. This can dramatically increase seek time by keeping the arm from settling. In one case, upgrading a chassis fan caused disk throughput to go down by a factor of 16. The solution is careful attention to vibration damping in mountings. Vibration caused by fan bearing wear could have a similar effect.

Can you go back to your old kernel to verify that it is still giving you the same performance?

-- Rob Fowler

Quoted 31 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius

Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).

Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.

	--david

[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html

_______________________________________________
Gelato-technical mailing list
[email protected]
https://www.gelato.unsw.edu.au/mailman/listinfo/gelato-technical

#2 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11

In reply to David Mosberger
Quoted 4 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius

Is there any reason to believe this change in behavior is ia64-specific? I doubt it.

Probably not ia64-specifc again, but I wonder what you make of
http://bugme.osdl.org/show_bug.cgi?idB68 . The LSI MPT scsi hosts are _particularly_ prevalent in the ia64 world.

Duraid

#3 lore raw
In reply to David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Duraid Madina

Duraid> On Tue, Apr 05, 2005 at 01:25:40PM -0700, David Mosberger

Quoted 5 lines · Duraid> wrote:
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius

<[email protected]> said:

Is there any reason to believe this change in behavior is ia64-specific? I doubt it.

Duraid> Probably not ia64-specifc again, but I wonder what you make Duraid> of http://bugme.osdl.org/show_bug.cgi?idB68 . The LSI MPT Duraid> scsi hosts are _particularly_ prevalent in the ia64 world.

Did you compile the driver as a module? If so, this should be of interest:

http://linux.bkbits.net:8080/linux-2.5/cset@4241a3b18VBJETehoTH3y5vcZilSwA

	--david
#4 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11

In reply to David Mosberger
Quoted 3 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Duraid Madina

Did you compile the driver as a module?

Nope. You're not seeing the problem?

Duraid

#5 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11

In reply to David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Duraid Madina

Duraid> On Tue, Apr 05, 2005 at 09:45:34PM -0700, David Mosberger

Quoted 4 lines · Duraid> wrote:
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Wed, 6 Apr 2005 11:13:30 +0900, Duraid Madina

<[email protected]> said:

Did you compile the driver as a module?

Duraid> Nope. You're not seeing the problem?

I see it with 2.6.11 (38MB/s). 2.6.12-rc2 is definitely better (53MB/s) but IIRC we used to be closer to 60MB/s, so perhaps something still isn't quite optimal.

	--david
#6 lore raw

RE: [Gelato-technical] Serious performance degradation on a RAID withkernel 2.6.11

In reply to David Mosberger
Quoted 2 lines · David Mosberger

Did you compile the driver as a module?

Duraid Madina wrote on Tuesday, April 05, 2005 9:53 PM

Quoted 1 line

Nope. You're not seeing the problem?

That is the reason for the performance problem. And this is definitely *not* ia64 specific. See this discussion which started with performance problem seen on opteron and reproduced on DELL machine as well.
http://marc.theaimsgroup.com/?t1028927300002&r=1&w=2

The changeset that david pointed out is the right fix. Or you can just take the small patch I posted in the above thread.

- Ken

#7 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 11 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Andreas Hirstius

Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).

Quoted 15 lines

Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.

The problems started with 2.6.10-bk7...
(and I've tested 2.6.9, 2.6.10, 2.6.10-bk5 - -bk14, 2.6.11-rc[1-5], 2.6.11, 2.6.11-mm[12], 2.6.12-rc[12])

The only other platform I have is a dual-xeon and I don't see the problem there (same controllers/disks)...

Because of this strange correlation between block size and page size and the fact, that I don't see a problem on x86 I suspected an ia64-specific problem. My "favourite" candidate is the conversion to 4 level pagetables between 2.6.10-bk6 and 2.6.10-bk7.

Andreas

Quoted 10 lines
	--david

[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 9 lines · Rob Fowler

I've recently had discussions with several vendors who have mentioned similar magnitudes of disk performance degradation due to the coupling of vibration between cooling fans and disks. This can dramatically increase seek time by keeping the arm from settling. In one case, upgrading a chassis fan caused disk throughput to go down by a factor of 16. The solution is careful attention to vibration damping in mountings. Vibration caused by fan bearing wear could have a similar effect.

We've also seen similiar problems before... So I made sure it's not something "mechanical" ;-)

Quoted 3 lines

Can you go back to your old kernel to verify that it is still giving you the same performance?

I'm constantly going back and forth between "working" and "not working" kernels.
The performance with a given kernel doesn't change...

Andreas

Quoted 40 lines

-- Rob Fowler

Quoted 33 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius <[email protected]> said:

Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).

Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.

    --david

[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html

_______________________________________________
Gelato-technical mailing list
[email protected]
https://www.gelato.unsw.edu.au/mailman/listinfo/gelato-technical

#9 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius

Andreas> Because of this strange correlation between block size and Andreas> page size and the fact, that I don't see a problem on x86 I Andreas> suspected an ia64-specific problem. My "favourite" Andreas> candidate is the conversion to 4 level pagetables between Andreas> 2.6.10-bk6 and 2.6.10-bk7.

That's certainly a possibility. But didn't you say the problem also occurred when using 4KB page size on ia64?

	--david
#10 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 4 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Andreas Hirstius
Quoted 11 lines

Andreas> Because of this strange correlation between block size and Andreas> page size and the fact, that I don't see a problem on x86 I Andreas> suspected an ia64-specific problem. My "favourite" Andreas> candidate is the conversion to 4 level pagetables between Andreas> 2.6.10-bk6 and 2.6.10-bk7.

That's certainly a possibility. But didn't you say the problem also occurred when using 4KB page size on ia64?

Yes, the problem appears at any page size...

Andreas

Quoted 4 lines
	--david
#11 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 15 lines · Kenneth W
Quoted 3 lines · David Mosberger

Did you compile the driver as a module?

Duraid Madina wrote on Tuesday, April 05, 2005 9:53 PM

Quoted 3 lines

Nope. You're not seeing the problem?

Quoted 10 lines

That is the reason for the performance problem. And this is definitely *not* ia64 specific. See this discussion which started with performance problem seen on opteron and reproduced on DELL machine as well.
http://marc.theaimsgroup.com/?t1028927300002&r=1&w=2

The changeset that david pointed out is the right fix. Or you can just take the small patch I posted in the above thread.

The problems mentioned in this discussion about the Fusion-MPT I can perfectly reproduce on my boxes with 2.6.11, but only with pages sizes 4k and 8k;
With 16k and 64k pages I don't see any problem at all !!! Actually I see block size <-> page size correlations which are similiar to the correlations I see in my setup...
i.e. 2.6.11 and 4k pages:

    dd ...bs@96: 40MB/s
    dd ...bs�92: 74MB/s

i.e. 2.6.11 and 8k pages:

    dd ...bs�92: 56MB/s
    dd ...bs384: 74MB/s

with the driver as module I don't see any change (??).
And I also don't see any change in performance for 4k/8k pages, when applying either your small patch or the cset david pointed to... Since the fusion-mpt driver in 2.6.12-rc2 has this cset applied and no other changes, I also don't see improvements with this kernel... (Again with 16k or 64k pages no performance problem in 2.6.12-rc2 with the fusion-mpt driver...)

But the driver in questions is 3w-9xxx and there have been no chances at all in this driver between 2.6.10 and 2.6.12-rc1. The changed driver in -rc2 still has the problem.
And I've tried the driver built-in and as a module (default)...

Andreas

Quoted 6 lines

- Ken

#12 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius

Andreas> But the driver in questions is 3w-9xxx and there have been Andreas> no chances at all in this driver between 2.6.10 and Andreas> 2.6.12-rc1. The changed driver in -rc2 still has the Andreas> problem. And I've tried the driver built-in and as a Andreas> module (default)...

Are you sure the problem is due to the driver and not RAID0? What if you dd directly from the underlying device?

	--david
#13 lore raw

Re: [Gelato-technical] Serious performance degradation on a RAID

In reply to David Mosberger
Quoted 4 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Andreas Hirstius
Quoted 11 lines

Andreas> But the driver in questions is 3w-9xxx and there have been Andreas> no chances at all in this driver between 2.6.10 and Andreas> 2.6.12-rc1. The changed driver in -rc2 still has the Andreas> problem. And I've tried the driver built-in and as a Andreas> module (default)...

Are you sure the problem is due to the driver and not RAID0? What if you dd directly from the underlying device?

To make sure it's not the md/raid I just took the driver/md directory from 2.6.9 and 2.6.10 and used it in 2.6.11 => no change in behaviour ... (The changes in md.c are minor and there are no changes in raid0.c between .10 and .11 anyway)

As I say on the webpage, a dd on the underlying devices works fine...

It might still be a higher level interface to the md, that's wrong.

And I'm not saying that it's the 3w-9xx driver!! It hasn't changed between 2.6.10 and 2.6.11, so pretty much impossible to introduce bugs there...
(It was just mentioned wrt to the fusion-mpt driver.)

Andreas

Quoted 4 lines
	--david
#14 lore raw

RE: [Gelato-technical] Serious performance degradation on a RAID withkernel 2.6.11

In reply to David Mosberger

Andreas Hirstius wrote on Wednesday, April 06, 2005 1:33 AM

Quoted 7 lines

The problems mentioned in this discussion about the Fusion-MPT I can perfectly reproduce on my boxes with 2.6.11, but only with pages sizes 4k and 8k;
With 16k and 64k pages I don't see any problem at all !!! Actually I see block size <-> page size correlations which are similiar to the correlations I see in my setup...
i.e. 2.6.11 and 4k pages:

Quoted 6 lines
    dd ...bs@96: 40MB/s
    dd ...bs�92: 74MB/s

i.e. 2.6.11 and 8k pages:

    dd ...bs�92: 56MB/s
    dd ...bs384: 74MB/s

It is very odd. I think it has something to do with generic file read ahead logic. The read ahead window did not push far enough to keep the disk busy. Some of the size calculation are in unit of page, so smaller page size will have smaller effective read ahead size. Try look into mm/readahead.c. I bet you can find the answer to your performance problem there. If not, then I guess we have to keep on looking ;-)