Serious performance degradation on a RAID with
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius
Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).
Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.
--david
[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
Re: [Gelato-technical] Serious performance degradation on a RAID
I've recently had discussions with several vendors who have mentioned similar magnitudes of disk performance degradation due to the coupling of vibration between cooling fans and disks. This can dramatically increase seek time by keeping the arm from settling. In one case, upgrading a chassis fan caused disk throughput to go down by a factor of 16. The solution is careful attention to vibration damping in mountings. Vibration caused by fan bearing wear could have a similar effect.
Can you go back to your old kernel to verify that it is still giving you the same performance?
-- Rob Fowler
Quoted 31 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius
Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).
Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.
--david[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
_______________________________________________
Gelato-technical mailing list
[email protected]
https://www.gelato.unsw.edu.au/mailman/listinfo/gelato-technical
Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11
In reply to David MosbergerQuoted 4 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius
Is there any reason to believe this change in behavior is ia64-specific? I doubt it.
Probably not ia64-specifc again, but I wonder what you make of
http://bugme.osdl.org/show_bug.cgi?idB68 . The LSI MPT scsi hosts are _particularly_ prevalent in the ia64 world.
Duraid
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Duraid Madina
Duraid> On Tue, Apr 05, 2005 at 01:25:40PM -0700, David Mosberger
Quoted 5 lines · Duraid> wrote:
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius
<[email protected]> said:
Is there any reason to believe this change in behavior is ia64-specific? I doubt it.
Duraid> Probably not ia64-specifc again, but I wonder what you make Duraid> of http://bugme.osdl.org/show_bug.cgi?idB68 . The LSI MPT Duraid> scsi hosts are _particularly_ prevalent in the ia64 world.
Did you compile the driver as a module? If so, this should be of interest:
http://linux.bkbits.net:8080/linux-2.5/cset@4241a3b18VBJETehoTH3y5vcZilSwA
--david
Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11
In reply to David MosbergerQuoted 3 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Duraid Madina
Did you compile the driver as a module?
Nope. You're not seeing the problem?
Duraid
Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11
In reply to David MosbergerQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Duraid Madina
Duraid> On Tue, Apr 05, 2005 at 09:45:34PM -0700, David Mosberger
Quoted 4 lines · Duraid> wrote:
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Wed, 6 Apr 2005 11:13:30 +0900, Duraid Madina
<[email protected]> said:
Did you compile the driver as a module?
Duraid> Nope. You're not seeing the problem?
I see it with 2.6.11 (38MB/s). 2.6.12-rc2 is definitely better (53MB/s) but IIRC we used to be closer to 60MB/s, so perhaps something still isn't quite optimal.
--david
RE: [Gelato-technical] Serious performance degradation on a RAID withkernel 2.6.11
In reply to David MosbergerQuoted 2 lines · David Mosberger
Did you compile the driver as a module?
Duraid Madina wrote on Tuesday, April 05, 2005 9:53 PM
Quoted 1 line
Nope. You're not seeing the problem?
That is the reason for the performance problem. And this is definitely *not* ia64 specific. See this discussion which started with performance problem seen on opteron and reproduced on DELL machine as well.
http://marc.theaimsgroup.com/?t1028927300002&r=1&w=2
The changeset that david pointed out is the right fix. Or you can just take the small patch I posted in the above thread.
- Ken
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 11 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Andreas Hirstius
Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).
Quoted 15 lines
Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.
The problems started with 2.6.10-bk7...
(and I've tested 2.6.9, 2.6.10, 2.6.10-bk5 - -bk14, 2.6.11-rc[1-5], 2.6.11, 2.6.11-mm[12], 2.6.12-rc[12])
The only other platform I have is a dual-xeon and I don't see the problem there (same controllers/disks)...
Because of this strange correlation between block size and page size and the fact, that I don't see a problem on x86 I suspected an ia64-specific problem. My "favourite" candidate is the conversion to 4 level pagetables between 2.6.10-bk6 and 2.6.10-bk7.
Andreas
Quoted 10 lines
--david[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 9 lines · Rob Fowler
I've recently had discussions with several vendors who have mentioned similar magnitudes of disk performance degradation due to the coupling of vibration between cooling fans and disks. This can dramatically increase seek time by keeping the arm from settling. In one case, upgrading a chassis fan caused disk throughput to go down by a factor of 16. The solution is careful attention to vibration damping in mountings. Vibration caused by fan bearing wear could have a similar effect.
We've also seen similiar problems before... So I made sure it's not something "mechanical" ;-)
Quoted 3 lines
Can you go back to your old kernel to verify that it is still giving you the same performance?
I'm constantly going back and forth between "working" and "not working" kernels.
The performance with a given kernel doesn't change...
Andreas
Quoted 40 lines
-- Rob Fowler
Quoted 33 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius <[email protected]> said:
Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers Andreas> and 24x WD740GD HDD in a software RAID0 configuration Andreas> (using md). With kernel 2.6.11 the read performance on the Andreas> md is reduced by a factor of 20 (!!) compared to previous Andreas> kernels. The write rate to the md doesn't change!! (it Andreas> actually improves a bit).
Is there any reason to believe this change in behavior is ia64-specific? I doubt it. The q-syscollect profiles on your web-site shows that the CPU is basically completely idle. You may want to try 2.6.10 to see whether the bad behavior was introduced before or after that. Unfortunately, 2.6 is developing very rapidly so you may have to do more binary searching between 2.6.9 and 2.6.10 or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity. You may also want to look at the bitkeeper changelogs to see if you can find anything suspicious (you can do that easily via the web by browsing the source code [1]). Lastly, you may want to ask on linux-kernel whether anybody can think of a change that could have this effect.
--david[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
_______________________________________________
Gelato-technical mailing list
[email protected]
https://www.gelato.unsw.edu.au/mailman/listinfo/gelato-technical
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius
Andreas> Because of this strange correlation between block size and Andreas> page size and the fact, that I don't see a problem on x86 I Andreas> suspected an ia64-specific problem. My "favourite" Andreas> candidate is the conversion to 4 level pagetables between Andreas> 2.6.10-bk6 and 2.6.10-bk7.
That's certainly a possibility. But didn't you say the problem also occurred when using 4KB page size on ia64?
--david
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 4 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Andreas Hirstius
Quoted 11 lines
Andreas> Because of this strange correlation between block size and Andreas> page size and the fact, that I don't see a problem on x86 I Andreas> suspected an ia64-specific problem. My "favourite" Andreas> candidate is the conversion to 4 level pagetables between Andreas> 2.6.10-bk6 and 2.6.10-bk7.
That's certainly a possibility. But didn't you say the problem also occurred when using 4KB page size on ia64?
Yes, the problem appears at any page size...
Andreas
Quoted 4 lines
--david
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 15 lines · Kenneth W
Quoted 3 lines · David Mosberger
Did you compile the driver as a module?
Duraid Madina wrote on Tuesday, April 05, 2005 9:53 PM
Quoted 3 lines
Nope. You're not seeing the problem?
Quoted 10 lines
That is the reason for the performance problem. And this is definitely *not* ia64 specific. See this discussion which started with performance problem seen on opteron and reproduced on DELL machine as well.
http://marc.theaimsgroup.com/?t1028927300002&r=1&w=2The changeset that david pointed out is the right fix. Or you can just take the small patch I posted in the above thread.
The problems mentioned in this discussion about the Fusion-MPT I can perfectly reproduce on my boxes with 2.6.11, but only with pages sizes 4k and 8k;
With 16k and 64k pages I don't see any problem at all !!! Actually I see block size <-> page size correlations which are similiar to the correlations I see in my setup...
i.e. 2.6.11 and 4k pages:
dd ...bs@96: 40MB/s
dd ...bs�92: 74MB/si.e. 2.6.11 and 8k pages:
dd ...bs�92: 56MB/s
dd ...bs384: 74MB/swith the driver as module I don't see any change (??).
And I also don't see any change in performance for 4k/8k pages, when applying either your small patch or the cset david pointed to... Since the fusion-mpt driver in 2.6.12-rc2 has this cset applied and no other changes, I also don't see improvements with this kernel... (Again with 16k or 64k pages no performance problem in 2.6.12-rc2 with the fusion-mpt driver...)
But the driver in questions is 3w-9xxx and there have been no chances at all in this driver between 2.6.10 and 2.6.12-rc1. The changed driver in -rc2 still has the problem.
And I've tried the driver built-in and as a module (default)...
Andreas
Quoted 6 lines
- Ken
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Andreas Hirstius
Andreas> But the driver in questions is 3w-9xxx and there have been Andreas> no chances at all in this driver between 2.6.10 and Andreas> 2.6.12-rc1. The changed driver in -rc2 still has the Andreas> problem. And I've tried the driver built-in and as a Andreas> module (default)...
Are you sure the problem is due to the driver and not RAID0? What if you dd directly from the underlying device?
--david
Re: [Gelato-technical] Serious performance degradation on a RAID
In reply to David MosbergerQuoted 4 lines · David Mosberger
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Andreas Hirstius
Quoted 11 lines
Andreas> But the driver in questions is 3w-9xxx and there have been Andreas> no chances at all in this driver between 2.6.10 and Andreas> 2.6.12-rc1. The changed driver in -rc2 still has the Andreas> problem. And I've tried the driver built-in and as a Andreas> module (default)...
Are you sure the problem is due to the driver and not RAID0? What if you dd directly from the underlying device?
To make sure it's not the md/raid I just took the driver/md directory from 2.6.9 and 2.6.10 and used it in 2.6.11 => no change in behaviour ... (The changes in md.c are minor and there are no changes in raid0.c between .10 and .11 anyway)
As I say on the webpage, a dd on the underlying devices works fine...
It might still be a higher level interface to the md, that's wrong.
And I'm not saying that it's the 3w-9xx driver!! It hasn't changed between 2.6.10 and 2.6.11, so pretty much impossible to introduce bugs there...
(It was just mentioned wrt to the fusion-mpt driver.)
Andreas
Quoted 4 lines
--david
RE: [Gelato-technical] Serious performance degradation on a RAID withkernel 2.6.11
In reply to David MosbergerAndreas Hirstius wrote on Wednesday, April 06, 2005 1:33 AM
Quoted 7 lines
The problems mentioned in this discussion about the Fusion-MPT I can perfectly reproduce on my boxes with 2.6.11, but only with pages sizes 4k and 8k;
With 16k and 64k pages I don't see any problem at all !!! Actually I see block size <-> page size correlations which are similiar to the correlations I see in my setup...
i.e. 2.6.11 and 4k pages:
Quoted 6 lines
dd ...bs@96: 40MB/s dd ...bs�92: 74MB/si.e. 2.6.11 and 8k pages:
dd ...bs�92: 56MB/s dd ...bs384: 74MB/s
It is very odd. I think it has something to do with generic file read ahead logic. The read ahead window did not push far enough to keep the disk busy. Some of the size calculation are in unit of page, so smaller page size will have smaller effective read ahead size. Try look into mm/readahead.c. I bet you can find the answer to your performance problem there. If not, then I guess we have to keep on looking ;-)