Skip to contentSkip to conversation

Performance issues with Areca 1680 SAS Controllers

DISCUSSION Michael Fuckner 6 participants 13 messages Nov 19, 2008 – Aug 20, 2009 2 messages not in the archive SCSI

lore
Share & tools
lore

Michael Fuckner

author
#0 lore raw

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

After doing some IO, the areca raidset is slower compared to the rate directly after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k iflag=direct 51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k iflag=direct 51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and 12XX)

Regards,
Michael!

#1 lore raw

(cc's added)

Quoted 35 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k iflag=direct 51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k iflag=direct 51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and 12XX)

Regards,
Michael!

#2 lore raw

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 3 lines

Regards,
Michael!

#3 lore raw
Quoted 4 lines · Nick Cheng

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,

Was this issue resolved? I'm seeing similar behaviour.. I think it's related to flushing the controller cache?

Waiting a while (after the disk leds stop blinking) performance is back to normal..

-- Pasi

Quoted 57 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 3 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 24 lines · Wang Jinpu

???????? Pasi K?rkk?inen
?????????? 2009-08-19 22:34:28
???????? Nick Cheng
?????? 'Andrew Morton'; 'Michael Fuckner'; linux-kernel; linux-scsi; 'Erich Chen' ?????? Re: Performance issues with Areca 1680 SAS Controllers

Quoted 4 lines · Nick Cheng

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,
Was this issue resolved? I'm seeing similar behaviour.. I think it's related to flushing the controller cache?
Waiting a while (after the disk leds stop blinking) performance is back to normal..
-- Pasi

Quoted 2 lines

In my memory ??It's because the write cache is not enable??You can verify it using sdparm

I'm using battery backed write-back write cache on Areca controller, and it's enabled. Or did you mean caches in each harddisk?

More information about my setup:

- Areca 1680 24-port SAS RAID controller
- Physical disks 1-4 are raid-10 for the OS (boot volume, /dev/sda)
- Physical disks 5-18 are raid-60 for the data (/dev/sdb)

I downloaded ltp-base-20090731.tgz from http://ltp.sf.net and compiled it.

from testcases/kernel/io/disktest/ I run:

# echo 3 > /proc/sys/vm/drop_caches
# ./disktest -w -K16 -B 4k -T 60 -pr -Ibd -PA /dev/sdb

(sdb is the raid-60 data array).

So the disktest benchmark is running 4 kB random writes using 16 threads and direct-IO (bypassing kernel caches) for 60 seconds.

The test completes after 60 seconds, just like it should, but the disk leds will keep blinking for many minutes more..

So basicly Areca controller is first caching the random write IOs and then flushing them from the cache (2 GB) to the disks..

The problem is while Areca is doing the flushing _all_ IOs are really slow, including the other raid-10 array for the OS, which uses totally separate physical disks.

Opening another shell in screen takes at least 30 seconds, starting "top" takes forever etc..

While Areca is flushing the caches (and all the IOs are slow), "iostat 1" doesn't show any "leftover" IOs from the benchmark. So the benchmark was really using direct IO, bypassing kernel caches.

I tried with different io-schedulers (cfq,deadline,noop) but they didn't have big effect.. which makes sense, since the OS/kernel is not doing any big IO when the 'stalling' happens.

Is there some way to make Areca NOT use all cpu-power for cache flushing?

OS is CentOS 5.3 x86_64 with linux-2.6.18-128 kernel.

-- Pasi

Quoted 69 lines

2009-08-19

Wang Jinpu

Quoted 49 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4347 (20090819) __________ The message was checked by ESET NOD32 Antivirus.
http://www.eset.com

#6 lore raw
Quoted 12 lines · Nick Cheng
Quoted 4 lines

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,
Was this issue resolved? I'm seeing similar behaviour.. I think it's related to flushing the controller cache?
Waiting a while (after the disk leds stop blinking) performance is back to normal..
-- Pasi

Quoted 2 lines

In my memory ??It's because the write cache is not enable??You can verify it using sdparm

I'm using battery backed write-back write cache on Areca controller, and it's enabled. Or did you mean caches in each harddisk?

[Jack] Yes , I mean the write cache of each HD.

More information about my setup:

- Areca 1680 24-port SAS RAID controller
- Physical disks 1-4 are raid-10 for the OS (boot volume, /dev/sda)
- Physical disks 5-18 are raid-60 for the data (/dev/sdb)

I downloaded ltp-base-20090731.tgz from http://ltp.sf.net and compiled it.

from testcases/kernel/io/disktest/ I run:

# echo 3 > /proc/sys/vm/drop_caches
# ./disktest -w -K16 -B 4k -T 60 -pr -Ibd -PA /dev/sdb

(sdb is the raid-60 data array).

So the disktest benchmark is running 4 kB random writes using 16 threads and direct-IO (bypassing kernel caches) for 60 seconds.

The test completes after 60 seconds, just like it should, but the disk leds will keep blinking for many minutes more..

So basicly Areca controller is first caching the random write IOs and then flushing them from the cache (2 GB) to the disks..

The problem is while Areca is doing the flushing _all_ IOs are really slow, including the other raid-10 array for the OS, which uses totally separate physical disks.

Opening another shell in screen takes at least 30 seconds, starting "top" takes forever etc..

While Areca is flushing the caches (and all the IOs are slow), "iostat 1" doesn't show any "leftover" IOs from the benchmark. So the benchmark was really using direct IO, bypassing kernel caches.

I tried with different io-schedulers (cfq,deadline,noop) but they didn't have big effect.. which makes sense, since the OS/kernel is not doing any big IO when the 'stalling' happens.

Is there some way to make Areca NOT use all cpu-power for cache flushing?

OS is CentOS 5.3 x86_64 with linux-2.6.18-128 kernel.

-- Pasi

Quoted 69 lines

2009-08-19

Wang Jinpu

Quoted 49 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
__________ Information from ESET NOD32 Antivirus, version of virus signature database 4347 (20090819) __________ The message was checked by ESET NOD32 Antivirus.
http://www.eset.com

#7 lore raw
In reply to Pasi Kärkkäinen
Quoted 10 lines

The problem is while Areca is doing the flushing _all_ IOs are really slow, including the other raid-10 array for the OS, which uses totally separate physical disks.

Opening another shell in screen takes at least 30 seconds, starting "top" takes forever etc..

While Areca is flushing the caches (and all the IOs are slow), "iostat 1" doesn't show any "leftover" IOs from the benchmark. So the benchmark was really using direct IO, bypassing kernel caches.

Quoted 5 lines

I tried with different io-schedulers (cfq,deadline,noop) but they didn't have big effect.. which makes sense, since the OS/kernel is not doing any big IO when the 'stalling' happens.

Is there some way to make Areca NOT use all cpu-power for cache flushing?

Is top showing 100% cpu usage?

It wouldn't surprise me if the Areca's internal bus / processor is getting swamped. With the RAID60 chewing through two PQ calculations over 14 disks I'd expect the performance of other arrays on the controller would take a hit

I've seen similar symptoms when moving massive amounts of data between disks on my servers. IO maxes out the bus, system responsiveness goes down as evidenced by slowly loading apps, and top still shows minimal CPU usage.

Signature · 4 lines
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
#8 lore raw
Quoted 10 lines · Drew
Quoted 10 lines

The problem is while Areca is doing the flushing _all_ IOs are really slow, including the other raid-10 array for the OS, which uses totally separate physical disks.

Opening another shell in screen takes at least 30 seconds, starting "top" takes forever etc..

While Areca is flushing the caches (and all the IOs are slow), "iostat 1" doesn't show any "leftover" IOs from the benchmark. So the benchmark was really using direct IO, bypassing kernel caches.

Quoted 8 lines
Quoted 5 lines

I tried with different io-schedulers (cfq,deadline,noop) but they didn't have big effect.. which makes sense, since the OS/kernel is not doing any big IO when the 'stalling' happens.

Is there some way to make Areca NOT use all cpu-power for cache flushing?

Is top showing 100% cpu usage?

No, cpu usage is almost 0%. I was talking about cpu-usage of _Areca_ controller.

When the disktest-benchmark ends, Linux cpu usage goes to 0%, and there's no IO anymore (checked with "iostat 1").

But _Areca_ is still flushing it's caches, and all the IO will be slow, even when there's no load in _Linux_.

-- Pasi

Quoted 16 lines

It wouldn't surprise me if the Areca's internal bus / processor is getting swamped. With the RAID60 chewing through two PQ calculations over 14 disks I'd expect the performance of other arrays on the controller would take a hit

I've seen similar symptoms when moving massive amounts of data between disks on my servers. IO maxes out the bus, system responsiveness goes down as evidenced by slowly loading apps, and top still shows minimal CPU usage.

Signature · 4 lines
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw
Quoted 7 lines · Nick Cheng

Hi Gentlemen,
We have worked it out with new firmware and driver.
You can download the latest F/W from ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1680/ and use the attached driver.
Please have a try.
Any comments are welcome.

Thanks! I'll give it a try today and report back.

-- Pasi

Quoted 17 lines

Thank you,
-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Wednesday, August 19, 2009 10:09 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]; 'Erich Chen'
Subject: Re: Performance issues with Areca 1680 SAS Controllers

Quoted 4 lines · Nick Cheng

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,

Quoted 65 lines

Was this issue resolved? I'm seeing similar behaviour.. I think it's related to flushing the controller cache?

Waiting a while (after the disk leds stop blinking) performance is back to normal..

-- Pasi

Quoted 57 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 3 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
Quoted 11 lines · Pasi Kärkkäinen
Quoted 7 lines · Nick Cheng

Hi Gentlemen,
We have worked it out with new firmware and driver.
You can download the latest F/W from ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1680/ and use the attached driver.
Please have a try.
Any comments are welcome.

Thanks! I'll give it a try today and report back.

v1.47 firmware seems to fix this problem! The system behaves _much_ better now even when Areca is busy flushing it's cache!

I didn't update the driver, only Areca firmware.

Is there something imported fixed in the latest driver? I'm using the default driver included in RHEL/CentOS 5.3.

# modinfo arcmsr.ko
filename:       arcmsr.ko
version:        Driver Version 1.20.00.15.RH1 2008/02/27

-- Pasi

Quoted 90 lines
Quoted 18 lines

Thank you,
-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Wednesday, August 19, 2009 10:09 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]; 'Erich Chen'
Subject: Re: Performance issues with Areca 1680 SAS Controllers

Quoted 4 lines · Nick Cheng

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,

Quoted 65 lines

Was this issue resolved? I'm seeing similar behaviour.. I think it's related to flushing the controller cache?

Waiting a while (after the disk leds stop blinking) performance is back to normal..

-- Pasi

Quoted 57 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 3 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#12 lore raw

It could improve the performance with F/W v1.47.

-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Thursday, August 20, 2009 8:16 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]
Subject: Re: Performance issues with Areca 1680 SAS Controllers

Quoted 5 lines · Pasi Kärkkäinen
Quoted 4 lines · Nick Cheng

Hi Gentlemen,
We have worked it out with new firmware and driver.
You can download the latest F/W from ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1680/ and use the

attached

Quoted 6 lines
Quoted 3 lines

driver.
Please have a try.
Any comments are welcome.

Thanks! I'll give it a try today and report back.

v1.47 firmware seems to fix this problem! The system behaves _much_ better now even when Areca is busy flushing it's cache!

I didn't update the driver, only Areca firmware.

Is there something imported fixed in the latest driver? I'm using the default driver included in RHEL/CentOS 5.3.

# modinfo arcmsr.ko
filename:       arcmsr.ko
version:        Driver Version 1.20.00.15.RH1 2008/02/27

-- Pasi

Quoted 11 lines
Quoted 11 lines

Thank you,
-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Wednesday, August 19, 2009 10:09 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]; 'Erich Chen'
Subject: Re: Performance issues with Areca 1680 SAS Controllers

On Thu, Nov 20, 2008 at 03:39:40PM +0800, Nick Cheng wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,

Was this issue resolved? I'm seeing similar behaviour.. I think it's

related

Quoted 3 lines
Quoted 3 lines

to flushing the controller cache?

Waiting a while (after the disk leds stop blinking) performance is back

to

Quoted 9 lines
Quoted 9 lines

normal..

-- Pasi

Quoted 5 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick

Cheng;

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it

with

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

After doing some IO, the areca raidset is slower compared to the

rate

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

directly
after boot.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 6 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8

-+r

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

-o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 5 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11,

Debian

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the

high

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

rate? Is this reproducible foer Areca SATA Controllers (Type 11XX

and

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

12XX)

Quoted 3 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe

linux-kernel" in

Quoted 10 lines
Quoted 3 lines
Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw
Quoted 2 lines · Nick Cheng

It could improve the performance with F/W v1.47.

Ok. Thanks!

-- Pasi

Quoted 137 lines

-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Thursday, August 20, 2009 8:16 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]
Subject: Re: Performance issues with Areca 1680 SAS Controllers

Quoted 5 lines · Pasi Kärkkäinen
Quoted 4 lines · Nick Cheng

Hi Gentlemen,
We have worked it out with new firmware and driver.
You can download the latest F/W from ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1680/ and use the

attached

Quoted 6 lines
Quoted 3 lines

driver.
Please have a try.
Any comments are welcome.

Thanks! I'll give it a try today and report back.

v1.47 firmware seems to fix this problem! The system behaves _much_ better now even when Areca is busy flushing it's cache!

I didn't update the driver, only Areca firmware.

Is there something imported fixed in the latest driver? I'm using the default driver included in RHEL/CentOS 5.3.

# modinfo arcmsr.ko
filename:       arcmsr.ko
version:        Driver Version 1.20.00.15.RH1 2008/02/27

-- Pasi

Quoted 11 lines
Quoted 11 lines

Thank you,
-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Wednesday, August 19, 2009 10:09 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]; 'Erich Chen'
Subject: Re: Performance issues with Areca 1680 SAS Controllers

On Thu, Nov 20, 2008 at 03:39:40PM +0800, Nick Cheng wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,

Was this issue resolved? I'm seeing similar behaviour.. I think it's

related

Quoted 3 lines
Quoted 3 lines

to flushing the controller cache?

Waiting a while (after the disk leds stop blinking) performance is back

to

Quoted 9 lines
Quoted 9 lines

normal..

-- Pasi

Quoted 5 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick

Cheng;

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it

with

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

After doing some IO, the areca raidset is slower compared to the

rate

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

directly
after boot.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 6 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8

-+r

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

-o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 5 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11,

Debian

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the

high

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

rate? Is this reproducible foer Areca SATA Controllers (Type 11XX

and

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

12XX)

Quoted 3 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe

linux-kernel" in

Quoted 10 lines
Quoted 3 lines
Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw
In reply to Pasi Kärkkäinen
Quoted 13 lines · Pasi Kärkkäinen
Quoted 11 lines · Pasi Kärkkäinen
Quoted 7 lines · Nick Cheng

Hi Gentlemen,
We have worked it out with new firmware and driver.
You can download the latest F/W from ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1680/ and use the attached driver.
Please have a try.
Any comments are welcome.

Thanks! I'll give it a try today and report back.

Quoted 6 lines

v1.47 firmware seems to fix this problem! The system behaves _much_ better now even when Areca is busy flushing it's cache!

I didn't update the driver, only Areca firmware.

Is there something imported fixed in the latest driver? I'm using the

                     ^^^^^^^^

.. That should have been important :)

-- Pasi

Quoted 98 lines

default driver included in RHEL/CentOS 5.3.

# modinfo arcmsr.ko
filename:       arcmsr.ko
version:        Driver Version 1.20.00.15.RH1 2008/02/27

-- Pasi

Quoted 85 lines
Quoted 18 lines

Thank you,
-----Original Message-----
From: Pasi Kärkkäinen [mailto:[email protected]]
Sent: Wednesday, August 19, 2009 10:09 PM
To: Nick Cheng
Cc: 'Andrew Morton'; 'Michael Fuckner'; [email protected]; [email protected]; 'Erich Chen'
Subject: Re: Performance issues with Areca 1680 SAS Controllers

Quoted 4 lines · Nick Cheng

Hi Michael,
I will get around handling your issue.
Thanks for your kindly patience,

Hello,

Quoted 65 lines

Was this issue resolved? I'm seeing similar behaviour.. I think it's related to flushing the controller cache?

Waiting a while (after the disk leds stop blinking) performance is back to normal..

-- Pasi

Quoted 57 lines

-----Original Message-----
From: Andrew Morton [mailto:[email protected]]
Sent: Thursday, November 20, 2008 9:18 AM
To: Michael Fuckner
Cc: [email protected]; [email protected]; Nick Cheng; Erich Chen
Subject: Re: Performance issues with Areca 1680 SAS Controllers

(cc's added)

Quoted 7 lines · On Wed, 19 Nov 2008 13:16:56 +0100 Michael Fuckner <[email protected]> wrote:

Hi,

I am using an Areca 1680-SAS-Controller with 16 SAS-HDD (Seagate 1TB ST31000640SS). I set up a Raid6 with all 16 disks and formatted it with XFS. The Controller has 512MB RAM and a BBU. The OS is installed to another disk attached to the onboard AHCI controller.

Quoted 6 lines

After doing some IO, the areca raidset is slower compared to the rate directly
after boot.

[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 10 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 59.6494 seconds, 900 MB/s [root@storage ~]# mount /dev/sdb1 /data
[root@storage ~]# cd /data/
[root@storage data]# ./iozone -i 0 -i 1 -s 32g -r 16m -S 6144 -t 8 -+r -o >raid6_sync_t8.log
[root@storage data]# cd
[root@storage ~]# umount /data/
[root@storage ~]# dd if=/dev/sdb1 of=/dev/null bs=1M count=50k

iflag=direct

Quoted 9 lines

51200+0 records in
51200+0 records out
53687091200 bytes (54 GB) copied, 76.4036 seconds, 703 MB/s

I tested different Versions of Linux (Centos 5.2, OpenSUSE 11, Debian Lenny) and Vanilla kernels 2.6.22-2.6.27, all show this behaviour.

Idea why the device slows down after IO- or better: how to keep the high rate? Is this reproducible foer Areca SATA Controllers (Type 11XX and

12XX)

Quoted 3 lines

Regards,
Michael!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html