Slow I/O performance on SAS1064

DISCUSSION Markus 2 participants 5 messages Mar 5, 2014 – Mar 7, 2014 SCSI

lore

Share & tools

lore

Markus

author

2014-03-05 17:50 UTC #0 lore raw

I have problem with SATA disks at my Sunfire v245 and its lsi controller they are very slow.

So I test it with dd and while dd in a nother terminal i see dstat results of sda:

Copy from ramdisk to sda.
=== Test ext4 ===
DD

root@outpost:/ramdisk# dd if=debian-7.4.0-sparc-netinst.iso of=/root/test.dd
353344+0 records in
353344+0 records out
180912128 bytes (181 MB) copied, 24.6384 s, 7.3 MB/s

Dstat:

----total-cpu-usage---- --dsk/sda-- ---io/sda--

usr sys idl wai hiq siq| read  writ| read  writ
  1   0 100   0   0   0|   0     0 |   0     0 
  0   0 100   0   0   0|   0     0 |   0     0 
  2   1  97   0   0   0|   0     0 |   0     0 
  1   5  94   0   0   0|   0     0 |   0     0 
  7  43  50   0   0   0|   0     0 |   0     0 
  7  43  50   0   0   0|   0     0 |   0     0 
 10  40  50   0   0   0|   0     0 |   0     0 
 12  45  41   3   0   0|   0     0 |   0     0 
  6  42  36  17   0   1|   0   512k|   0  1.00 
  0   0   0  99   0   0|   0  4096k|   0  8.00 
  0   2   0  98   0   0|   0  5120k|   0  10.0 
  0   1   0  99   0   0|   0  7680k|   0  15.0 
  1   0   0 100   0   0|   0  5632k|   0  11.0 
  0   0   0  99   0   0|   0  7168k|   0  14.0 
  1   1   0  99   0   0|   0  6656k|   0  13.0 
  0   0   0 100   0   0|   0  6144k|   0  12.0 
  1   0   0 100   0   0|   0  7680k|   0  15.0 
  0   1  22  78   0   0|   0  4608k|   0  9.00 
  0   0  50  50   0   0|   0  3072k|   0  6.00 
  1   0  50  50   0   1|   0  7168k|   0  14.0 
  1   1  50  49   0   0|4096B 5120k|1.00  10.0 
  0   0  50  50   0   0|   0  8704k|   0  17.0 
  0   0  50  49   0   0|   0  7680k|   0  15.0 
  0   1  50  50   0   0|   0  7168k|   0  14.0 
  1   0  49  49   0   0|   0  7168k|   0  14.0 
  1   0  50  50   0   0|   0  7680k|   0  15.0 
  0   0  50  50   0   0|   0  7168k|   0  14.0 
  0   0  50  50   0   0|   0  2560k|   0  5.00 
  0   1  30  69   0   0|   0  3624k|   0  8.00 
  1   0   0  99   0   0|   0  4100k|   0  9.00 
  0   0   0 100   0   0|   0  5632k|   0  11.0 
  0   0   0  99   0   0|   0  3584k|   0  7.00 
  1   0   0 100   0   0|   0  4608k|   0  9.00 
  0   0   0  99   0   1|   0  5120k|   0  10.0 
  1   0   4  96   0   0|   0  2560k|   0  6.00 
  0   0  50  50   0   0|   0  3072k|   0  6.00 
  0   0  49  50   0   0|   0  5120k|   0  10.0 
  1   0  50  50   0   0|   0  5120k|   0  10.0 
  0   0  50  50   0   0|   0  4096k|   0  8.00 
  1   0  50  50   0   0|   0  5120k|   0  10.0 
  0   0  50  50   0   0|   0  5120k|   0  10.0 
  0   1  57  42   0   0|   0  1125k|   0  17.0 
  0   0 100   0   0   0|   0     0 |   0     0 
  1   0 100   0   0   0|   0     0 |   0     0 
  1   0 100   0   0   0|   0     0 |   0     0 
  0   0 100   0   0   0|   0     0 |   0     0 
  0   0 100   0   0   0|   0     0 |   0     0

-------------------------------------------------------------------------------
Information:
Kernel 3.13.5 (stable release)
Debian7 (stable)
hdparm v9.39

Controller
LSI SAS1064

	cat /proc/mpt/ioc0/info 
	ioc0:
	  ProductID = 0x2701 (LSISAS1064 A3)
	  FWVersion = 0x01080400
	  MsgVersion = 0x0105
	  FirstWhoInit = 0x00
	  EventState = 0x00
	  CurrentHostMfaHighAddr = 0x00000000
	  CurrentSenseBufferHighAddr = 0x00000000
	  MaxChainDepth = 0x60 frames
	  MinBlockSize = 0x20 bytes
	  RequestFrames @ 0xfffffc137f602800 (Dma @ 0x00000000c000a800)
	    {CurReqSz=128} x {CurReqDepth=511} = 65408 bytes ^= 0x10000
	    {MaxReqSz=128}   {MaxReqDepth=511}
	  Frames   @ 0xfffffc137f600000 (Dma @ 0x00000000c0008000)
	    {CurRepSz=80} x {CurRepDepth=128} = 10240 bytes ^= 0x2880
	    {MaxRepSz=0}   {MaxRepDepth=511}
	  MaxDevices = 63
	  MaxBuses = 1
	  PortNumber = 1 (of 1)

	cat /proc/mpt/version 
		mptlinux-3.04.20
		  Fusion MPT base driver
		  Fusion MPT SAS host driver
		  Fusion MPT ioctl driver

-------------------------------------------------------------------------------
Hdparm Information:
hdparm -I /dev/sda

/dev/sda:
SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ATA device, with non-removable media Standards:
Likely used: 1
Configuration:
Logical max current
cylinders 0 0
heads 0 0
sectors/track 0 0
--
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 0 MBytes
device size with M = 1000*1000: 0 MBytes
cache/buffer size = unknown
Capabilities:
IORDY not likely
Cannot perform double-word IO
R/W multiple sector transfer: not supported
DMA: not supported
PIO: pio0

-------------------------------------------------------------------------------
Moduls load:
lsmod | grep mpt

	mptctl                 27359  0 
	mptsas                 45861  5 
	mptscsih               25765  1 mptsas
	mptbase                75530  3 mptctl,mptsas,mptscsih
	scsi_transport_sas     21624  1 mptsas

-------------------------------------------------------------------------------
The hdparm result looks like there is somethink not right . There were no features supported but why ?

Greetings Markus

Mark Knecht

2014-03-05 18:21 UTC #1 lore raw

On Wed, Mar 5, 2014 at 9:50 AM, Markus <[email protected]> wrote:
<SNIP>

Quoted 1 line

The hdparm result looks like there is somethink not right . There were no features supported but why ?

Does the HDD have S.M.A.R.T. features? Possibly

smartctl -a /dev/sda

would provide some additional visibility?

- Mark

markus

author

2014-03-05 21:34 UTC #2 lore raw

Quoted 9 lines · Mark Knecht

On Wed, Mar 5, 2014 at 9:50 AM, Markus <[email protected]> wrote:
<SNIP>
Quoted 1 line
The hdparm result looks like there is somethink not right . There were no features supported but why ?
Does the HDD have S.M.A.R.T. features? Possibly
smartctl -a /dev/sda

root@outpost:~# smartctl -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Probable ATA device behind a SAT layer
Try an additional '-d ata' or '-d sat' argument.

so i try it with scsi.

root@outpost:~# smartctl -d scsi -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Logical Unit id:      0x50000394e2380537
Serial number:                   7365P4KNT
Device type:          disk
Local Time is:        Wed Mar  5 22:32:09 2014 CET

Device supports SMART and is Disabled
Temperature Warning Disabled or Not Supported
SMART Health Status: OK

Error Counter logging not supported
Device does not support Self Test logging

Quoted 2 lines

would provide some additional visibility?

.....

[   98.037209] Uniform Multi-Platform E-IDE driver
[   98.107564] SLAB: Unable to allocate memory on node 0 (gfp=0x40d1)
[   98.188985]   cache: dma-kmalloc-512, object size: 512, order: 0
[   98.268127]   node 0: slabs: 0/0, objs: 0/0, free: 0
[   98.333538] sr: out of memory.
[   98.373628] cdrom: Uniform CD-ROM driver Revision: 3.20
[   98.443390] sr 0:0:0:0: Attached scsi CD-ROM sr0
[   98.448375] sr 0:0:0:0: Attached scsi generic sg0 type 5
[  103.198705] scsi2 : ioc0: LSISAS1064 A3, FwRev=01080400h, Ports=1, MaxQ=511, IRQ=10
[  103.329805] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 0, phy 0, sas_addr 0x1221000000000000
[  103.462762] scsi 2:0:0:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  103.571610] scsi 2:0:0:0: Attached scsi generic sg1 type 0
[  103.645418] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 1, phy 1, sas_addr 0x1221000001000000
[  103.777635] scsi 2:0:1:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  103.886607] scsi 2:0:1:0: Attached scsi generic sg2 type 0
[  103.960517] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 2, phy 2, sas_addr 0x1221000002000000
[  104.093109] scsi 2:0:2:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  104.201973] scsi 2:0:2:0: Attached scsi generic sg3 type 0
[  104.275830] mptsas: ioc0: attaching sata device: fw_channel 0, fw_id 3, phy 3, sas_addr 0x1221000003000000
[  104.408412] scsi 2:0:3:0: Direct-Access     ATA      TOSHIBA MQ01ABB2 0U   PQ: 0 ANSI: 5
[  104.517379] scsi 2:0:3:0: Attached scsi generic sg4 type 0
[  104.618997] sd 2:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.721185] sd 2:0:3:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.823771] sd 2:0:1:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.824017] sd 2:0:2:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[  104.828691] sd 2:0:0:0: [sda] Write Protect is off
[  104.828697] sd 2:0:0:0: [sda] Mode Sense: 67 00 00 08
[  104.829940] sd 2:0:2:0: [sdc] Write Protect is off
[  104.829945] sd 2:0:2:0: [sdc] Mode Sense: 67 00 00 08
[  104.830488] sd 2:0:3:0: [sdd] Write Protect is off
[  104.830493] sd 2:0:3:0: [sdd] Mode Sense: 67 00 00 08
[  104.833225] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  104.834471] sd 2:0:2:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  104.835118] sd 2:0:3:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  105.583175] sd 2:0:1:0: [sdb] Write Protect is off
[  105.623801]  sda: sda1 sda2 sda3
[  105.651118]  sdc: sdc1 sdc9
[  105.654024]  sdd: sdd1 sdd9
[  105.761966] sd 2:0:1:0: [sdb] Mode Sense: 67 00 00 08
[  105.769029] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  105.775293] sd 2:0:2:0: [sdc] Attached SCSI disk
[  105.775941] sd 2:0:3:0: [sdd] Attached SCSI disk
[  106.014357] sd 2:0:0:0: [sda] Attached SCSI disk
[  106.158864]  sdb: sdb1 sdb9
[  106.205879] sd 2:0:1:0: [sdb] Attached SCSI disk
[  106.502387] random: nonblocking pool is initialized
[  106.792978] device-mapper: uevent: version 1.0.3
[  106.861434] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: [email protected]
[  108.085062] bio: create slab <bio-1> at 1

.....

Quoted 2 lines

- Mark

Markus

Mark Knecht

2014-03-06 20:51 UTC #3 lore raw

Quoted 30 lines · markus

Quoted 9 lines · Mark Knecht
On Wed, Mar 5, 2014 at 9:50 AM, Markus <[email protected]> wrote:
<SNIP>
Quoted 1 line
The hdparm result looks like there is somethink not right . There were no features supported but why ?
Does the HDD have S.M.A.R.T. features? Possibly
smartctl -a /dev/sda
root@outpost:~# smartctl -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
Probable ATA device behind a SAT layer
Try an additional '-d ata' or '-d sat' argument.
so i try it with scsi.
root@outpost:~# smartctl -d scsi -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Logical Unit id:      0x50000394e2380537
Serial number:                   7365P4KNT
Device type:          disk
Local Time is:        Wed Mar  5 22:32:09 2014 CET

Quoted 6 lines

Device supports SMART and is Disabled
Temperature Warning Disabled or Not Supported
SMART Health Status: OK
Error Counter logging not supported
Device does not support Self Test logging

I am not familiar with this message:

"Device supports SMART and is Disabled"

but there are lots of posts out there with folks asking about it. Possibly you can figure out what's causing smart not to turn on, get it turned on and get some info.

Sorry I cannot help more.

Good luck,
Mark

markus

author

2014-03-07 09:40 UTC #4 lore raw

Quoted 11 lines · Mark Knecht

Quoted 3 lines · markus
Quoted 2 lines · Mark Knecht
On Wed, Mar 5, 2014 at 9:50 AM, Markus <[email protected]> wrote:
<SNIP>
I am not familiar with this message:
"Device supports SMART and is Disabled"
but there are lots of posts out there with folks asking about it. Possibly you can figure out what's causing smart not to turn on, get it turned on and get some info.

right , thats why i writing to the scsi and lsi devs , because die disk supports smart and lots of other features, but the controller driver failes to enable support for this.

Now i get that the mpt driver provides /dev/sg[0-1]

root@outpost:~# smartctl -d scsi -x /dev/sg1
smartctl 6.2 2013-07-26 r3841 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Logical Unit id:      0x50000394e2380537
Serial number:        7365P4KNT
Device type:          disk
Local Time is:        Fri Mar  7 09:54:47 2014 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Disabled
Temperature Warning:  Disabled or Not Supported
Read Cache is:        Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Error Counter logging not supported

Device does not support Self Test logging
Device does not support Background scan results logging scsiPrintSasPhy Log Sense Failed [unsupported field in scsi command]

so i had read those posts and i tried but it won't work.

root@outpost:~/lsiutil# smartctl -r scsiioctl,3 -s on -d scsi /dev/sg1
smartctl 6.2 2013-07-26 r3841 [sparc64-linux-3.13.5-mar] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

[inquiry: 12 01 00 01 fc 00 ]
scsi_status=0x0, host_status=0x0, driver_status=0x0 info=0x0 duration=0 milliseconds resid=500 Incoming data, len=8:
00 00 00 00 04 00 80 83 89
[inquiry: 12 00 00 00 24 00 ]
scsi_status=0x0, host_status=0x0, driver_status=0x0 info=0x0 duration=4 milliseconds resid=0 Incoming data, len=36:

 00     00 00 05 12 45 00 00 02  41 54 41 20 20 20 20 20                        
 10     54 4f 53 48 49 42 41 20  4d 51 30 31 41 42 42 32                        
 20     30 55 20 20

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
[mode sense(6): 1a 00 1c 00 40 00 ]
scsi_status=0x0, host_status=0x0, driver_status=0x0 info=0x0 duration=4 milliseconds resid=40 Incoming data, len=24:
00 17 00 00 08 00 00 00 00 00 00 02 00 1c 0a 08 06
10 08 00 00 00 00 00 00 00
[mode sense(6): 1a 00 5c 00 40 00 ]
scsi_status=0x0, host_status=0x0, driver_status=0x0 info=0x0 duration=0 milliseconds resid=40 Incoming data, len=24:

 00     17 00 00 08 00 00 00 00  00 00 02 00 1c 0a 00 00                        
 10     00 00 00 00 00 00 00 00                                                 
 00     00 00 00 08 00 00 00 00  00 00 02 00 1c 0a 08 06                        
 10     08 00 00 00 00 00 00 00

[mode select(6): 15 10 00 00 18 00 ]
Outgoing data, len=24:
scsi_status=0x2, host_status=0x0, driver_status=0x8 info=0x1 duration=0 milliseconds resid=24

Quoted 1 line

Quoted 1 line
Quoted 1 line
Sense buffer, len=18:

00 70 00 0b 00 00 00 00 0a 00 00 00 00 00 00 00 00
10 00 00
status=2: sense_key=b asc=0 ascq=0
unable to enable Exception control and warning [aborted command]

Quoted 5 lines

Sorry I cannot help more.
Good luck,
Mark