hot plug on ICH9 with AHCI on

Jeff!

I have some questions concerning HOTPLUG and RAID support for ICH9 chip used in my server.
I spent some time to know what is linux kernel support for enclosure LEDs in HDD bays. I found no such support however it is described into ICH9' datasheet. I am using linux debian with kernel 2.6.25. Latest kernel for debian is 2.6.26 and it also does not have any enclosure management support. However, I found some support in sources of kernel 2.6.28, however it seems only activity led is supported. That's why I wrote a little kernel module wich takes control over that ACHI functionality and tries to do enclosure management just behind traditional ahci driver. It works fine but I faced to one strange thing. Namely, I can control LEDs of those drives that are physically inserted into backplane. If the slot is empty since last reboot its LEDS cannot be controlled. I studied why this is so. An answer to this seems to be that ahci driver does not enable entire port of SATA controller if it is not populated during boot. To check this I tried some hot swap actions. First, I have extracted one of my spare drives and push it back. I got the following logs:
--
ata3: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen
ata3: irq_stat 0x00400040, connection status changed
ata3: SError: { HostInt PHYRdyChg 10B8B DevExch }
ata3: hard resetting link
ata3: SATA link down (SStatus 0 SControl 300)
ata3: failed to recover some devices, retrying in 5 secs
ata3: hard resetting link
ata3: SATA link down (SStatus 0 SControl 300)
ata3: failed to recover some devices, retrying in 5 secs
ata3: hard resetting link
ata3: SATA link down (SStatus 0 SControl 300)
ata3.00: disabled
ata3: EH complete
ata3.00: detaching (SCSI 2:0:0:0)
sd 2:0:0:0: [sdc] Synchronizing SCSI cache
sd 2:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
sd 2:0:0:0: [sdc] Stopping disk
sd 2:0:0:0: [sdc] START_STOP FAILED
sd 2:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
ahci em: 40: post command 00080002
ata3: exception Emask 0x10 SAct 0x0 SErr 0x4050002 action 0xa frozen
ata3: irq_stat 0x00400040, connection status changed
ata3: SError: { RecovComm PHYRdyChg CommWake DevExch }
ata3: hard resetting link
ata3: softreset failed (device not ready)
ata3: hard resetting link
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-7: ST3500320AS, SD04, max UDMA/133
ata3.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata3.00: configured for UDMA/133
ata3: EH complete
sd 2:0:0:0: [sdc] Attached SCSI disk
sd 2:0:0:0: Attached scsi generic sg2 type 0
--
This log is strange for me. It seems that system missed the point that the drives was going out. First it tried to reinitialize the SATA link for three times. Then, it tried to sync caches and stop the drive when it has actually lost connection with HBA. Then disk was returned to the slot and its softreset failed. Why? I suspect the drive did not fully start when the host tried to establish connection to it.

Another thing happened when I extracted the drive from one slot and pushed it back into its neigbor that was empty during linux boot up. Kernel desided this slot is dummy:
---
ahci 0000:00:1f.2: version 3.0
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 17 (level, low) -> IRQ 17
ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0xb impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf led clo pmp pio slum part
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m2048@0xd8601000 port 0xd8601100 irq 1275
ata2: SATA max UDMA/133 abar m2048@0xd8601000 port 0xd8601180 irq 1275
ata3: DUMMY
ata4: SATA max UDMA/133 abar m2048@0xd8601000 port 0xd8601280 irq 1275
ata5: DUMMY
ata6: DUMMY
---

So, even if I put the drive as ata3 device kernel does nothing to start it.

Now my questions:
1. Is it possible to force all ports to be potentially populated during startup. I would prefer that all ICH9 SATA ports will have their own fixed names, eg. /dev/sata0, ..., /dev/sata5. For now I have 3 drives and they allways get names /dev/sda /dev/sdb /dev/sdc even if there is some empty port as shown above. This is not convenient because enclosure management is linked to physical ports, not to only populated ones.
2. How can I remove SATA drive safely? I mean the behavior similar to USB drives removing. I'd like to notify the system that i wish to remove the drive. Then it performs some actions as closing all current connections, stopping new connections, flushing caches etc. After all that it updates indicators on backplane showing me that the drive is ready to be removed. As I see, some portions of this procedure can be done using hdparm -f -F -Y, but not all.

With best regards, Vladimir Dashevsky

Hello,

That's the intended behavior. Oh PHY event, libata EH tries to revive the link at least for 15 secs so that transient PHY glitch doesn't kill your root fs.

That's SCSI sd driver shutting down. As hot unplugging is surprise-removal, sd's shutdown sequence arrives after the device is actually gone and failed immediately.

Yeah, it sometimes depends on the spin up time. Sometimes some controllers just can't get things working for the first trial and so on. The timeout mechanism is there to achieve acceptable delay even when devices slightly malfunction, so the timeouts are a bit aggressive.

DUMMY ports are determined by the BIOS and dummy state is recorded in an ahci register. Does your board have all six ports exposed?

If you have exposed ports which are marked dummy by the ahci driver. It's a BIOS bug. It either needs to be quirked and reported to the motherboard vendor.

echo 1 > /sys/block/sdX/device/delete

Thanks.

tejun

Tejun!

First, thanks for your reply.
I want to inroduce my platform so you could get some info of it:
http://www.supermicro.com/products/system/1u/5015/sys-5015b-mt.cfm

Below ther are some comments from me.

Well, I partially agree. Surely, EMI problems should not break the link forever but I do not agree with the algorithm. When the drive is being removed it gets out during millisectonds. I mean the time between loss of link and detection that port is not populated. So, I can imagine that driver could be going to retry reset one but it had to abort this action once it got the drive is removed at all. Not in 15 second and even not in 5 seconds but in 0.01 second. So, I think log should be:
ata3: exception Emask 0x50 SAct 0x0 SErr 0x4090800 action 0xe frozen
ata3: irq_stat 0x00400040, connection status changed
ata3: SError: { HostInt PHYRdyChg 10B8B DevExch }
ata3: hard resetting link
ata3: SATA link down (SStatus 0 SControl 300)
ata3: drive is out
ata3.00: disabled
ata3: EH complete

Ok. So, this is notmal. We just need to inform SCSI driver first, isn't it?

Well, some drives store their firmware on disk, so they cannot work with host until fully spinned up. I heard that drive started to spin up in two or more seconds after being inserted. So, what is the indended driver behavior? It simply performs soft resets until drive answer ot this, isn't it? If the drive gets ready faster it will be fewer failed soft resets in log, right?

Yes. The board has ICH9 which supports how plug capability. And this is claimed by SuperMicro (its vendor).
I can say more. When I read datasheet on ICH9, I found that it has register named:
"14.1.31 PCS---Port Control and Status Register (SATA--D31:F2)"
As stated, it contains 6 port enables and 6 port present flags. First, similar to you, I thought that some ports were disabled by BIOS. Then I printed the contents of this register into my enclosure driver and saw that PCS is 8B3F. According to the datasheet that means that all 6 ports are enabled, but onlt 3 have connected links. If I reinstall the drive to neighbour slot I see the PCS changes to 873F, just according to the change. So, I suppose there is some AHCI driver bug. It should not assume, that port is dummy if it is enabled but not present.

See my argues above.

Can I be sure this will stop the drive sefely (without of cached data loss)?

With best regards, Vladimir Dashevsky

Tejun!

I will write some comment later. Now I send dmidecode output.

# dmidecode 2.9
SMBIOS 2.5 present.
38 structures occupying 1584 bytes.
Table at 0x7FEDF000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Phoenix Technologies LTD
Version: 6.00
Release Date: 02/25/2008
Address: 0xE2DA0
Runtime Size: 119392 bytes
ROM Size: 2048 kB
Characteristics:
ISA is supported
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
ESCD support is available
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 KB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) USB legacy is supported
Smart battery is supported
BIOS boot specification is supported Targeted content distribution is supported

Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Supermicro
Product Name: X7SBi
Version: 0123456789
Serial Number: 0123456789
UUID: 53D1A494-D663-A0E7-890B-00304862E142 Wake-up Type: Power Switch
SKU Number: Not Specified
Family: Not Specified

Handle 0x0002, DMI type 2, 8 bytes Base Board Information
Manufacturer: Supermicro Product Name: X7SBi
Version: PCB Version Serial Number: 0123456789

Handle 0x0003, DMI type 3, 17 bytes Chassis Information
Manufacturer: Supermicro
Type: Other
Lock: Not Present
Version: 0123456789 Serial Number: 0123456789 Asset Tag:
Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: None OEM Information: 0x00001234

Handle 0x0004, DMI type 4, 35 bytes Processor Information
Socket Designation: CPU 1
Type: Central Processor
Family: Unknown
Manufacturer: Intel
ID: FD 06 00 00 FF FB EB BF
Version: 00000000000000000000000000000000
Voltage: 1.8 V
External Clock: Unknown
Max Speed: 3300 MHz
Current Speed: 2200 MHz
Status: Populated, Enabled
Upgrade: Socket LGA775
L1 Cache Handle: 0x0005
L2 Cache Handle: 0x0006
L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified
Part Number: Not Specified

Handle 0x0005, DMI type 7, 19 bytes
Cache Information
Socket Designation: L1 Cache
Configuration: Enabled, Socketed, Level 1 Operational Mode: Write Back
Location: Internal
Installed Size: 16 KB
Maximum Size: 16 KB
Supported SRAM Types:
Burst
Pipeline Burst Asynchronous
Installed SRAM Type: Asynchronous
Speed: Unknown
Error Correction Type: Unknown System Type: Unknown
Associativity: Unknown

Handle 0x0006, DMI type 7, 19 bytes
Cache Information
Socket Designation: L2 Cache
Configuration: Enabled, Socketed, Level 2 Operational Mode: Write Back
Location: Internal
Installed Size: 2048 KB
Maximum Size: 512 KB
Supported SRAM Types:
Burst
Pipeline Burst Asynchronous
Installed SRAM Type: Burst
Speed: Unknown
Error Correction Type: Unknown System Type: Unknown
Associativity: Unknown

Handle 0x0007, DMI type 7, 19 bytes
Cache Information
Socket Designation: L3 Cache
Configuration: Enabled, Socketed, Level 3 Operational Mode: Write Back
Location: Internal
Installed Size: 2048 KB
Maximum Size: 512 KB
Supported SRAM Types:
Burst
Pipeline Burst Asynchronous
Installed SRAM Type: Burst
Speed: Unknown
Error Correction Type: Unknown System Type: Unknown
Associativity: Unknown

Handle 0x0008, DMI type 8, 9 bytes
Port Connector Information
Internal Reference Designator: J2A1
Internal Connector Type: 9 Pin Dual Inline (pin 10 cut) External Reference Designator: COM 1
External Connector Type: DB-9 male
Port Type: Serial Port 16550A Compatible

Handle 0x0009, DMI type 8, 9 bytes
Port Connector Information
Internal Reference Designator: J3A1
Internal Connector Type: 25 Pin Dual Inline (pin 26 cut) External Reference Designator: Parallel External Connector Type: DB-25 female
Port Type: Parallel Port ECP/EPP

Handle 0x000A, DMI type 8, 9 bytes
Port Connector Information
Internal Reference Designator: J1A1 Internal Connector Type: None External Reference Designator: Keyboard External Connector Type: Circular DIN-8 male Port Type: Keyboard Port

Handle 0x000B, DMI type 8, 9 bytes
Port Connector Information
Internal Reference Designator: J1A1 Internal Connector Type: None External Reference Designator: PS/2 Mouse External Connector Type: Circular DIN-8 male Port Type: Keyboard Port

Handle 0x000C, DMI type 9, 13 bytes System Slot Information
Designation: PCI Slot #1 - J5B1
Type: 32-bit PCI
Current Usage: Available
Length: Long
ID: 1
Characteristics:
5.0 V is provided 3.3 V is provided

Handle 0x000D, DMI type 9, 13 bytes System Slot Information
Designation: PCIe Slot #1 - J7C1
Type: 32-bit PCI Express Current Usage: In Use
Length: Long
ID: 4
Characteristics:
5.0 V is provided 3.3 V is provided

Handle 0x000E, DMI type 10, 6 bytes On Board Device Information
Type: Sound
Status: Disabled
Description: ADI1886

Handle 0x000F, DMI type 11, 5 bytes OEM Strings
String 1: This is the Intel 965 String 2: Customer Reference Board

Handle 0x0010, DMI type 12, 5 bytes
System Configuration Options
Option 1: Jumper settings can be described here.

Handle 0x0011, DMI type 15, 29 bytes
System Event Log
Area Length: 176 bytes
Header Start Offset: 0x0000
Header Length: 16 bytes
Data Start Offset: 0x0010
Access Method: General-purpose non-volatile data functions Access Address: 0x0000
Status: Valid, Not Full
Change Token: 0x00000086
Header Format: Type 1
Supported Log Type Descriptors: 3
Descriptor 1: POST error
Data Format 1: POST results bitmap
Descriptor 2: Single-bit ECC memory error
Data Format 2: Multiple-event
Descriptor 3: Multi-bit ECC memory error
Data Format 3: Multiple-event

Handle 0x0012, DMI type 16, 15 bytes Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None Maximum Capacity: 8 GB
Error Information Handle: Not Provided Number Of Devices: 4

Handle 0x0013, DMI type 17, 27 bytes Memory Device

	Array Handle: 0x0012
	Error Information Handle: 0xFF01
	Total Width: 40968 bits
	Data Width: 41024 bits
	Size: 1024 MB
	Form Factor: DIMM
	Set: 1
	Locator: J6G1
	Bank Locator: DIMM 0
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz (1.5 ns)
	Manufacturer: Kingston                                        
	Serial Number: D132F5F5
	Asset Tag: 00000730
	Part Number: 393930353331362D3030352E4130344C4600

Handle 0x0014, DMI type 17, 27 bytes Memory Device

	Array Handle: 0x0012
	Error Information Handle: 0xFF01
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: 1
	Locator: J6G2
	Bank Locator: DIMM 1
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz (1.5 ns)
	Manufacturer: 48spaces                                        
	Serial Number: 01234567
	Asset Tag: 01234567
	Part Number: 012345678901234567890123456789012345

Handle 0x0015, DMI type 17, 27 bytes Memory Device

	Array Handle: 0x0012
	Error Information Handle: 0xFF01
	Total Width: 41992 bits
	Data Width: 42048 bits
	Size: 1024 MB
	Form Factor: DIMM
	Set: 1
	Locator: J6H1
	Bank Locator: DIMM 2
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz (1.5 ns)
	Manufacturer: Kingston                                        
	Serial Number: D0321BF6
	Asset Tag: 00000730
	Part Number: 393930353331362D3030352E4130344C4600

Handle 0x0016, DMI type 17, 27 bytes Memory Device

	Array Handle: 0x0012
	Error Information Handle: 0xFF01
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: 1
	Locator: J6H2
	Bank Locator: DIMM 3
	Type: DDR2
	Type Detail: Synchronous
	Speed: 667 MHz (1.5 ns)
	Manufacturer: 48spaces                                        
	Serial Number: 01234567
	Asset Tag: 01234567
	Part Number: 012345678901234567890123456789012345

Handle 0x0017, DMI type 19, 15 bytes Memory Array Mapped Address

	Starting Address: 0x00000000000
	Ending Address: 0x0007FFFFFFF
	Range Size: 2 GB
	Physical Array Handle: 0x0012
	Partition Width: 0

Handle 0x0018, DMI type 20, 19 bytes Memory Device Mapped Address

	Starting Address: 0x00000000000
	Ending Address: 0x0003FFFFFFF
	Range Size: 1 GB
	Physical Device Handle: 0x0013
	Memory Array Mapped Address Handle: 0x0017
	Partition Row Position: Unknown
	Interleave Position: Unknown
	Interleaved Data Depth: Unknown

Handle 0x0019, DMI type 20, 19 bytes Memory Device Mapped Address

	Starting Address: 0x0003FFFFC00
	Ending Address: 0x0003FFFFFFF
	Range Size: 1 kB
	Physical Device Handle: 0x0014
	Memory Array Mapped Address Handle: 0x0017
	Partition Row Position: Unknown
	Interleave Position: Unknown
	Interleaved Data Depth: Unknown

Handle 0x001A, DMI type 23, 13 bytes System Reset
Status: Enabled
Watchdog Timer: Present
Boot Option: Do Not Reboot Boot Option On Limit: Do Not Reboot Reset Count: Unknown
Reset Limit: Unknown
Timer Interval: Unknown
Timeout: Unknown

Handle 0x001B, DMI type 24, 5 bytes Hardware Security
Power-On Password Status: Disabled Keyboard Password Status: Unknown Administrator Password Status: Enabled Front Panel Reset Status: Unknown

Handle 0x001C, DMI type 25, 9 bytes

	System Power Controls
	Next Scheduled Power-on: 12-31 23:59:59

Handle 0x001D, DMI type 26, 20 bytes Voltage Probe
Description: Voltage Probe
Location: Processor
Status: OK
Maximum Value: Unknown Minimum Value: Unknown
Resolution: Unknown
Tolerance: Unknown
Accuracy: Unknown OEM-specific Information: 0x00000000

Handle 0x001E, DMI type 27, 12 bytes Cooling Device

	Temperature Probe Handle: 0x001F
	Type: Fan
	Status: OK
	OEM-specific Information: 0x00000000

Handle 0x001F, DMI type 28, 20 bytes Temperature Probe
Description: Temperature Probe
Location: Processor
Status: OK
Maximum Value: Unknown Minimum Value Unknown
Resolution: Unknown
Tolerance: Unknown
Accuracy: Unknown OEM-specific Information: 0x00000000

Handle 0x0020, DMI type 29, 20 bytes Electrical Current Probe
Description: Electrical Current Probe
Location: Processor
Status: OK
Maximum Value: Unknown
Minimum Value: Unknown
Resolution: Unknown
Tolerance: Unknown
Accuracy: Unknown
OEM-specific Information: 0x00000000

Handle 0x0021, DMI type 30, 6 bytes Out-of-band Remote Access
Manufacturer Name: Intel Inbound Connection: Enabled Outbound Connection: Disabled

Handle 0x0022, DMI type 32, 20 bytes System Boot Information
Status: <OUT OF SPEC>

Handle 0x0023, DMI type 126, 4 bytes Inactive

Handle 0x0024, DMI type 127, 4 bytes End Of Table

Handle 0x0025, DMI type 127, 4 bytes End Of Table

Tejun!

Here are my comments on our previous conversation

I think you are not right here. If we are talking about EMI problems I can say that the strategy of many retries is worse than one read of port present status. EMI noise occurs at the time while software tries to re-establish connection with empty link because there is no link terminatiion. It's just like a car engine that has lost its muffler. It produces lots of noise. It is better to turn the link off as soon as we know there is no device on the port. That's why retries should last only until that state is reported by hardware. And I think hardware reports that state much faster than in 15 or even 5 seconds. In ICH9 it reports this just immediately.

Well, there are some strange comments in FAQs that older ICH chips (5 to 8) do not fully support hot plug, while the newer ones have better support for this. Then there is an explanation of what is proper hotplug support. That explanation says of surprise-removal support. However, I do not mind of notifying system before a actually remove the drive. Just because it may take a long time to shut down drive activity and it will be better to indicate this explicitly to user.

It's just because one of my jobs is writing high performance embedded real-time software were logs are almost the only way to know what happens in hardware. After several years of such practice there is a habit of suspecting side effects of each stralge log line. That's whay I do care :-) I just want to understand that this softreset failure is a normal behavior for that particular case.

Yes, you were right, it was a BIOS bug. I have downloaded latest BIOS from the vendor's web site and now PI register allways reports all six ports implemented.

Ok. So, one more question. How can I know exactly when device deletion has completed after sending this command? For example, consider that there some data in cache that needs 3 seconds to be sync-ed to disk. How can I know that I must wait for 3 seconds before a can actually remove the drive? Should I check the presence of some other filename in /sys/block/sdX/ or do something else?

Thank you!

With best regards, Vladimir Dashevsky

Hello,

Not all EMIs are one-shot events. Some can span seconds. Links don't always come up right after failures. Sometimes they require more than one hardresets to get back to working order. Link status report is not reliable. Sometimes they report offline for a while after certain events. If you know how to work around the above problems under a second, I'm all ears but I doubt it unless it involves an additional mechanical switch.

I don't know of a practical downside to lingering for limited amount of time. If you know one, please let me know.

If you sync and spin down the drive before hot unplugging, there's no practical difference. If you do a surprise hot-unplug, whatever was on the buffer will be lost and the drive will have to do an emergency unload.

Yes, they're expected. If you really don't like those messages, feel free to comment them out in your tree but please stop obsessing over the messages. I'll be happy to improve EH behavior but you need to come up with better reasons.

Great. :-)

The echo to delete node is synchronous. It will return after the device is completely removed but please note that "removing" in this sense only covers the device itself. It will flush the request queue and spin the drive down but won't do anything about filesystems. You need to unmount first. hal and desktop stuff already do the right thing for devices marked removable.

Thanks.

tejun

Tejun!

Tejun Heo пишет:

Well, for example, USB devices have a pull-up resistor on their D+ line. DC bias can be used for detection of device presence without mechanical switch.

Ok.

Ok, but two more questions:
1. Is there any generic mechanism of notifiing processes which had previously opened device being deleted of this event? What will happen to such processes? Is it possible to check who are those who uses the drive at the moment?
2. If the drive was deleted is it possible to start it back without physical re-connection? Can I simulate status change og that port to force the driver to auto-detect block device?

PS: as for this:

I can tell that for me enclosure management support is quite a good reason. Unfortunately, there is no this support in official kernel. I have seen only limited support of activity LED in kernel 2.6.28. However, I am using Debian where the latest kernel is only 2.6.26. As a result I had to write a simple ahci_em module which register simple proc interface to send LED states to all ICH9 ports. However, final goal is to integrate this module with mdadm to have proper indication of RAID state.

Best regards, Vladimir Dashevsky

Hello,

SATA is not USB and onlineness detection isn't that simple. Also, have you tried to run a system on a USB device over flaky connection?

-EIO will happen, fuser, but if you want something intelligent, hal + dbus.

I don't really follow what you're trying to achieve but if you want some fancy snapshotting + remapping trick, the best place would be dm.

How is that in any way exclusive against longer detach delay?

The biggest obstacle is that there aren't too many enclosure devices floating around. What kind of device are you using?

tejun

Tejun!

Well, I cannot argue with you here. All that I wanted to say is that I would prefer more optimistic software behavior if the hardware really supports device connection status.

Sorry, I missed the sense of this sentence. I tried this deletion with fdisk and see that fdisk does not even comply for device failure. It just starts to print empty partition table and so on. So the question is how to properly close any activity concerned with device being deleted if I do not know exactly what is that activity? Are the most typical programs which are allowed to use raw block devices aware of unexpected block device loss?

Well, I didn't think of any tricks. I just deleted the drive as you taught me and tried to get it back without moving myself in front of the server. :-) However, I think that some call to rescan scsi devices will be useful.

I just answered with better reasons to make you happy, not with another advice of detach delay.

I don't know exactly what device are you talking about. I was talking about LED message types that are supported in ICH9.
As for my server, ICH9 provides SGPIO interface that is routed to 4-drive hot-swap backplane based on AMI MG9071 chip. However, this information isn't needed to program ICH9 since the LED message mechanism is supported in it. Other message types are not supported. And it is very strange that linux ahci still does not support this functionality since it was first introduced in ICH8 (datasheet first release in June of 2006).

PS: My code has about 11Kb of text and supports all useful RAID states: NORMAL, LOCATE, REBUILD, FAILURE, HOTSPARE, PREDICTED FAILURE SOON. I have tested in on my server and it works. I think it can be useful for other implementations of soft RAID systems with hat swap support.

Best regards, Vladimir Dashevsky

Hello,

I really don't follow your train of thoughts here. Are you saying that the driver should be optimistic about the reliability about status reported by the hardware even when it is inherently imprecise (please read the spec) and real world experiments prove that?

-EIO will happen to any processes trying to do IO on the removed device. fuser will find out who's using the block device but if you want something more intelligent, look at hal + dbus.

Please take a look at how desktop guys are handling the issue. It's not something which can be handled in kernel proper.

Ah.. in that case, you can do

 # echo - - - > /sys/class/scsi_host/hostN/scan

Well, I was asking for better reasons to change the detach delay.

Yeah, I know it has been in the spec but without hardware to play with it's difficult to add driver features and lack of general availability also means lower demand.

I think it should be independent from RAID but having general enclosure support will be nice. Care to post the patches?

Thanks.

tejun

Tejun!

No. I ment that driver should performs better if the hardware supports some features for that. Consider two different cases.
1. hardware derives port population status by sensing the carrier in the data link. In this case it is possible that some EMI noise can damage link integrity so strongly that not data bits but also a carrier will be lost for a short time. This will lead to 'port is not present' status however noone has actually removed the drive.
2. Hardware implements some feature like pull-up resistor in USB, or special shorter 'present' contact as in PCI or CPCI connectors, or it simply senses some dc current through power lines etc. In this case port status is robust over EMI noise and be used to inform driver of actual connection. My thought was to improve driver behavior in case 2, either autodetected by PCI IDs or manually overriden by some configure script.

Hm, I tried to write fuser /dev/sda and got empty output. It seems that file system does not open sda. How it works?

Ok.

well, it works but it takes of about 10 seconds to finish scan for deleted drive. is this ok? Probably, that's because drive goes down after deletion and it starts to spin up during this scan.

Well, I just cannot imagine how software raid can work without clearly visible state. One drive mixed up in RAID5 and the whole array can get damaged. And it is not so difficult to mix them up because drive names may differ from physical slot numbers.

Well, I can provide you with a code which works on my ICH9 Supermicro platform. I believe it will also work with both ICH8 and ICH10.
However, since I could not install this module as traditional pci driver (the kernel decided not to claim my ahci device since the main driver present in the system) I had to rewrite it as a general linux kernel module. It justs scan pci devices for AHCI capable ones and remaps their ABAR to try enclosure management support. For now, only my ICH9 PCI IDs are in my try list. All AHCI EM-capable devices get their associated proc interface - /proc/ahci_emX/leds*. This module actually works in parallel with kernel ahci driver but I think it will be a conflict with it once the kernel driver starts to support em by itself. I guess, the best way would be to document some API for controlling the EM, then to declare some kernel ahci flag that will indicate full EM presence in the kernel. Then I can improve my ahci_em module to skip its installation when similar functions are built into the kernel.

My interface is quite simple. You just write a char to leds-controlling proc file to set state of leds, for example:
echo r > /proc/ahci_em0/leds0 means you asked for REBUILD state indicated in the bay of port 0.
I think that most of users would prefer additional module rather that kernel udgrade, for the first time. Also, I am not very close to linux kernel to provide a kernel patch.

Thanks.

Best regards, Vladimir Dashevsky

Hello,

SATA is even weaker than this. If you put a machine under an EMI test, SATA links check out the first together with unshielded serial connections.

There is no extra line in SATA connector. There are only two pairs of wires, one for each direction. Each pair is used to carry voltage differential. There is no common ground or closed circuits formed. The only thing which distinguishes an online link from an offline one is live signal on the receiving side, so the earlier mention of mechnical switch. ahci actually has support for it although I haven't seen any which actually implemented it.

No it won't show up and I can't really answer all your linux related questions. Sorry. :-P

Yeah, probably.

Yeah, it can be tricky but highend machine usually go with sas and consumer grade machines don't really care, so there just aren't too many ahci machines with enclosure support. Well, not here anyway.

I don't think the design you described would fit into upstream kernel too well but if you have it working it's some place to start. Just refresh in on top of the current devel kernel and let's see what can be done.

Thanks.

tejun

Tejun!

Nice system...

As for me, there is a lot of tasks where the main issue is a reliablity and not nesessary a performance.
SAS cannot be used for RAID anyway, it's closer to RAED. I guess that software RAID can be the best solution for every case where reliablity and online scalability is needed.

Sorry, I am not so close to linux kernel to perform a jump from my current kernel
(2.6.25) to latest one. I am sending you the sources of kernel module which performs all LEDs EM functions. The code is simple and relatively small. I have brushed it up to be more readable.

If you decide it useful and will change user API for this (eg, move proc interface to sys interface) then let me know.
I would like to design this software so that module could be used with older kernels and have the same behavior as newer kenels will have. this will help to spread this technology among machines where kernel cannot be upgraded for some reasons.

Usage notes:
1. Use 'make' to make driver module.
2. Use 'make reload' to (re)install driver into a system
3. Use 'echo X > /proc/ahci_em0/ledsN' to set state of leds for porn N (N is 0 to 5 for ICH9). X is one of the:
'N' - normal state (leds off)
'L' - locate state (4Hz blinking)
'R' - rebuild state (1Hz blinking)
'F' - failure state (solid + beep)
'P' - predicted failure state (two 4Hz blinks in 1 second )
'H' - hot spare ( two 4Hz blinks in 4 seconds).
My enclosure has only one state LED (besides activity LED) so I developed blink patterns as described here:
http://en.wikipedia.org/wiki/IBPI

Best regards, Vladimir Dashevsky