Separating out libata out of SCSI (finally)

PS. Brian, can you please point me to the latest version of libata EH integration patch?

Hello, all.

This item was on TODO list for years now. People all agree that it's necessary but it always had relatively low priority probably because it's a bit difficult and isn't really necessary to make disks and optical drives work.

Anyways, I think it's about time to take some action as SAS-ATA integration (Brian, sorry about staying so silent about this for long time, I was following the threads but couldn't really think of a quick solution) and other ATA specific things including link power management and bunch of other deferred ones due to lack of proper sysfs interface or high level driver (parallel probing, parallel resume).

Currently, my plan is...

* Move high level driver handling to request_queue.

* Implement queue quiescing and other state management on request_queue.
* Implement block_queue_group which...
- Handles command scheduling.
- Handles grouped queue quiescing and EH handling

* Move SCSI high level drivers to new infrastructure
* Convert libata to use new command scheduling and EH infrastructure
* Apply driver model to libata.
* Implement ata_disk ATA high level driver.

In the process, I'm planning to remove ata_host requirement and break down libata EH into actions and sequencers so that SAS can use them easily.

The biggest problem is how to keep userland happy. hdX -> sdX transition was painful enough and I have a strong feeling that everyone will come after and hunt down us if we try something like sdX -> bdX now. :-)

It would be ideal if those ATA specific sysfs stuff just shows up without disturbing the original SCSI things which are now widely used for enumeration and manipulation. I think we can get pretty close by modifying SCSI high level drivers a bit such that they don't register block devices for SCSI devices created to keep backward compatibility. This is an extra burden on SCSI but it's gonna be the last one.

*** Currently, sysfs nodes for a libata disk is like the following.

/devices/DEV/hostH/targetH:C:I/H:C:I:L/

	 driver			-> /bus/scsi/drivers/sd
	 generic			-> /class/scsi_generic/sgN
	 block/sdX/
		 device		-> ../../
		 partitions...

/bus/scsi/devices/H:C:I:L -> /devices/DEV/hostH/targetH:C:I/H:C:I:L
/bus/scsi/drivers/sd/H:C:I:L -> /devices/DEV/hostH/targetH:C:I/H:C:I:L
/class/scsi_host/hostH/device -> /devices/DEV/hostH
/class/scsi_device/H:C:I:L/device -> /devices/DEV/hostH/targetH:C:I/H:C:I:L /class/scsi_disk/H:C:I:L/device -> /devices/DEV/hostH/targetH:C:I/H:C:I:L
/class/scsi_generic/sgN/device -> /devices/DEV/hostH/targetH:C:I/H:C:I:L
/block/sda -> /devices/DEV/hostH/targetH:C:I/H:C:I:L/block/sdX

*** After conversion, it will look something like the following.

/devices/DEV/ata_port/P/ata_link/P:L/ata_device/P:D

	 driver			-> /bus/ata/drivers/ata_disk
	 block/sdX/
		 device		-> ../../
		 partitions...

C scsi_device -> /devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L
/bus/ata/devices/P:D -> /devices/DEV/ata_port/P/ata_link/P:L/ata_device/P:D
/bus/ata/drivers/ata_disk/P:D -> /devices/DEV/ata_port/P/ata_link/P:L/ata_device/P:D
/block/sda -> /devices/DEV/ata_port/P/ata_link/P:L/ata_device/P:D/block/sdX

/devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L/

	 driver			-> /bus/scsi/drivers/sd
	 generic			-> /class/scsi_generic/sgN

C block -> ata_port/P/ata_link/P:L/ata_device/P:D/block
C ata_device -> /devices/DEV/ata_port/P/ata_link/P:L/ata_device/P:D

/bus/scsi/devices/H:C:I:L -> /devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L
/bus/scsi/drivers/sd/H:C:I:L -> /devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L
/class/scsi_host/hostH/device -> /devices/DEV/scsi_compat/hostH
/class/scsi_device/H:C:I:L/device -> /devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L
/class/scsi_disk/H:C:I:L/device -> /devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L
/class/scsi_generic/sgN/device -> /devices/DEV/scsi_compat/hostH/targetH:C:I/H:C:I:L

The SCSI side of interface will remain as functional as now as it will go through the same libata SAT layer.

The only surprise userspace
will see aside from the extra ata nodes is that /sys/block/sdX/device will lead to an ATA device node instead of SCSI device node. Well, that's the whole point of the converion but I think the surprise can be minimized by reusing names used in SCSI device node and possibly making symlinks for nodes which only makes sense for SCSI device.

So, what do you guys think?

Quoted 4 lines · Tejun Heo

The biggest problem is how to keep userland happy. hdX -> sdX transition was painful enough and I have a strong feeling that everyone will come after and hunt down us if we try something like sdX -> bdX now. :-)

In theory mounting by label or ID should have fixed a lot of this. However, if we need to head off a revolt, the sdX allocation algorithm can be placed into it's own module so both sd and a ULD ata driver could use it ...

Actually, surely we can mostly dump the SAT layer?

1) Make SCSI block devices themselves an allocate-able resource (I think that's what you meant by "placed into it's own module so both sd and a ULD ata driver could use it"?)

2) Ensure that any ata_disk ULD would support the same partition limits and ioctl set, enough to ensure binary compatibility.

Because that's the real need -- maintaining binary compatibility with SCSI block devices, so major/minor, ioctl supported set, partition limits, and other relevant details need to remain unchanged.

The underlying software we're of course free to change...

Quoted 8 lines · James Bottomley

Quoted 5 lines · Tejun Heo

The biggest problem is how to keep userland happy. hdX -> sdX transition was painful enough and I have a strong feeling that everyone will come after and hunt down us if we try something like sdX -> bdX now. :-)

Quoted 6 lines

In theory mounting by label or ID should have fixed a lot of this. However, if we need to head off a revolt, the sdX allocation algorithm can be placed into it's own module so both sd and a ULD ata driver could use it ...

Actually, surely we can mostly dump the SAT layer?

I don't see that we can do that for a long time... And it's not just the sdX allocation algorithm in question -- SCSI block devices come with their own partition limits and set of supported ioctls.

Therefore, my recommended path has always been

* create ata_disk block device driver (ULD, in your terminology)

* make SAT an optional piece, which maintains compatibility with existing SCSI blkdevs, ioctls, command sets

I just don't see a valid path moving forward that breaks userland /again/...
we (ATA hackers) would be drummed out of a job I think :)

Another option that's been discussed is

1) Make SCSI block devices themselves an allocate-able resource (I think that's what you meant by "placed into it's own module so both sd and a ULD ata driver could use it"?)

2) Ensure that any ata_disk ULD would support the same partition limits and ioctl set, enough to ensure binary compatibility.

Because that's the real need -- maintaining binary compatibility with SCSI block devices, so major/minor, ioctl supported set, partition limits, and other relevant details need to remain unchanged.

Quoted 41 lines · Jeff Garzik

Quoted 8 lines · James Bottomley

Quoted 5 lines · Tejun Heo

The biggest problem is how to keep userland happy. hdX -> sdX transition was painful enough and I have a strong feeling that everyone will come after and hunt down us if we try something like sdX -> bdX now. :-)

Quoted 6 lines

In theory mounting by label or ID should have fixed a lot of this. However, if we need to head off a revolt, the sdX allocation algorithm can be placed into it's own module so both sd and a ULD ata driver could use it ...

Actually, surely we can mostly dump the SAT layer?

I don't see that we can do that for a long time... And it's not just the sdX allocation algorithm in question -- SCSI block devices come with their own partition limits and set of supported ioctls.

Therefore, my recommended path has always been

* create ata_disk block device driver (ULD, in your terminology)

* make SAT an optional piece, which maintains compatibility with existing SCSI blkdevs, ioctls, command sets

I just don't see a valid path moving forward that breaks userland /again/...
we (ATA hackers) would be drummed out of a job I think :)

Another option that's been discussed is

1) Make SCSI block devices themselves an allocate-able resource (I think that's what you meant by "placed into it's own module so both sd and a ULD ata driver could use it"?)

2) Ensure that any ata_disk ULD would support the same partition limits and ioctl set, enough to ensure binary compatibility.

Quoted 4 lines

Because that's the real need -- maintaining binary compatibility with SCSI block devices, so major/minor, ioctl supported set, partition limits, and other relevant details need to remain unchanged.

I've seen a lot of end user complaints about libata only supporting 15(14?) partitions. Will that limit be moved back to the traditional drivers/ide limit as part of this?

Quoted 4 lines

I've seen a lot of end user complaints about libata only supporting 15(14?) partitions. Will that limit be moved back to the traditional drivers/ide limit as part of this?

Number of partitions is directly related to number of minors, so it can't be changed without a change in the allocation of major/minor space in sd ... that could only be done compatibly by permuting the space. The only other way to do it is incompatibly by changing major (again).

Quoted 37 lines · Greg Freemyer

Quoted 36 lines · Jeff Garzik

Quoted 10 lines · James Bottomley

Quoted 4 lines · Tejun Heo

The biggest problem is how to keep userland happy. hdX -> sdX transition was painful enough and I have a strong feeling that everyone will come after and hunt down us if we try something like sdX -> bdX now. :-)

In theory mounting by label or ID should have fixed a lot of this. However, if we need to head off a revolt, the sdX allocation algorithm can be placed into it's own module so both sd and a ULD ata driver could use it ...
Actually, surely we can mostly dump the SAT layer?

I don't see that we can do that for a long time... And it's not just the sdX allocation algorithm in question -- SCSI block devices come with their own partition limits and set of supported ioctls.

Therefore, my recommended path has always been

* create ata_disk block device driver (ULD, in your terminology)

* make SAT an optional piece, which maintains compatibility with existing SCSI blkdevs, ioctls, command sets

I just don't see a valid path moving forward that breaks userland /again/...
we (ATA hackers) would be drummed out of a job I think :)

Another option that's been discussed is

1) Make SCSI block devices themselves an allocate-able resource (I think that's what you meant by "placed into it's own module so both sd and a ULD ata driver could use it"?)

2) Ensure that any ata_disk ULD would support the same partition limits and ioctl set, enough to ensure binary compatibility.

Quoted 7 lines

Quoted 4 lines

Because that's the real need -- maintaining binary compatibility with SCSI block devices, so major/minor, ioctl supported set, partition limits, and other relevant details need to remain unchanged.

I've seen a lot of end user complaints about libata only supporting 15(14?) partitions. Will that limit be moved back to the traditional drivers/ide limit as part of this?

Number of partitions is directly related to number of minors, so it can't be changed without a change in the allocation of major/minor space in sd ... that could only be done compatibly by permuting the space. The only other way to do it is incompatibly by changing major (again).

James

Quoted 5 lines · James Bottomley

Quoted 3 lines · Greg Freemyer

I've seen a lot of end user complaints about libata only supporting 15(14?) partitions. Will that limit be moved back to the traditional drivers/ide limit as part of this?

Quoted 5 lines

Number of partitions is directly related to number of minors, so it can't be changed without a change in the allocation of major/minor space in sd ... that could only be done compatibly by permuting the space. The only other way to do it is incompatibly by changing major (again).

Could we do both? I mean use the legacy, up to 15, with the old major, then use the new major for bigger then 15. Since user mode that knows about more then 15 partitions is new, it'll know it needs to jump a major.

Quoted 6 lines · Boaz Harrosh

Quoted 5 lines · James Bottomley

Quoted 3 lines · Greg Freemyer

I've seen a lot of end user complaints about libata only supporting 15(14?) partitions. Will that limit be moved back to the traditional drivers/ide limit as part of this?

Quoted 8 lines

Quoted 5 lines

Number of partitions is directly related to number of minors, so it can't be changed without a change in the allocation of major/minor space in sd ... that could only be done compatibly by permuting the space. The only other way to do it is incompatibly by changing major (again).

Could we do both? I mean use the legacy, up to 15, with the old major, then use the new major for bigger then 15. Since user mode that knows about more then 15 partitions is new, it'll know it needs to jump a major.

Not simultaneously, which is the problem; you can't have two separate block devices for the same physical device unless you want aliasing issues in the page cache.

It might be possible to add an extra device to give access to the missing partitions, but that would require a bit of re-engineering in gendisk (which is the in-kernel code to manage the partitions).

What might be far more feasible is to set up udev to use kpartx to provide the missing partitions if it detects a partition table that has them ... of course, that requires a udev setup and most of the complaints about the lost partitions seem to come from non-udev systems.

But .... if everyone (particularly the people with these problems) had udev, we could simply migrate to a new major with more partitions, get udev to fix it all up for us and everyone would be happy because no-one would even notice that we'd moved majors ...

James

From my limited perspective, every complaint I've seen was related to

From my limited perspective, every complaint I've seen was related to OpenSUSE which I believe is a udev based distro, so addressing this for the udev based distros would address that contingent of users.

Quoted 5 lines · Boaz Harrosh

Quoted 4 lines · James Bottomley

Quoted 3 lines · Greg Freemyer

I've seen a lot of end user complaints about libata only supporting 15(14?) partitions. Will that limit be moved back to the traditional drivers/ide limit as part of this?

Quoted 8 lines

Quoted 5 lines

Number of partitions is directly related to number of minors, so it can't be changed without a change in the allocation of major/minor space in sd ... that could only be done compatibly by permuting the space. The only other way to do it is incompatibly by changing major (again).

Could we do both? I mean use the legacy, up to 15, with the old major, then use the new major for bigger then 15. Since user mode that knows about more then 15 partitions is new, it'll know it needs to jump a major.

Not simultaneously, which is the problem; you can't have two separate block devices for the same physical device unless you want aliasing issues in the page cache.

It might be possible to add an extra device to give access to the missing partitions, but that would require a bit of re-engineering in gendisk (which is the in-kernel code to manage the partitions).

What might be far more feasible is to set up udev to use kpartx to provide the missing partitions if it detects a partition table that has them ... of course, that requires a udev setup and most of the complaints about the lost partitions seem to come from non-udev systems.

But .... if everyone (particularly the people with these problems) had udev, we could simply migrate to a new major with more partitions, get udev to fix it all up for us and everyone would be happy because no-one would even notice that we'd moved majors ...

I'm currently working on a scheme where partitions above gd->minors get allocated dynamic MAJ:MIN. It looks like it can be done mostly in block layer proper. The only problem I can foresee is not being able to specify MAJ:MIN as root device but that shouldn't be a major problem. I'll report back when I make more progress.

On 2008/06/24 23:59 (GMT+0900) Tejun Heo apparently typed:

Quoted 5 lines

I'm currently working on a scheme where partitions above gd->minors get allocated dynamic MAJ:MIN. It looks like it can be done mostly in block layer proper. The only problem I can foresee is not being able to specify MAJ:MIN as root device but that shouldn't be a major problem. I'll report back when I make more progress.

Please correct me if I'm wrong, but on the following, would it not be a major problem booting, without deleting any existing or moving any existing, with only a few more upgrade cycles to replace what's there installed now with current and near future distros?

Quoted 7 lines · Felix Miata

On 2008/06/24 23:59 (GMT+0900) Tejun Heo apparently typed:

Quoted 5 lines

I'm currently working on a scheme where partitions above gd->minors get allocated dynamic MAJ:MIN. It looks like it can be done mostly in block layer proper. The only problem I can foresee is not being able to specify MAJ:MIN as root device but that shouldn't be a major problem. I'll report back when I make more progress.

Quoted 4 lines

Please correct me if I'm wrong, but on the following, would it not be a major problem booting, without deleting any existing or moving any existing, with only a few more upgrade cycles to replace what's there installed now with current and near future distros?

I'm not sure what you mean. If you switch to libata now, you'll only be able to see 15 partitions, so, yes, you'll have problem updating to a

distro which uses libata now and in near future. If the dynamic partition thing goes well, hopefully the next major distro releases work fine.

Quoted 3 lines

distro which uses libata now and in near future. If the dynamic partition thing goes well, hopefully the next major distro releases work fine.

I responded as I did due to your statement "only problem I can foresee is not being able to specify MAJ:MIN as root device", hoping to see some elaboration on the statement from you. I read your statement as "don't expect to be able to use any partition >15 as a root device". So, in the past root=3:17 would have been a valid replacement for root =/dev/hda17? If so, sorry for the noise, as I'd never seen that type of usage, and certainly would not miss it.

I'm currently working on a scheme where partitions above gd->minors get allocated dynamic MAJ:MIN. It looks like it can be done mostly in block layer proper. The only problem I can foresee is not being able to specify MAJ:MIN as root device but that shouldn't be a major problem. I'll report back when I make more progress.

Quoted 15 lines · Tejun Heo

Hello, all.

This item was on TODO list for years now. People all agree that it's necessary but it always had relatively low priority probably because it's a bit difficult and isn't really necessary to make disks and optical drives work.

Anyways, I think it's about time to take some action as SAS-ATA integration (Brian, sorry about staying so silent about this for long time, I was following the threads but couldn't really think of a quick solution) and other ATA specific things including link power management and bunch of other deferred ones due to lack of proper sysfs interface or high level driver (parallel probing, parallel resume).

Quoted 3 lines

Currently, my plan is...

* Move high level driver handling to request_queue.

Actually, I already did quite a lot of that here:

commit 7f9a6bc4e9d59e7fcf03ed23f60cd81ca5d80b65

Author: James Bottomley <[email protected]>
Date:   Sat Aug 4 10:06:25 2007 -0500

[SCSI] move ULD attachment into the prep function

But there's still more to be done. The way I was thinking of it was some type of protocol label (as in a ULD spits out protocols, like SCSI or ATA) and then passes them to a LLD which uses libraries (what libata and the scsi mid layer become) to process them.

Quoted 4 lines

* Implement queue quiescing and other state management on request_queue.
* Implement block_queue_group which...
- Handles command scheduling.
- Handles grouped queue quiescing and EH handling

There's the beginnings of this in Jens' unmerged block timers work

Quoted 8 lines

In the process, I'm planning to remove ata_host requirement and break down libata EH into actions and sequencers so that SAS can use them easily.

The biggest problem is how to keep userland happy. hdX -> sdX transition was painful enough and I have a strong feeling that everyone will come after and hunt down us if we try something like sdX -> bdX now. :-)

In theory mounting by label or ID should have fixed a lot of this.

However, if we need to head off a revolt, the sdX allocation algorithm can be placed into it's own module so both sd and a ULD ata driver could use it ...

Could you, perhaps, make the port multipler visible in this as a new device, a bit like we do today for SAS expanders?

Quoted 2 lines

The SCSI side of interface will remain as functional as now as it will go through the same libata SAT layer.

Actually, surely we can mostly dump the SAT layer? libsas should be made capable of taking ATA protocol packets straight from your ULD ATA driver and sending them out.

I could see us still needing it as an optional component so we can send SCSI SG_IO to ATA devices.

Quoted 1 line

So, what do you guys think?

I think the devil will be in the details, but that it certainly won't be obvious until the conversion is actually tried.