I request inclusion of SAS Transport Layer and AIC-94xx into the kernel

Hi Linus,

I request inclusion of the SAS Transport Layer and the AIC-94xx SAS LLDD into the Linux kernel.

Please find your latest git tree with SAS Transport Layer and AIC-94xx, patches and
a whole tar.bz2 tree here:

http://linux.adaptec.com/sas/

There you will also find the original announcements posted to this list.

The code is updated twice daily or more often as needed.

Thank you,
Luben
P.S. This is a second resend of an identical message I posted to lkml and lsml yesterday.

And it's not gotten anymore includable. Please fix the major structural issues pointed out when you first sent it out. That's in the following order:

- not trying to integrate with the sas transport class in Linus' tree at all, not even using the transport class infrastructure at all, creating it's own sysfs trees in rather wierd ways
- not beeing structures as a library (ala libata which is a very good model for it)
- duplicating the lun scanning code instead of using the scsi core one

Major?

Christoph, why diseminate FUD, when we can concentrate on the _technical_ merits of SCSI and SAS instead?

Why talk non-constructive things, when we can have a SCSI Storage discussion?

Well here we go _again_:

The SAS Transport Class (your an JB's incarnation) is _not_ a management infrastructure, it was _never_ _intended_ to be.

The whole point of it is to _export_ *attributes* of MPT-technology like drivers. All those drivers that it caters to _do_ _not_ need a _management_ layer (Discovery, Expander configuration, etc.). They "export" SCSI LUs directly to SCSI Core through their LLDDs.

If you cared to read any of the "techno-babble" (as you call it) documents and specs you'd clearly see how and _why_ it fits with a SCSI Stack. As baby food, see this _picture_:
http://www.t10.org/scsi-3.gif
For an adult meal, maybe some reading of "techno-babble" would help?

"weird ways" ? Did you care to see what a SAS domain looks like? There is plenty of references, slides, course notes, etc on the Internet.

Christoph, I work with this technology every day. OTOH, you blocked LSI's drivers from going in until they sent you hardware just a month ago.

What you see in sysfs is _exactly_ what you'd see in the _physical_ world, _including_ the "locking" -- i.e. the "kobject_get". It "locks" object which are needed for the command to travel to, in _actuallity_.

If you say that it is "weird" then you are also saying that a SAS Domain in the *physical* world is weird.

It is the _natural_ representation of the SAS domain.

What seems "weird" to you, is what it _actually_ is.

This is what this new technology is. You can learn it or you can call it "weird" and "techno-babble", and invent your own understanding of SAS and shove it down the throat of thousands of Linux users and vendors.

It _cannot_ be presented as a library, because _again_ it is a *management layer*, an infrastructure.

What it manages is, _again_ to repeat myself, SAS objects:
ports, connections, devices, discovery, expander management, etc.

See the original Announcement 0 here, it explains this in detail:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112629423714248&w=2

The LUN scanning code is, uum, how to say this nicely... wrong? It did its job for Parallel SCSI and for broken arrays who do not respond to REPORT LUNS, but have a bunch of disks behind, but it is wrong _by design_ and it is _not_ _relevant_ in SAS. To properly see how LUN scanning is done, look at sas_discover.c.

SCSI Core does not know about everyday SCSI objects like a "SCSI Device with a Target port". It does things backwarads _and_ foremost of all, it uses _HCIL_.

What needs fixing is, SCSI Core to
- not use HCIL,
- use 64 bit LUNs,
- know about SCSI devices with Target ports,
- proper representation of SCSI Domains (FC, USB, IEEE 1394, Infiniband, SAS, iSCSI)

Christoph, SCSI Core is 20 years behind.

I appreciate your aggresiveness and JB's political games, but what it comes down to, Linux SCSI, vendors, users, and its _other_ developers suffer.

Anyway, when all you have is an AIC-94xx or BCM8603 chip on your mainboard (check with SuperMicro) you *do not need* the semantically fat, broken and wrong SCSI Core (catering to old, outdated, broken SPI machines).

You need a _minimal_ SCSI Core, SAM/SPC-like, when you have a new technology like SAS and _none_ of the semantically fat and broken SCSI Core is _relevant_ in a SAS world.

Christoph how long are you and James Bottomley going to hold SCSI Core _hostage_ to new technologies?

How long are you going to _block_ SCSI storage innovation from the Linux kernel?

Or is this the new way of embezzling hardware from vendors? Is this what is done in the other Linux kernel subsystems?

I don't get it. If you or James Bottomley think that
- LUNS can be represented as "u64", and
- sending REQUEST SENSE clears ACA,
- "SCSI Device with Target port" is "techno-babble",
how can you drive new SCSI technology?

Someone please pinch me, because I'm dreaming this horrible nightmare.

Luben

That's what *we* are concentrating on. Technically, I have no problem with the aic94xx driver being based on a domain device. However, you cannot have two separate transport classes for SAS. So these two need to be unified before this driver becomes includable.

It looks to me to be a fairly simple exercise to unify these two classes giving LSI a functional interface and you a domain device based one and removing all the duplication of mid-layer functionality as part of doing this.

Actually, it was intended to be such. The sysfs components of the transport class are the unified management interface to the transport.

No, the point of a transport class is to export the underlying attributes of the actual devices that are present. This is supposed to be independent of the driver used to connect to them (and that's what your sas class breaks).

I've told you before, if you find something that's broken send a patch in to fix it. I've already told you why the code in sas_discover can't work for other drivers (although I still don't have an explanation from you of why scsi_scan_target can't work for sas_discover).

As I've already said several times you're welcome to send in a patch to change this as well ... as long as you either don't break every other driver, or fix them all with the patch. You were given a step by step procedure at least for the I replacement piece.

The only power of a maintainer is to say "no". So, although I cannot make you fix any of the problems in your submission, I can say no until an acceptable submission comes along for this driver. At this point, I believe all the technical issues of what needs to happen are well understood; I also believe that this is an important enough piece of hardware that an acceptable driver will come along even if you want to play dog in the manger, so all I can do is wait.

James

Nah. You're concentrating on blocking new drivers and _innovation_ from getting into SCSI Core.

Just look, for all these years you've been "maintaining" SCSI Core it still uses HCIL, LUNS are u32, and you think that REQUEST SENSE clears ACA, and scsi core has no representation of targets to save its life.

It just hurts us all.

Apparently you do _not_ understand what the SCSI architecture model is, again I refer you to the _picture_ here:
http://www.t10.org/scsi-3.gif

Now,

1) MPT _gives_ you LUs -- all transport _and_ SCSI specific work has already been done for you.

2) With MPT you export attributes by poking, in a firmware _dependent_ way, into the firmware of the controller.

That is, _all_ the "stuff" below SAM, as you can see in the picture link, happens in the firmware! What the firmware _gives_ you in a _uniform_ way is SAM/SPC objects only (via the LLDD) so you can register the LUs with SCSI Core, etc.

None of this is true for the SAS Transport Layer and
the AIC-94xx driver and BCM8603. That is, for those solutions
the transport (SAS) architecture is exposed right down to the phy/link layer.

So you _need_ transport management. And since SAS sticks to SAM so nicely and is so very well layered, it implements right into what you have in the SAS Transport Layer.

Ok, I'm aware that you're well aware that very few people actually understand indepth MPT and other architectures.

You're posting FUD here, and hope that, since no one understands things indepth, you can post anything you want, and sound credible.

I repeat again:

Those are two *radically* _different_ and _distinct_ technologies.

What MPT does is _everything_ *below* SCSI Core in terms of SAS Transport Layer and AIC-94xx LLDD.

That is all the work you see done in the SAS Transport Layer and in AIC-94xx IS DONE ALREADY IN THE FIRMWARE!

What our solution and apparently BCM9603 does is expose _all_ that.

What you need to do is:
- Let the code in _now_, as is.
- Let _people play_ with it as hardware comes more available. I said "people" not necessarily you and/or Christoph.
- Let it evolve in the hands of the people.
- Hear feedback from the people.
- Accept patches.
- Let it evolve.

IOW, do not bastardise it now when you have _no_ SAS or SCSI expertise, or hardware (well, maybe you have some on the way from your friends at XYZ).

JB's statement A:

If it was _ever_ intended to be such, then it would sit _below_ SCSI Core _and_ SCSI Core would NEVER be aware of it. (just as it is on the picture, see?)

Even the _name_ suggests it: SCSI transport _attributes_, and it is a template to export _attributes_.

You *CANNOT* write a template for _all_ transports, since this is what SCSI Core is supposed to be! This is what SAM/SPC _is_! This is what SCSI Core should be striving to.

The transport management layer sits _BELOW_ SCSI Core, and _manages_ the particular transport, just like the SAS Transport Layer. SCSI Core does only SAM/SPC tasks, and is _unaware_ of the transport. (How hard is this to understand?)

See the _pictures_ on t10.org and the "techno-gibberish" posted there?

JB's statment B:

Exactly! And this statement B here, contradicts your statement A) above.

Note: it is a transport _class_, _not_ a management _layer_, which is what you have with the SAS Transport Layer.

Mine is _not_ a class. It is a transport layer to _manage_ and drive SAS LLDDs.

LSI's driver is NOT, to repeat it is NOT a SAS LLDD! It is an *MPT* LLDD.

Your "transport attribute class" provides hooks for exporting _attributes_ from MPT-based LLDDs.

It does _NOT_ provide for a management infrastructure:
1) because there is none needed -- the LLDD is MPT and you're providing attribute exporting only,
2) All this is _firmware_ implemented.

I repeat again: you cannot have a single _template_ for all-protocol _management_ as you're implying.

For the record: I think MPT is a wonderful technology in its right and the SAS LSI's solution should've been accepted right when it was posted, since it didn't have any SAS in it. The only thing it had was a few different PCI IDs, AFAIR looking at the patch.

Nah, lets not go with this BS: "I've already told you...", it just doesn't work. Be specific. See above how much effort I put in repeating myself over and over and over again. And I put the effort to actually type everything out.

Why? Because it uses HCIL? Because LUN are represented as u32? Because you think that a LUN is a "u64". Because you think that REQUEST SENSE clears ACA?

This is complete FUD and you know it and it is just the political game you've been playing all this time.

Proof:

1) Any patches submitted to lsml, you change and then you resubmit yourself, unless of course they come in from a few "chosen" people. I.e. people who do not challenge your antics.

This _policy_ disourages people from actually doing any new and innovative work with SCSI Core.

2) You are well aware that NONE of SCSI Core is relevant as it stands today for a new, SAM/SPC abiding architecture like SAS. You know very well that other than LUN scanning and some SAM/SPC tasks, one needs very little to support SAS.

3) Your own effort on this (converting LUN to "u64") failed:
http://marc.10east.com/?l=linux-scsi&m=112664309529813&w=2

YET, you try to keep this semantically fat, overloaded with lots of little quirks for old hardware, SCSI Core.

You have to let things in, so that they can evolve naturally. You cannot give heart attack to 50 or so LLDDs.

How many times do I have to tell you that this was Christoph's reply to an email of mine _asking_ me if this were the way to do it?

And then I told him that it is not quite the way to do it? See this link:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112507133514575&w=2

You see, you and I do not see eye to eye, since we do not have the same
base: T10.org.

I read books, specs, drafts, firmware, code, presentations, lectures, etc. It isn't easy, but necessary to do my job better and better every day. It is necessary so that I'm knowlegable and can provide immediate response and help when asked to.

So I'm not sure about your point of view. SCSI 20 years ago?

I'm not sure what kind of convicing it would take?

Maybe we can do a presentation in front of an audience of _storage_ folks? We can both explain out points of view.

Nah. You're just saying this to manipulate the readers of this thread. Or maybe those "needs" are well understoood _in your head_ only? I told you before: be _specific_.

What needs improvement is NOT the SAS Transport Layer and/or aic94xx LLDDs.

What needs improvement, in order to _better_ _accomodate_ _new_ technologies is SCSI Core. If not even a new SCSI Core developed in parallel (so the old one can be config-not-compile if one has no legacy controllers and devices and vice-versa).

SCSI Core is 20 years _behind_. And thus _Linux_ SCSI is 20 years behind. Apparently no one cares.

When you get your new shiny mainboard with an AIC-94xx chip on it (check with SuperMicro), which can handle SAS and SATA, you do not need to compile this fat, old and semantically broken SCSI Core, but a smaller, faster one.

Since when do you think it is important? This is the first time you're saying this and I think someone was talking to you. XYZ?

So what you're saying is "my way or the highway"?

"all I can do is wait"?
James, why not just simply move out of the way?

Instead of you _waiting_ and thus do nothing, you can move out of the way, or listen and accept _technical_ advice.

This is unfortunate. History shows us that being
inflexible to new ideas (or technologies) becomes
one's undoing.

I wonder if SCSI Core could be Linux's undoing? Could this and reiser4, and OBDs (yet another new SCSI technology in the making), show us how inflexible Linux has become?

Now its SAS and reiser4. When OBDs come out full force it will be the block layer, then who knows whatelse...

Is Linux going to be just as obstinate?

Why has Linux become like this?

Luben

Luben,

The fact that your responses are constantly filled with non-technical paranoia does not help the inclusion of aic94xx at all.

Maybe you need to write your driver as a block driver, and completely skip the SCSI core, if it bothers you so much? That would get everybody else out of the loop, and free you to write the driver as you see fit.

As it stands now, you're making an end run around the SCSI core, rather than fixing it up. SCSI needs to be modified for SAS, not just complained about.

Jeff

Hi Jeff, how are you doing?

Yes, that has been suggested before. But do you remember what happened when I posted a patch to IDR? James moved from SCSI Core to IDR. hehehe ;-)

In the mids of the FUD, I completely removed IDR from being used in aic94xx, long before I posted the driver.

Well, back in 2001-2 I was asking for 64 bit LUN support and for native SCSI target support (since iSCSI "exports" targets), and it uses 64 bit LUNs (as any other transport). Both sniffed at and rejected by your friends.

See this thread from me, all the way back in 2002:
http://marc.theaimsgroup.com/?l=linux-scsi&m=103013448713434&w=2

You still have people from IBM who claim that 64 bit LUN support is completely unnecessary as recently as 2 weeks ago. (link to email on marc.10east.com available upon request)

As it has come to now, 2005, we cannot afford any more "putting off".

The driver and the infrastructure needs to go in.

Give it exposure to the people, let people play with it.

If we start "fixing" SCSI Core now (this in itself is JB red herring), how long before it is "fixed" and we can "rest"? And how long then before the driver and infrastructure makes it in?

"How long is the long hair?"

You are calling for fixing SCSI Core. JB is calling for integrating MPT with open transport. I'm sure people have other (crazy) requests.

At the end of the day the driver is not in, and business suffers. And its not like the driver is using static struct file_operations megasas_mgmt_fops, ;-) IOCTLs or other char dev for management...

The driver does _not_ alter anything in the kernel, it only integrates with it.

There needs to be a "passing gate":
Linus, let the driver and transport layer in, as is and then patches "fixing SCSI Core" would start coming, naturally.

Luben

Merging into the upstream kernel is not necessary for exposure.

Historically, saying "no" to a single vendor pushing really hard -- as you are doing -- has resulted in a superior solution.

Just follow the recipe Christoph outlined. It's not difficult, just requires some work.

So far, this is an Adaptec-only solution.

It does an end run around 90% of the SCSI core. You might as well make it a block driver, if you're going to do that.

Jeff

This of course implies that everything in Linux is a superiour solution _or_ if it is not, then everything in linux is a vendor solution.

Anyone have a clear technical plan and/or infrastructure on how to do this? Any specs, plans, consolidations, etc?

I know its a wishful thinking, but when the architectures differ, not sure how to do this.

"You can do everything with a big long if ()".

Yes, since Adaptec is the first company to come out with open transport SAS chip. Broadcom seems to be doing the same thing.

The "90%" part isn't quite true.

But going with a block device isn't that bad:
Now since the architecture _is_ SCSI after all, what would be needed is a minimal, fast, straightforward, SAM/SPC fully complient new SCSI Core. That SCSI Core would register block devices with the block layer. Maybe Jens can point out cool things to do and make the whole infrastructure fast and very fast. (since we don't need to be bound by this legacy stuff)

Essentially other new technology LLDD and Transport layers can probably make use of this: Infiniband, USB2 Storage, etc.

So if all you have is an AIC-94xx or BCM8603 storage chip you can exclule all of the legacy SCSI Core and compile this new mean, lean, fast SAM Core.

Jeff, this isn't bad at all!

Are you willing to contribute to such an effort?

Thanks,
Luben

Or perhaps that it tends to result in better solutions than what would exist if every vendor wrote their drivers with no consideration for other vendors.

This sounds equivalent to "write your own block driver".

Except that you would have to re-implement the SCSI upper-layer drivers (at a minimum). Seems like it would be easier to take the existing code in your driver/layers and modify to work with the existing SCSI infrastructure.

Seems silly to have all this code duplication. Why not write to the existing spec (even if it is an informal one), and then work to get the spec changed, hopefully without pissing off all the maintainers.

I am not sure if Jeff meant this as a tongue-in-cheek suggestion or is just trying to get you off peoples backs. Perhaps he is indeed serious.

Andrew

Yes, read Christoph's todo list.

We need to separate out bits specific to parallel SCSI, moving the bits to transport-specific code, since SAS doesn't need them.

No. I'm much more willing to help integrate aic94xx and Broadcom/ServerWorks into the existing SCSI core, working with the community.

As Andrew guessed, it's a way to get you your own playpen, where you won't be constrained by members of the Linux-SCSI community.

Jeff