I request inclusion of SAS Transport Layer and AIC-94xx into the kernel

Hi everyone,

I request inclusion of the SAS Transport Layer and the AIC-94xx SAS LLDD into the Linux kernel.

Please find the latest Linus' git tree with SAS Transport Layer and AIC-94xx, patches and
a whole tar.bz2 tree here:

http://linux.adaptec.com/sas/

There you will also find the original announcements posted to this list.

The code is updated twice daily or more often as needed.

Thank you,
Luben

Overall, I see the following major issues. My recommendation is that this driver live in -mm, or outside the kernel tree, for a while to let things shake out.

Background
----------
* There is no question that Adaptec's aic94xx and SAS code is correct WRT the SCSI specifications. The maintainer eats, sleeps, and breathes SAS specs. :)

* aic94xx hardware supports both SAS and SATA connections. Since SAS and SATA are intentionally similar electrically, other vendors are coming out with SATA+SAS hardware too.

* Acronym: "HCIL" Stands for Host/Channel/ID/LUN. Pre-SAS legacy addressing method.

Issues
------
* Avoids existing SAS code, rather than working with it.

* Avoids existing SATA code, rather than working with it.

* Avoids (rather than fix) several SCSI core false dependencies on HCIL. Results in code duplication and/or avoidance of needed code.

* So far, it's an Adaptec-only solution. Since it pointedly avoids the existing SAS transport code, this results in two SAS solutions in Linux: one for Adaptec, one for {everyone else}.

* Maintainer reminds me of my ATA mentor, Andre Hedrick: knows his shit, but has difficulties working with the community. May need a filter if we want long term maintenance to continue.

Resolution
----------
AFAICS, there are two paths:

Easy path: make Adaptec's solution a block driver, which allows it to sidestep all the "doesn't play well with others" issues. Still an Adaptec-only solution, but at least its in a separate playpen.

Hard path: Update the SCSI core and libata to work with SATA+SAS hardware such as Adaptec's.

The hard path takes time, and won't be solved simply by shoving it in.

Jeff

No, it's the _other_ way around. There is NO existing SAS code.

My SAS code has been around _long_ before Christophs code.

Christoph's code is
* MPT based only,
* doesn't follow a spec to save its life,
* far inferior in SAS capabilities and SAS representation again, due to the fact that it is MPT based.

Since the whole point of MPT is to _hide_ the transport.

FUD! Why? It does _not_ use SATA code at all.

Why? SATA devices are discovered on the domain, but there is _no_ SATL in the kernel to represent them.

And libata is _not_ SATL. The reasons are that
libata is closely coupled to the hardware, i.e.
ata_port.

While in SAS open transport architecures, you'd have execute ATA/SAS/SMP task just as you have in aic-94xx. Why? Because all the chip is, is an interface to the interconect.

But I'm sure you know all this having looked at the specs of BCM8603.

No, not true. It _integrates_ with SCSI Core. The sad truth is that SCSI Core knows only HCIL.

Jeff, please be more specific.

Hmm, again: _architecture_. And you have to stop these
accusations: "pointedly avoids the existing SAS transport code".

I repeat again that I had this code _long_ before Christoph ever dreamt up SAS. And he got my code via James B sometime before OLS this year. I think he got it July 12, 2005.

The question is: why didn't _he_ use the solution already available?

You have to understand the differences between MPT and open transport architecture.

At some point I thought Christoph seemed to have understood it. Now I'm not sure any more.

Now since the open transport solution completely encompasses and _absolves_ MPT, it is not hard for an MPT driver to generate a bunch of events and use that infrastructure.

I take offence in your liking me to Andre -- I don't know Andre personally, but is seems that you're expressing personal opinion against him, against me and labeling me in some way.

I take offence in that, Jeff.

Why are you making this a _political_ and personal game?

All you're doing is trying to aliken me to someone and brandish me as someone I'm not.

This is rude, offensive and done in desperation.

Shall we concentrate on the _technical_ part of
the argument?

I repeat again: _technical_ part of the argument.

What _exactly_ does it mean "don't play well with others"?

Do you mean:
- the solution is far superiour to anything available now
- it presents the SAS Domain in sysfs as it looks in the physical world,
- does domain discovery and expander configuration,
- allows for complete control of the domain to user space,
- it pokes at SCSI Core's 20 year old relics.
- it's a thorn in the flesh to other people striving for similar functionality.

Or do you mean:
- it does not change a single line of code in the kernel,
- does not break anything existing,
- etc, etc.

I'm sure James Bottomley will move from SCSI Core to the block layer as he did for IDR. hehehe :-)

And no, it is not Adaptec's only solution. Your BCM8603 SAS LLDD when you write it could use it without any problems.

Cannot do for libata -- ever. Why? You know best: because libata uses direct access to the hardware! There is no layered architecture.

What you need to do is to write a SATL layer, just as you can see in SAT-r6, page 2, Figure 3. I'm on top of this already.

But in order to do this you need a unified architecture for accessing ATA devices.

Now libata, uses ata_port to do this, which is _hardware_ access.

The SAS Transport layer uses Execute SCSI RPC (defined in SAM, provided by aic94xx) to do this.

So in effect, what SATL would end up to be is a
data transformation function. But I digress...

No, you can do it now. It will not affect anyone or anything. (Other than hurting a couple of people's pride.)

The code doesn't alter Linux SCSI or anyone else's behaviour. It only _provides_ SAS support to the kernel.

Including it in as is would does not hurt anyone as you can see here:
http://linux.adaptec.com/sas/git/

Concentrate on the _technical_ merit, let's be more specific.

Luben

Luben,

Welcome to the club ...

You have managed to prove your knowledge base and forgot everything learned in kindergarden. Since I have kids, and spent time back in kindergarden, there are some valuable lessons there many have forgotten.

(working to get sanity back) is the Maintainer defines direction while following a bigger picture.

The issues wrt to HCIL v/s WWN v/s Multiplier v/s Target-Mode v/s blahblah blahblah are important questions.

Remember, everything about storage is a lie.

Help create a better lie by mapping into the design set forward by James and company. If the goal of Adaptec is to have support then they need to buckup and play be to rules at hand. Remember, most people are open to new ideas and better models. Propose one and you will find backing.

Since you are in the bay-area ... want to met who you have been associated with ... or is the idea to weird?

Cheers,

Andre

Hi Andre,

Yes, and I think you'll agree with me, the Maintainer isn't/shouldn't necessarily be the person "defining direction". The reasons are many but mostly:
* Documentation/ManagingStyle document (valuable read),
That document _really_ says it _all_. I suggest all developers, maintainers, and corporate _management_ reading this thread to read it.

Why should the Maintainer be "defining direction"? Is this some religous cult where "Beaver knows best?" (punt intended!) Or is this a theocratic society here at Linux SCSI? "Pi = 3" ;-)

The maintainership role is and has been _clearly_ defined! For the sake of eveyone else, take a look at 2.4 maintainer, 2.6 maintainer, other subsystem maintainers: defined by example and in word. So much unlike SCSI Core.

Another also great reason is:
* Complete, utter and infinite _incompetence_ coupled with too much pride.
It hurts us all.

When it comes down to it SCSI Core is 20 years behind and thus Linux Storage is 20 years behind.

Yes, this is indeed a very valuable advice. I hadn't ever looked at it like that.

What I thought, albeit erroneously, mea culpa, is that Linux actually _wanted_ to be "the storage OS of choice". Now I see and realize that due to neglected management, Linux has very little to do with _storage_.

Linux need to thank the mutitude of storage ignorant customers willing to pay the buck, because it is _Linux_ (put gasping sound here).

Linux _can_ be the "storage OS of choice", but this will not happen through neglect, or sheer incompetence coupled with lots of pride.

Back on topic: I'll try to keep up this "Linux storage lie" set forth by the james-bottomleys of the Linux community.

I never have found backing. It is all in the archives of linux-scsi mailing list. What happens is that even if people like your idea and write to you _privately_ to tell you that they like your idea, somewhere in their email, they tell you not to cross-post their message to the list because they have storage products which need Linux. The reason for this is that James Bottomley has established this politics that if you don't agree to his antics, your patches are never ever going in. Why else do you think IBM-ers agree with him that Linux SCSI doesn't need 64 bit LUNS?

_But_ I'm willing to try _again_. So I'll be proposing something later this morning. (You know, all engineers are optimists.)

Luben

Dude, that document is written in a very tongue-in-cheek style. There's a few clues:

"the preferred (or made up, depending on who you ask) management style"

"this document may or may not have anything to do with reality. It
started as a lark"

It's really not a club for you to beat James with.

True, true. But sometimes you can say painful truths more easily if you do it as a joke. Most of the ManagementStyle document is perfectly valid.

Whether it has any bearing on this discussion is another matter ;)

Linus

Yes, I thought I understood it when I read it first, but I later realized that my understanding was very superficial.

When I re-read it now, I cannot help chuckling, remembering how you kept saying "go wild", "make it so", "That is good, but it strikes me that there is no fundamental reason to limit ourselves to ..." on the git list ;-).

Please stop repeating that, no one said we should *not* implement a 64 bit LUN in linux scsi, and James posted a patch for 64 bit LUN. I posted a clarification in response to your earlier postings, you seem to have ignored or forgotten this post:

http://marc.theaimsgroup.com/?l=linux-scsi&m=112671847228275&w=2

I said:

"I am talking about the scsi spec, not the code. IMO linux scsi code should support W_LUN and 64 bit LUN."

-- Patrick Mansfield

Sorry.

I was refering to the bottom of this email:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112665156918643&w=2

Luben

Hi Patrick,

You have hit on one of the key word of my downfall.

Specifications!!!

I believe in them and they are the inflexable state machine which all OSes are required to address. Linux has a very bad history of avoiding the boundary conditions related to storage.

I am for following the rules of the spec, and will bet Linus would now agree more so than before. The problem is SCSI is a strange beast without a formal FSM. It is more of a BusPhase psuedo stated transport. It is smart enough to laugh at bad software designs and keep going. Sheesh, look at M$'s miniport.

This leads me to a point where a similar (but smarter) miniport could look interesting. However, this is also where the transport classes have their bases, afaics. Anyone please correct me where I have mistated (other than Linus, :-p).

Luben, I have a vested interest in seeing SAS run via SCSI. So this means you have one ex-demi-god from the world of maintainers looking to pull you have towards the current path and open to ideas and willing to back a better design and push it.

Cheers,

Andre Hedrick
LAD Storage Consulting Group

Me too. I live and breathe by them.

Me too.

An interesting thing which "the community" would appreciate is that M$ has aggressively started to "go by the spec" as far as SCSI is concerned.

Ding-ding!

Oh, no, no, no! So much has changed Andre.

Just take a look at SAM, and I'm sure that you'll appreciate the object oriented design, the abstractions, etc. Really!

Recently all new protocols follow _explicit_ state machine definitions at each layer they define, and how it interacts with the layer above and below again by FSMs. It's all a good thing.

Ok, thanks Andre. Much appreciated.

You are the first person to back me up _publicly_. Now if we can find a person from "the community" to do that, and get all the other people who've written me _privately_, we'd be in good shape.

Luben
P.S. Not sure if you have seen this link:
http://linux.adaptec.com/sas/

Hi all, hi Luben,

On Wed, Sep 28, 2005 at 04:56:20PM -0400, Luben Tuikov wrote:
(...)

I'm sure I'm not one of those who qualify best here, but having largely contributed to Linux acceptance at critical points at a handful of big customers here, I'd like to send some general feeling I get from there.

There are people buying expensive hardware. The type of hardware that costs $100k for just a few CPUs. Those people don't buy "the Adaptec XXX which runs best with Red Hat Enterprise" nor the "LSI YYY which runs best with Solaris", they buy a few $100k systems with 3 TB disk to store their monthly logs. They cannot even imagine that the hardware in the $100k system will not be fully supported by some recent OS, that's just plain silly. The OS cost is just a water drop in the middle of this. When they install Solaris on it because they're used to run it, it just works. When I sometimes just show them that Solaris is not adequate for daily greps in logs, and I show them how faster it is on a $1k Linux machine in the next rack, they feel they can switch to it easily. But if they discover that this system does not correctly support their $100k hardware, do you know which one was is the crap ? the $300 Red Hat or the $100k hardware ? [ oh, BTW, I forgot to tell you : they say "Red Hat", and not "Linux" because it does not sound like what they long considered an OS for tamagotchis - to use their own words - :-( ]

And when they go to adaptec site to find latest drivers and they only find patches which forces them to find another Linux to install the sources and guess how to patch and build, do you know which OS they consider as hobbyist's ? The Red Hat ! (which they can call "Linux" again then).

For those reasons, I could put Linux there on very specific places, mostly firewalls and load-balancers. A self-made distro with kernel 2.4 with patch-o-matic still is one of the most powerful and reliable firewalls around, and at only a few bucks. The same hardened install is used for load-balancers with my proxy. Seeing that the proxies have been stable for years, they bought between 50-100 RHEL licences. But that's all. Nowhere you will find anything serious using Linux in other areas. It's already a nightmare for them to get a hardware RAID card working while they just have to click on stupid icons in Windows to do the same. Right now, new Linux-based machines are turning back to plain IDE. SCSI was too much boring for them and not needed to do just networking. Period.

When they will buy hundreds of TB of SAS-based racks in the next few years, and they will learn the hard way that Linux does not even see them as disks, it will be too late to give my preferred OS a second chance.

Hans Reiser once said that every software needs a complete rewrite every 3 or 5 years (I don't precisely remember). I tend to agree with him. Maybe it's time to completely rewrite the SCSI subsystem, but maybe it will be too long, too risky and not worth the effort. Maybe it can simply coexist with another new subsystem. This is what Luben seems to have done : break no code, just bring a new subsystem which should not even give the SCSI maintainer too much work if he maintains it himself. At least I could understand some arguments against Reiser4 because it touched the VFS, but here we have a perfect example of something new which can live next to SCSI and probably some time will replace it definitely. Jeff has done this with libata (it even had to touch some pieces to coexist with piix4 and probably a few other chips).

Having read the discussion from the start here a few days ago, I believe that Luben maybe has not explained well to non-competent people like me what the goal of his work is. I've looked at the GIF on T10.org, but I think that the equivalent with what it currently implemented in Linux would be worth doing. Maybe we would even notice that current maintainers cannot agree on a same representation. Maybe it will enlighten some of us about the poor error reporting, the reasons why USB storage sometimes fails to assign a device when plugging a flash, etc...

Luben, you seem to have enough knowledge to draw both diagrams side-by-side :
- the T10 spec with colors on the boxes covered by your work
- the Linux view of SCSI

Perhaps we will see that current code can be extended without too much work, or perhaps we will see that it's definitely a dead end and that a new design will help a lot.

Anyway Luben, I fear that for some time, you'll have to provide pre-patched sources as well as binary kernels to enterprise customers who still try to get Linux working in production. I hope that this sad experience will not discourage other vendors from trying to take the opensource wagon, as it clearly brings fuel to closed-source drivers at the moment (no need to argue).

Eventhough I don't have SAS, I sincerely hope that a quick and smart solution will be found which keeps everyone's pride intact, as it seems to matter much those days. In an ideal situation, 2.7 would have been opened for a long time, and Luben's code would have been discussed to death as a new development needed to be merged before 2.8. Right now, as 2.7 is 2.6.<odd>, probably that ideas can gem before 2.6.15.

Just my personnal thoughts
Willy

PS: please don't CC me in return, I can read LKML. I haven't trimmed the CC list as nobody has complained yet.

Adaptec is unfortunately a special case. QLogic and other enterprise vendors get along with quite well on $100k machines.

Both Luben and his predecessor, Justin Gibbs, were severely dissatisfied with the SCSI core. Often they have raised valid issues that need addressing, but their choice has been to work around or ignore existing code (and maintainers), rather than work with it, and fix it.

Hardly. SAS support is coming, whether from Adaptec or someone else.

The current maintainers seem to agree on the path to transport independence.

Yes, let's not argue this silliness. Other vendors don't seem to have this level of problem.

Sigh. This is not about pride. There's already a path to fixing up the SCSI core to work nicely with SAS (and nicer with FC/iSCSI). Changing to a new path midstream, in the middle of addressing the stated problems, causes more delay, more harm than good.

Jeff

From: Jeff Garzik <[email protected]>
Date: Wed, 28 Sep 2005 19:22:53 -0400

I'm in violent agreement here.

Justin was just as anti-social of an engineer as one could get. And, when you put an ex-FreeBSD guy onto Linux driver maintainence, what in the world could anyone expect. :-)

For example, instead of accepting that the symbol "current" is a reserved symbol when compiling under the Linux kernel, he decided that "sticking a square peg into a round hole" was a better way to deal with this, and thus he put an "#undef current" into the adaptec driver instead of simply renaming a structure member from "current" to something else.

I don't know how else to define "control freak".

Dave,

Was it really necessary to this far and rude?

I heard the snow is falling and there is a fresh shipment of hash headed out to the slopes, don't be late.

Not sure who to credit the following to:

When TOE's were introduced to Linux, there was a violent rejection of this hardware because Linux is superior in the NetStack than any other possible NetStack every created.

The point is there is a known history in Linux to reject things which steps on peoples' egos.

Have a great ski trip.

Cheers,

Andre

From: Andre Hedrick <[email protected]>
Date: Wed, 28 Sep 2005 22:30:58 -0700 (PDT)

In that case, it is indeed a vendor trying to shove their particular solution down our throats. They never even attempt to try out the alternatives, and we've even gone through the trouble of coming up with several. And they do this because their whole buisness model is all about their scheme to the exclusion of anything else, not because what they have is better.

On Thu, 29 Sep 2005, David S. Miller wrote:

Dave,

Thanks for filling in the details I was missing about the TOE history.

Glad to see you laugh about the ski stuff. I thought about snowboarding then realized I would make a great mogal for someone to do an ali off. If you are open to questions about TOE/RDMA stuff, would like to chat with you and see your POV on the subject.

Luben,

Reading Dave's points above tends to point to adaptec's current direction, as we all know. TOEs were rejected.

I stated I would help with SAS adoption because there is a SAS-Transport model. I asked about a possible libadaptec + libsas, and still waiting to see if you and adaptec are up for the task. Right now the only path open is the one Jeff Garzik is putting forward along with James and Christop. I have a vested interest in seeing SAS-Transport, otherwise I would have cut and run a long time ago.

These long email threads where everyone is shout from the top of their hill never wins anything. After a while the hill becomes flat (from all the stomping), and you become old and tired.

LSI pointed out they mask there SAS in firmware and make it show up in a scsi-like or scsi state. They also pointed out other vendors have taken this road. Even if Adaptec did not go this way in hardware, there still has to be a way to map into SCSI ... sheesh this is Adaptec known for SCSI.

Just an FYI, would suggest you cool your heels and listen for the quiet responses. There is more heat than light right now; maybe this thread will offset some of the cost in the energy criss. Will pass on advice handed to me (when I was a maintainer) relax and listen, nobody is out to get you (and they were right).

Cheers,

Andre

PS I didn't listen to that advice back then, don't make the same mistake.

Hi Andre,

Let me know if this 4 section write up satisfies:

Section 1: MPT, SCSI Core and JB's "transport attributes"
---------------------------------------------------------

SPI drivers (say 5 years ago)
-----------------------------
hardware implementation (bus connect)
firmware implementation (transport: SPI)
LLDD (exports SCSI devices (LUs))
SCSI Core
Command Sets

MPT-based drivers (now)
-----------------------
hardware implementation (interconnect/physical)
--> Transport Layer (firmware: FC/SPI/SAS/etc) <--
LLDD (exports SCSI devices (LUs))
SCSI Core
Command Sets

As you can see SCSI Core is _unaware_ of the transport. The transport is completely implemented in FIRMWARE, relieving the LLDD from worrying about it. Theoretically/ideally the same LLDD would work for _all_ transports, since the FW would export LUs to the LLDD, which would in turn register those with SCSI Core.

The layout is the same as before, achieving 100% backward software compatibility.

MPT-based drivers + James Bottomley's "transport attributes"
-------------------------------------------------------------
hardware implementation
Transport Layer <------+ FW/Transport dependent access method
LLDD <---------- + LLDD dependent access method
SCSI Core ---------------------'
Command Sets

This picture is _identical_ to the one above, but I've also shown what the "transport attributes" achieve:
They hook to the _LLDD_ to get to LLDD dependent way of accessing "transport attributes" (any transport). This is JB's template unifying _all_ transports.

Note that this isn't a _management_ layer or infrastructure, since, it _does not lie between_ the LLDD and SCSI Core. It merely implements attribute exporting.

Section 2: USB/SBP/SAS and SCSI Core
------------------------------------

hardware implementation (interconnect/physical)
firmware implementation (SDS: TMFs + Execute Command SCSI RPC)*
LLDD (Coherent interface to SDS)
--> Transport Layer (Transport common to LLDDs) <--
SCSI Core
Command Sets

* SDS, Service Delivery Subsystem (see SAM 4.6) TMF, Task Management Function (see SAM 7) Execute Command SCSI RPC (see SAM 5.1)

Most immediate difference from Section 1 is that
- the Transport Layer is _above_ the LLDD (in top-down).
- no need for LLDD/Firmware dependent access method to the Transport,
- Transport is accessed in the same way across all LLDDs of the same transport (USB/SBP/SAS).

Now since, the LLDDs all implement TMFs + Exec Cmnd SCSI RPC, you can have
- a transport common error handling and
- a transport common "attribute" access (sysfs),
across all LLDD of the same transport directly from userspace.

The reason the LLDDs all implement TMFs + Exec Cmnd SCSI RPC, is that the Firmware implements it, and the reason that the Firmware implements it is that the chip implements it (following the transport spec).

That is, you can have _different_ LLDDs connecting you to the _same place_ (maybe not at the same time, depending on the transport).

SCSI Core should be completely unaware of the Transport being used and Transport specific
Error handling is common to all such Transport specific LLDDs (via TMFs): one for SBP, one for USB, one for SAS.

The whole point of this USB/SBP/SAS Transport Layering is that you can access at each level, thus you can add new abstractions, e.g. SATL (libata), as is shown in the SATr06 spec.

Furthermore, when you look at the USB/SBP/SAS chips you will see nothing about "scsi_host", and all about "transport".

And the sysfs representation of the SAS domain is analogous to, say, the USB representation of the USB domain.

Section 3: Legacy Thinking or Thinking Legacy
---------------------------------------------

Traditionally Error Handling has been done in SCSI Core. The reason for this is that the first and only SCSI was Parallel SCSI and no other SCSI was around (and that SAM didn't exist back then).

This is how we have the SPI-centric EH methods
in the scsi host template right now:

	int (* eh_abort_handler)(struct scsi_cmnd *);
	int (* eh_device_reset_handler)(struct scsi_cmnd *);
	int (* eh_bus_reset_handler)(struct scsi_cmnd *);
	int (* eh_host_reset_handler)(struct scsi_cmnd *);

This is fine for Parallel SCSI (SPI) but for other transports it doesn't quite satisfy.

Let's admit it, with HCIL and with the EH, SCSI Core is still very SPI centric.

But we should _not_ break legacy drivers and backward support, yet we have to face the future.

The way we do this is we slowly, without disruption to older drivers introduce, in parallel, emerge a new, simpler, slimmer, faster SCSI Core, whereby we accommodate new infrastructures, yet, have 100% backward compatibility, via the current older SCSI Core. After all, both would be a bunch of functions in a bunch of files.

If X works with Y, do not disrupt it. Fix it if it's broken. Introduce innovation, functionality, better design, but not at the expense of breaking legacy.

To eliminate bugs, tweak old as little as possible, since you cannot test on _all_ hardware the old code works on.

To achieve maximum innovation and relevance in the fast changing world of storage, create, innovate new better, more accommodating design and infrastructures.

Section 4: Politics
-------------------

Let's face it: SAS is a new emerging technology. It will be the technology for the next 10-15 years, and *everybody* in Linux SCSI wants a piece of it. Everybody wants their name and contribution to it.

This is fine, but we need people who clearly understand the technology and clearly understand what, how and why it works. We need well-read and well educated people. Linux dedication is fine, but protocol knowlege is needed too.

Can Linux afford people who have never even read SAS to write SAS Code for Linux. Yes, sure. It is the Linux's ideology: "specs are cr@p".

Conclusion
----------

Even though the SAS Transport Layer follows an _already_ establised layering infrastructure for more (less?) exotic transports such as USB and SBP, James Bottomley has resisted its inclusion in Linux SCSI.

Have we lost our touch with the new calling it "non-traditional"? Will Linux become a dinosaur?

I'm not sure how many times one can split the SAS code? Will there be enough for everyone?

Thank you Andre for the warm advice.

James, Linus, can we have this driver in the kernel now, please?

Thank you,
Luben

So submit patches to fix it! You clearly understand what is wrong, so why not help change it?

WRONG. This is not the way Linux works. We break internal APIs all the time. If you need to change one _thats_OK_. Userspace ABI is mostly another matter, but that's different from the internal data structures and functions.

Except this introduces bloat and multiplies maintainer load. Fix the existing one. If it breaks other in-core drivers, fix those to match. If it breaks out-of-tree drivers, too bad! They should get their code upstream and then they wouldn't need to worry.

This is not the way Linux works. It may be the way Adaptec works, but that's not relevant here.

s/Politics.*//g; I hate politics. Keep it off this list.

Cheers,
Kyle Moffett

Because
- I do not want to give heart attack to all existing LLDDs
- Some LLDD would never be able to be changed
- Some LLDD work on very _scarce_ hardware which we cannot test.
- plus such radical changes are neither warranted nor necessary.

It is better to keep legacy around, until all you'll have on your new serverboard is a SAS/SATA storage chip such as AIC-94xx or say BCM8603. Then you can compile out most of the legacy stuff.

Well, I can never have it right. Some people say you shouldn't break it, others say let's break the whole thing and give heart attack to all LLDDs, other say it is impossible to change all LLDD since the hardware is not around, etc, etc.

I think not breaking anything (for now at least) would be the _easiest_ and most painless way to transition.

Well, not necessarily. It would be more painful and more maintainer load if we did what you suggest. The overhead would be enormous.

Me too, but we are idealists. Politics is an integral part of life.

Long time ago, in a galaxy far, far away...
I literally sat in a meeting whereby technical staff of _several_ companies agreed that Pi=3.0.

Luben

Significance of change is not an issue, assuming the change is broken up into a collection of small obvious changes as highlighted in Documentation/SubmittingPatches

Why not? It's easy to change APIs, even stuff as invasive as the VM, the device driver model, etc, and those get changed all the time.

You don't have to worry all that much about testing. If your patches are small and obviously correct (like they should be), then they will get enough review during submission that there will only be a very small number of bugs. The few remaining bugs will probably be ironed out in -mm. In any case, if nobody uses hardware anymore, eventually the driver will get sufficiently crufty and out of date that it will be recognized as such and removed.

Jeff Garzik et. al. seem to think that they are necessary, and I agree. You don't need to fork SCSI-core; doing so would just double the maintainer load, and _that_ is neither warranted nor necessary.

Precisely. When nobody uses the legacy drivers to the point that they aren't fixed or maintained anymore because no-one reports bugs, then said drivers can be removed from the kernel entirely, along with any support code. The model I describe here works better because it keeps a _single_ clean core subsystem, and leaves any lack-of-maintenance crap in the old drivers.

Until somebody wants to add a new high-level SCSI feature that works for both the new and the old devices. Then they have to do it _twice_.

So you're saying fixing the current SCSI subsystem once *now* costs more than applying all *future* SCSI fixes to _two_ SCSI subsystems, handling bug reports for _two_ SCSI subsystems, etc.

Politics are not an integral part of productive technical discussions, though. If you discuss technical topics and provide realistic technical descriptions, examples, reasons, code, etc, then politics tends not to matter in the discussion, and we're all happier people.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

Luben has more than once called for adding a small number of additional calls to the existing SCSI core. These calls would implement the new (reduced) functionallity. The old calls would continue to support the full SPI functionallity. No existing LLDD would need modification.

Then, over time, more radical restructuring could be done that pulls SPI out of SCSI core.

I believe this proposal is what he was talking about, not the brand new simplified SCSI core that has been discussed recently in this thread.

Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

See? There _are_ smart people out there.

Luben

IOW, what Luben wants is:

if (Luben)
do this
else
do current stuff

If this is the case, why bother touching drivers/scsi/* at all?

Regards,

Jeff

that's the reason why I proposed that this moved to drivers/sas/* or somewhere else so that there is no maintaining conflict.

Regards,
Willy

Yes, it has been moved already there.
http://linux.adaptec.com/sas/

Luben

In order to have a new better, faster, slimmer SCSI Core.

Luben

I've been contending this since before Jeff started work on libata. But none of the ideas: 64 bit LUN, HCIL removal, etc., were accepted with "submit a patch".

I'm saying that the current "old" one is already obsolete, when all you have is a SAS chip on your mainboard.

All you need is a small, tiny, fast, slim SCSI Core.

Yes, please re-read this thread, and open and read all the references I've included to SAM, SPC, SAS and SAT of T10.org.

Politics: "Nah, whatever you say, specs are *crap* and we'll do it our way. We are not interested in your way, even if it were better. Oh, and BTW, REQUEST SENSE clears ACA and LUN is a u64."

See?
Luben

I concede this may have been the response in the past. Its not, now.

Then don't use it at all. Write a block driver, if you really feel we need two SCSI cores.

This is a misrepresentation. -We- understand the stuff you have posted.

But you continue to demonstrate that you simply do not understand the existing SCSI core code.

The SAS transport class supports commonality across all SAS implementations. This includes both MPT and Adaptec 94xx.

SAS transport class + libsas supports software implementations of SAS, including transport layer management. This includes Adaptec 94xx but NOT MPT.

Jeff

You almost get it right, other than the layering infrastructure.

The SAS Transport Layer is a layer in its own right. It is not a "libsas".

MPT and open transport a very different, one hides the transport, i.e. the transport layer is in firmware; the other exposes it and needs a transport layer. See the pictures here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=112810649712793&w=2

Luben

Different open transport hardware will expose the underlying network at different levels.

Hardware A might provide direct access to SSP and IDENTIFY/CONNECT frame handling, while Hardware B may handle that internally, while still providing full SMP access for the transport layer to discovery topology and build its routing table.

We cannot assume that all open transport hardware will function the same, hence the better approach is a library of helpers that function in concert with LLDD to form a transport layer.

For the 1001th time, we know all this.

Jeff

A while ago I told you that if you could show that the transport class abstraction could not support both the aic94xx and LSI SAS cards then we could look at doing SAS differently. Since then you have asserted many times that a transport class could not work for the aic94xx (mostly by making inaccurate statements about what the transport class abstraction is) but have never actually provided any evidence for your assertion.

So very soon now, with proof by code, we shall have confirmation or negation of that assertion. I'll wait quietly for this to happen, and I would suggest you do the same.

James

I have, by example and by spec. They both use two different and distinct architectures.

Now _you_ have decided that this isn't the case, simply because you want to prove your own idea. (which breaks SAM layering)

I think I explained the similarity of the SAS Transport Layer to *USB and SBP*.

As I said: everyone wants a pice of the SAS code.
You do too.

I'm sure you'll do whatever humanly possible to show that _your_ idea can be applied: you can do this now:
just use a big if () { ... } else { ... } statement and you're done.

Furthermore I do not see the reasons to umbrella both "aic94xx and LSI cards" when they are _completely different_ in architecture: MPT and open transport (ala USB Storage and SBP).

The code is here:
http://linux.adaptec.com/sas/

I just thought it was only fare to the rest of the community to know how far the politics has escalated.

Everyone wants a piece of the SAS code.

Luben

This is not how we do things in Linux. You're doubling the maintenance burden.

If you really want to do this, at least don't fill up drivers/scsi/ with an additional, completely unrelated codepath.

There is commonality between aic94xx and MPT/LSI stuff. aic94xx SAS transport layer is a superset of MPT/LSI SAS transport: it clearly needs far more management code.

We understand this. The part you don't understand is that we want to emphasize the commonality, rather than let aic94xx and MPT/LSI go in completely different directions.

Read it again: aic94xx/BCMxxx is a superset of functionality, not completely different.

Jeff

No necessarily. See below.

How do you say it is unrelated?

Is USB storage unrelated? (other than the fact that it doesn't live in drivers/scsi/)

And MPT/LSI SAS does not need this managament as this layer is completely implemented in FW and not exposed (and for a reason).

But this was LSI's decision, remember? We did work together, until LSI and Dell decided that they'd rather let Christoph do it. (Since who cares about the technological merit of the code when it will be accepted into the kernel?)

Now you want to integrate the two? Apparently LSI and Dell haven't made up their mind.

As I said: Vendors are completely playing to the tune of a couple of people at linux-scsi, for this reason we haven't seen _any_ SCSI or Storage _innovation_ in SCSI Core.

As opposed to those vendors saying: We _really_ need 64 bit LUNS and it would be really nice to get rid of HCIL, etc, etc.

If all vendors pushed for that and were not afraid to speak up (because they have drivers to write and patches to submit and want acceptance), then SCSI Core would be a better place.

IMO, 64 bit LUNs and no HCIL is more important than "transport attributes" and should've _preceded_ them.

The fact that you're trying to umbrella them together, doesn't make it _technologically_ correct.

Remember, a person's fall starts when they're surrounded by "Yes" men.

One implements all transport related tasks in FW and exposes only LUs to the LLDD, the other implements only the interface to the transport in the chip (the interconnect), and the rest is handed to upper layers.

If you sit down with a clean sheet of _paper and a pencil_ and try to draw out the layering infrastructure for both and how they interface with SCSI Core, you'll see that with MPT, things are _upside_ down compared to USB/SBP/SAS. Now trying to reconcile both, while possible, would be extremely _ugly_, unless say, you can fake out event formation in an MPT based LLDD, but then again, you'd need to resolve the host template thing...

It would just be extremely ugly and not as flowing and straightforward as the current code is.

Luben

Everybody agrees with this 100%

Note that James submitted a sdev_printk() patch in the past few days, which assists in removing HCIL from the SCSI core.

Agreed. But that's a lot of work, and I doubt people want to delay SAS further to wait for complete HCIL elimination.

It's muddled together, yes. That doesn't make it any less correct. It's just not a clean, immediate separation. Its a separation that takes time.

Jeff

No true, many of us could careless about credit list.

Sometimes teaching others is a better way to bring them around to your view point while learning about their goals. Since everyone/most agree your T10 knowledge is strong, others have pointed out you are savy enough to work around problems over fixing them in general. Blah blah ...

Just show everyone why it can not work the "Linux Way" and defend the points logically. Should the defense of it not be possible, then the point is lost.

SPECs become crap when organizations like STA with T10 make joining and voting on the technology under NCITS $10,000.00 annual membership. This is old boys club rules, otherwise I would have joined and terrorized T10 like I did to T13. Exclusion breeds distrust.

If this is accurate then it is fair to raise a question; however, few can stand in judgement because they were present in for the discussions. The remander (self included) are in the dark.

Cheers,

Andre

I wasn't talking about you. They know who they are.

Thank you Andre.

I think I have. All the references to T10, all the pictures to the email you replied to which you deleted, all the words.

Well, the T10.org reflector is open to everyone.

Of course voting-membership is expensive. And it should be so. They don't want every "Joe" (no offence to Joe or to LKML or LSML participants) to participate.

If you are a voting member then you really have a reason, you really know what you're talking about and voting for and you really are serious. It means that you or your company care.
It also becoms quickly enough obvious if you know your stuff after a meeting or two.

Luben

You continue to fail to see that a transport class is more than just transport attributes.

You continue to fail to see that working on transport class support IS a transport layer, that includes management.

Is you don't understand this fundamental stuff, how can we expect you to get it right?

Jeff

JB's "transport attributes" cannot satisfy open transport.

The reason is that MPT-based drivers indeed do need host template in the LLDD.

Open Transport (SBP/USB/SAS) do not, since the chip is only an interface to the transport.

The host template is implemented by a transport layer, say USB Storage or the SAS Transport Layer.

This allows you to do great things, like layer
intersection.

Luben

Repeating verbatim the above quote: a transport class is more than just transport attributes.

Every chip is ultimately an interface to the transport, regardless of whether the transport layer is largely managed by software (aic94xx) or firmware (MPT). SCSI host template can work just fine with open transport hardware.

Jeff

a) "Transport Attributes" _is_ its name,
b) It sits across SCSI Core, i.e. on the same level.
c) It was never intended to add management.
d) Its inteface to SCSI Core is badly defined and an "invention", (and very poor at that).

The reason for d) is that
1) it tries to unify different _transports_,
2) does _not_ follow _any_ spec or standard.

Look at this, while you repeat verbaitm a single quote, I give you technical arguments, then you just repeat a single quote verbatim... Sad.

Maybe the picures and the write up here will help:
http://marc.theaimsgroup.com/?l=linux-kernel&m=112810649712793&w=2

Luben

No, transport class is its name. Think about a standard object-oriented paradigm. Each transport has unique characteristics. The proper place to export these and manage these characteristics is the transport layer, as SAM agrees. The manifestation of the transport layer is the transport class.

You have to look beyond the current code, to see this.

An implementation of a transport class, in conjunction with helper functions common to all transports (SCSI core), combines to form the transport layer for a specific transport.

SCSI core is nothing but a set of helper functions and support code that enable the transport class and LLDD to implement the transport layer.

Strongly disagree. This invention is defined by -needs-, as is other code in Linux. If we have new needs, we change the code.

That's fine, since its an internal kernel API.

Jeff

Not according to Kconfig:

menu "SCSI Transport Attributes"
depends on SCSI

config SCSI_SPI_ATTRS
tristate "Parallel SCSI (SPI) Transport Attributes" depends on SCSI
help
If you wish to export transport-specific information about each attached SCSI device to sysfs, say Y. Otherwise, say N.

config SCSI_FC_ATTRS
tristate "FiberChannel Transport Attributes" depends on SCSI
help
If you wish to export transport-specific information about each attached FiberChannel device to sysfs, say Y. Otherwise, say N.

config SCSI_ISCSI_ATTRS
tristate "iSCSI Transport Attributes"
depends on SCSI
help
If you wish to export transport-specific information about each attached iSCSI device to sysfs, say Y. Otherwise, say N.

config SCSI_SAS_ATTRS
tristate "SAS Transport Attributes"
depends on SCSI
help
If you wish to export transport-specific information about each attached SAS device to sysfs, say Y.

endmenu

But:

Bzzzt! Wrong.

This is where you and James Bottomley make the wrong assesment.

Having the SCSI host template in the LLDD tells everyone immeditealy that you have it all wrong as far as anything SAM/SPC is concerned.

Look at any transport spec, how little SCSI Host template is there.

You need to understand that declaring the host template in the LLDD is, and has always been, the _exception_, not the rule. The reason is legacy Parallel SCSI.

It is MPT based drivers who are the exception, not the rule, and JB's "transport attributes" makes it the rule, and a transport layer (what you call libsas), the exception.

It is wrong for a host template to "branch out" to a transport layer as you're doing it now. Think about it. The layering infrastructure is upside down.

Eh, well, not that you put it like this, I expect that you'll completely pull out the implementation from my code and put it in the "transport attributes" and call it your own.

I don't want to look beyond the current code, I'm discussing things now, as they are. How many times is JB going to change the "transport attributes" just because FC needed one more thing or because of, as often has been recently seen, "brown paper bag" fix?

Should:

The transport layer sits below SCSI Core and above the LLDD. SCSI Core is completely unaware of the transport layer. The LLDD communicates with the transport layer via the event interface (as shown in the SAS Transport layer) and the transport layer communicates with the LLDD via the Execute Command SCSI RPC and TMF. All as outlined here:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112629423714248&w=2

Again you fail to see that the LLDD should _not_ implement the host template as has been traditionally done. The reason with this is that simply the LLDD has nothing to do with the host template. Unless you're dealing with MPT-based LLDD which implement everything in FW and export LUs to SCSI Core (which is fine too, as long as they do not dictate how SCSI Core should work).

Notice that for MPT-based solution, SCSI Core wouldn't even do LU discovery (unless you can turn that off via FW dependent ways).

You defence of your friends architecture is duly noted. But if, as has been pointed out, you take a look at the evolvement of this "invention" you'll see how it was and still is _meandering_ in the technological maze to find its place.

All the while a transport layer like USB/SPB/SAS and maybe iSCSI just sits "right". Think about it.

Now, lets shove it down the throat of eveybody.

You forgot to say something about
1) it tries to unify different _transports_.
I don't expect a blurb on that, btw.

Luben

For FC there is code like the fc_rport stuff which exports a sysfs interface but also does a lot more like probing and queue blocking.

And the iSCSI class does do a lot more now too. This just has not been updated. It handles the userspace to kernel communication, session and connection sysfs interface setup, and there were some patches that added the ability to tell scsi-ml to stop queueing commands to a iscsi driver.

Structures like the fc_rport and iscsi_session are managed by the transport classes so scsi-ml never knows about them (except for that scanning bug). And it is possible to share them between HW and software or partial software solutions. I do agree for some iscsi sitautions having a layer over the eh or command exection where the transport really is more of a layer like your design would be nice (I am not refferring to the code duplication though becuase iSCSI would like some of yrou fixes :), but at the same time there are places where code can be shared between a interface that hides the lower level details and one that implements them in software. Maybe it is not this way for SAS though.

Great. Exactly.

Adaptec's SAS stuff implements the standard high level SCSI model as an API, which is a pretty decent direction for SCSI long term: provides a transport-independent execute-scsi-command high level hook (along with the other task management functions), and hides the transport details for the most part.

That's fine, and works for iSCSI as well.

I just disagree that we need to have two concurrent APIs for SCSI, completely separate from each other, and mildly duplicating each other's code.

The current SCSI code will morph in the proper direction given time and love. Segregating generic [SPI | FC | SAS | iSCSI | ATA | RAID | ...] transport layer code into their own modules -- the transport classes and associated libs -- will point out the sore spots where scsi-ml needs changes. This approach also implies we improve the existing scsi-ml where needed, rather than the proposed "if legacy, let it bitrot" approach.

Jeff

I think I can understand some of Luben's reasons for the layering. We are facing similar problems with software iscsi and hw iscsi. For software iscsi it would be nice to consolodate some of the common software iscsi code into a layer or lib. Following Luben's path for example our queuecommand would be:

scsi-ml -> scsi_host_template->queuecommand -> iscsi transport common queuecommand (do things like check session state, that we are not in session level recovery, scsi to iscsi pdu prep like setting the data direction, and other iSCSI PDU prep) -> iscsi_transport module -> iscsi_transport->queuepdu (you can probably reccomend a better name) -> tcp, sctp, iwarp, or some iSCSI HW that exposes a iSCSI interface rather than SCSI (note that qla4xxx would use its own scsi_host_template->queuecommand since it does not expose enough iSCSI internals for it to be useful to plug in here).

However, HW iscsi cards and software/partial-software iscsi solutions can share code for things like session and connection creation where we would have transport class lib functions: iscsi_add_session/iscsi_remove_session which both the HW iscsi cards like qla4xxx and software/partial-software iscsi drivers could use to setup things like a common sysfs representation. iscsi_add_session/iscsi_remove_session would work similar to the fc_rport code where the midlayer doesn't really know they exist (this is similar to our session and connection code today but it is bound to the scsi host which prevents qla4xxx from using it).

Is the direction we are going where iscsi would have to put the "iscsi transport common queuecommand" code into something similar to libata? Or
is it that Luben's transport layer code is performing something
different than software/partial-software iscsi?

Hi Andre,

Let me know if this satisfies:

The infrastructure is broken into
* SAS LLDD,
* SAS Layer.

The SAS LLDD does phy/OOB management, and generates SAS events to the SAS Layer. Those events are *the only way* a SAS LLDD communicates with the SAS Layer. If you can generate 2 types of event, then you can use this infrastructure. The first two are, loosely, "link was severed", "bytes were dmaed". The third kind is "received a primitive", used for domain revalidation.

A SAS LLDD should implement the Execute Command SCSI RPC and at least one SCSI TMF (Task Management Function), in order for the SAS Layer to communicate with the SAS LLDD.

The SAS Layer is concerned with
* SAS Phy/Port/HA event management (LLDD generates, SAS Layer processes),
* SAS Port management (creation/destruction),
* SAS Domain discovery and revalidation,
* SAS Domain device management,
* SCSI Host registration/deregistration,
* Device registration with SCSI Core (SAS) or libata (SATA/PI), and
* Expander management and exporting expander control to user space.

The SAS Layer uses the Execute Command SCSI RPC, and the TMFs implemented by the SAS LLDD in order to manage the domain and the domain devices.

For details please see drivers/scsi/sas/README.

The SAS Layer represents the SAS domain in sysfs. For each object represented, its parent is the physical entity it attaches to in the physical world. So in effect, kobject_get, gets the whole chain up on which that object depends on.

In effect, the sysfs representation of the SAS domain(s) is what you'd see in the physical world.

Hot plugging and hot unplugging of devices, domains and subdomains is supported. Repeated hot plugging and hot unplugging is also supported, naturally.

SAS introduces a new physical entity, an expander. Expanders are _not_ SAS devices, and thus are _not_ SCSI devices. Expanders are part of the Service Delivery Subsystem, in this case SAS.

Expanders are controlled using the Serial Management Protocol (SMP). Complete control is given to user space of all expanders found in the domain, using an "smp_portal". More of this in the second and third email in this series.

A user space program, "expander_conf.c" is also presented to show how one controls expanders in the domain. It is located here:
drivers/scsi/sas/expanders_conf.c

Thank you,
Luben

Yes, all true.

Sadly, most developers here do not _use_ Linux, or are at least shielded from using Linux, in a _business_.

All the while, Linux _development_, seems to follow some kind "yours/mine", "gimme that/take that" kindergarten kind of way. The reason I'm saying this is that _every_ successful entity (person, corporation, etc) knows that in order to _survive_ it needs to _quickly adapt to new things_: new technologies, new trends, new fads, etc.

But eventually, when entities get too successful, they get _complacent_ (with themselves) and turning _away_ from their younger style of functioning (of absolving everything in order to get successful in the first place).

All this, of course, it the _natural_ way of history. We see it when studying History, and we see it every day when reading the Technology section of our favourite publication.

Some companies are _fiercly_ trying to beat this _natural_ course of History, into turning 180 degrees from their mindset every single year in order to chase the latest technology, the latest fad, in order to please customers and stay on top.

I wish Linux would return to its roots.

OMG, "hobbyist" -- this so perfectly describes it! LOL

So now, if there is any coroprate management from RH here reading this they'd take heed in this.

Linux _development_ needs to catch up to Linux _deployment_.

Currently they are two different worlds.

Now _this_ is a smart suggestion: it wouldn't break legacy hardware _and_ it would give Linux SCSI a fresh start.

Next year, your new serverboard wouldn't have any of those old cumbersome storage chips to worry about. It would have only one storage chip which could do SAS and SATA and that'd be that. Why would anyone need this fat, old semanticaly overloaded, SPI-centric SCSI Core?

Yes, I'll get that jpg and try to color it, but there's also a lot of other components (Interconnect, SDS, Application Client) which are not visible in this interconnect.

I'll try to get something descriptive.

Foremost, this experience reminds other vendors that Linux _development_ model is _not_ en par with their Linux _deployment_ model (i.e. for a business).

Many things are left to the whim of developers whose educational and technical background could be in question especially when your only communication with them is via email.

I mean -- I don't even know the _age_ ballpark of most developers here. (Other than ones I've seen at OLS.) For all I know I could be having a discussion with a 9 year old kid, repeating SAM and SPC references over and over again.

I just don't see the technologically savvy and _open_ community here.

I'm not shoving my solution down the throats of LSI or James or Christoph. Why?
- because the technologies are different,
- beacause I'm following a SAM model, they are not.
- and because I'm not changing anybody else's code but integrating with it.

(Jeff, I know that on the 3rd point, you'd say "That's the problem, you should be improving SCSI Core", and I know that if I had been changing other people's code, you'd say "You should not change other people's code", so it's a win-win-manipulative situation for you. I'm aware of that, spare your keystrokes.)

This is all _fine_ with me as long as they are not shoving down my throat their solution.

I wonder the pride to technological merit ratio in Linux SCSI.

Maybe things are slowing down for Linux? Attitude? Complacency? History? Who knows?

We did have a Storage BOF (Birds Of Feather) at the Ottawa Linux Symposium (OLS) in July this year. It was called "SAS" and someone made me do the presentation. Drawing on a white board several layers and how they communicate with each other, how things work, why they work the way they work, how they are shown and represented and why.

The response I got from James and Chritsoph was:
"The sysfs tree would be too deep."

The next day I learned that such argument is bogus and a sysfs tree can be as deep as the representation calls for.

So you see, _this_ is where we started -- with _this_ kind of argument: "The sysfs tree would be too deep."

BTW, James already had already had my code for about two weeks before OLS.

Luben

The core problem is that a SAM-friendly path to SAS has already been chosen -- transport classes -- and your driver isn't following this path.

If we choose a new path every six months, we'll never arrive at the destination.

Linux today is the most successful its ever been. This system, however strange it may seem, does work.

The rest of the Linux-SCSI devs are trying to make it less SPI-centric.
Rather than just complain, we're doing something about it.

Background is irrelevant. Only results matter. Linux is a meritocracy.

Spare me your paranoia. I've been 100% honest with you in every email I've written.

You -should- change other people's code. That's how Linux gets better.

When I chose a better path for libata's error handling, the first step in that process was changing the locking, and modifying almost every $!#$@! SCSI driver in the kernel. Rather than forever complaining about an outdated SCSI layer, I stepped up and fixed things.

SCSI work is speeding up. The SCSI core has come a -long- way in the past couple years. 2.6.x SCSI is light years ahead of 2.4.x SCSI.

Jeff

Man you never stop do you?

I repeat again (I'll cut and paste this):

When JB's transport classes were introduced there was NO MENTIONING of SAS. THE REASON THEY WERE INTRODUCED INTO LINUX BY JB IS TO ACCOMODATE MPT-based SOLUTIONS FROM TWIDDLING WITH IOCTLS!

How do I know this: simple: JB's "transport attributes" have NOTHING to do with SAM.

They break the layering architecture for one, and are ATTRIBUTE EXPORTING FACILITY for another.

I understand that you want to preserve your friend's bal^w^w^wpride but, lets face it:
I do not try to shove my solution down JB's throat.

As I've said many times: they are different due to the *technology they represent*, which differs in implmentation, and they can coexist!

If you can say this statement:
"The core problem is that a SAM-friendly path to SAS
has already been chosen -- transport classes -- and your driver isn't following this path."

This means that you have NO CLUE ABOUT SAS or SAM!

I certainly hope that things would improve once you start reading specs and talking to the engineers who designed BCM8603. (If you are still going to write their driver for them.)

No one is choosing a new path every six months.

And this isn't a new path for _everybody_ to follow, it is only for LLDD who need a transport layer to drive the transport they expose an interface to.

I'm _not_ choosing a new path. No one is required to "follow" this. This is just _technology_ for LLDD who _need_ a management layer, and who _cannot_ inteface with a high level such as SCSI Core.

And as you can see, Linux today is the most anal retentive as it has ever been (e.g. SAS, reiser4, and other (revolutionary) technologies).

Remember, you can only go _down_ from the top. So please do not say
"Linux is the most successful it's ever been."

It's too immature as it means that it would either go down or that it cannot become even _more_ successful.

Oh this is such a political sap, Jeff -- I cannot believe you're actually saying this.

Who are you pleasing? Your management?

I've been asking for movement AWAY from HCIL from early 2003, when you were writing libata, remember? Or do you want a link to marc.10east.com?

Stop this political BS sap "we're doing something about it", please.

I suggested native scsi_target support so long ago, only to get snuffed at.

Single line statements like this with 0-content, FUD-like, only for you to reply right away, do not _contribute_ anything.

You're just throwing sand in the eyes of managament reading this.

I doubt you've ever been honest with me.* The reason is that you are trying to push down my throat JB's "transport classes", all the while you're saying I'm supposed to change other people's code?

Just see how you started this email (above).

So please, don't try to get out of any situation turning 180 degrees around.

* Well, maybe you _are_ 100% honest with me. Maybe you really do believe that your friend JB's "transport _attributes_" code has anything to do with SAM.

I flipped a card: "You'll get a promotion soon!".

What else can I reply to a self-gratifying statment like this?

Can I reply with someting like: "See my SAS stuff -- how it truthfully represents the whole domain and gives complete control of it over to user space? See?"

Well, this only tell you the (bad) state of affairs of _2.4.x_ SCSI. This does _not_ tell you the state of affairs of 2.6.x SCSI.

2.6.x SCSI is light years _behind_ SAM.

Luben

Wrong. This shows you fundamentally don't understand transport classes at all.

AFAIK, the first transport class was FC, for qla2xxx.

Read the code to see how FC avoids the SPI-centric scan -- an example of transport independence.

Transport class == transport layer. Eventually this will sink in.

Transport class allows for complete transport independence, be it SAS, FC, iSCSI, or other.

We've still got all that Microsoft and old-Unix marketshare to steal :)

I'm merely stating I'm submitting patches to clean up SCSI core. Others have submitted far more patches than I. And further patches to SCSI core are needed to properly integrate SAS as a transport completely independent from SPI. I'm going to be putting time and effort into moving the SCSI core away from SPI, so that SAS can be properly integrated.

All I've seen from you is
(a) complaints that the SCSI core is too SPI-centric
(b) a solution that does nothing to fix this

My goal is Linux. Always has been. I put quality of Linux code, and giving features to Linux users, above all else. Have been doing so regardless of who employs me, for many years now.

Maybe one day I will be independently wealthy, be a completely independent Linux maintainer, and then people will have to find something other than Red Hat as the reason for why their code is receiving criticism.

To get a fully SPI-independent SCSI core, we must change other people's code. That's the way Linux works. We evolve the existing code.

Jeff

And it shows that you do not understand SAM.

How do I know this: simple: JB's "transport attributes" have NOTHING to do with SAM.

They break the layering architecture for one, and are ATTRIBUTE EXPORTING FACILITY for another.

I understand that you want to preserve your friend's bal^w^w^wpride but, lets face it:
I do not try to shove my solution down JB's throat.

As I've said many times: they are different due to the *technology they represent*, which differs in implmentation, and they can coexist!

If you can say this statement:
"The core problem is that a SAM-friendly path to SAS
has already been chosen -- transport classes -- and your driver isn't following this path."

This means that you have NO CLUE ABOUT SAS or SAM!

I certainly hope that things would improve once you start reading specs and talking to the engineers who designed BCM8603. (If you are still going to write their driver for them.)

Nah -- all FUD.

See how you're talking general stuff. See how I talk specifics:

The host template was introduced to satisfy SPI only LLDD which was everything available at that time and SAM didn't exist yet.

Over time it was enlarged to accomodate others and all LLDD implemented it and simulated SPI centric view.

Now, you have a storage chip on a pci device which is NOT a host template material.

I.e. the LLDD is a _PCI_ device driver, NOT SCSI!

It exports only a SAM section 5, 6 and 7 view of
the Service Delivery Subsystem and interconnect.

It does _not_ export anything "scsi host" like.

For this reason you _need_ a management layer on top, but _below_ SCSI Core, since that management layer is _transport_ specific _and_ SCSI Core should be completely _unaware_ of the transport!

Then _this_ transport layer, presents things
to the SCSI Core as "scsi host" material.

Among the many, lets take for example this
member of the struct scsi_host_template
along with the comment:

/*
* In many instances, especially where disconnect / reconnect are
* supported, our host also has an ID on the SCSI bus. If this is
* the case, then it must be reserved. Please set this_id to -1 if
* your setup is in single initiator mode, and the host lacks an
* ID.
*/
int this_id;

SPI-centric?

How about this from scsi_host:

	/*
	 * These three parameters can be used to allow for wide scsi,
	 * and for host adapters that support multiple busses
	 * The first two should be set to 1 more than the actual max id
	 * or lun (i.e. 8 for normal systems).
	 */
	unsigned int max_id;
	unsigned int max_lun;
	unsigned int max_channel;

And I can continue to give examples of this for as many lines are in the header files of Linux SCSI.

Now Jeff will say: "Then submit patches to fix this."

I've also submitted patches to improve SCSI Core. Those improvements came directly from my own mini-SCSI Core implementation of iSCSI Target. For example, using the slab cache for scsi commands.

Thanks to Doug L, and Andi K, they made it in, if it had been left to James Bottomley, they'd never be in.

Then I continued to post RFCs and various other suggestions, like 64 bit LUN, elimination of HCIL -- all shot down by your friends in the community.

This was back when you had just started working on libata.

So please spare me the political sap -- I've tears in my eyes already.

Stop this FUD man -- it integrates right now:
http://linux.adaptec.com/sas/

So you are going to give all currently existsing legacy LLDD a heart attack?

Or are you going to create new functionality as I had outlined here:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112794008820004&w=2

You know, "struct scsi_domain_device", proper LU
scannint, etc.

I've been wanting to change this since 2003. I even wrote an email that I wanted to completely rewrite SCSI Core for 2.7 in 2003. See this email.
http://marc.theaimsgroup.com/?l=linux-scsi&m=104508658212335&w=2

Sadly as most discussions in linux-scsi nothing materializes, patches get dropped, etc, etc, et.

But it gets you one step closed to it. It merges _cleanly_, people can use it get comfortable with it and eventually things else where would improve as people get comfortable with it.

Jeff, if you think that if one day if you became independently wealthy you'd be an independent Linux maintainer, or do _any_ kind of work, you need to mature a bit more. I _guarantee_ you that in 5 years you'd think differently.

Independently wealthy people start doing charity work and then they use that to get into politics, in order to obtain that which lacks from just having a ton of money:
power.

Luben

[...]

I don't know how things were when you started using linux but linux has never been about that AFAIK. People usually buys windows licenses for that ;)

Oh well. May be Christoph and James are behaving like 9-years-old kids, but if that's true you're NOT BEING DIFFERENT than them if you look a bit to your mails.

You can be pretty sure that at this stage everyone is looking you as a 9-years-old kid who is raving because his mother don't buy him a candy. Maybe they're looking james and christoph int he same way aswell, but that's their issue.

On Thu, 2005-09-29 at 10:33 -0400, Luben Tuikov wrote:
[...]

That is probably the root and single cause of quality problems in
companies: Deployment folks (read: sales) set the delivery date (and get the bonus for selling) and the rest (read: tech folks) have to follow, no matter what (and get the blame for being late and buggy). Or why do you think that MSFT has such quite crappy software?

ACK - in the commercial world (and the bigger the company the worse because sales people are far more distant from the tech people).

[ software rewrite ]

Then submit your driver as a (separate) block device in parallel to the existing SCSI subsystem. People will use it for/with other parts if it makes sense (and you - as the maintainer - accept their patches). And in a few years the "old" SCSI core fades out as legacy drives fade out (or they will happily coexist forever).
The point is: If *you* want it that way, *you* must go that way (and do not expect others to do it just that *you* get *your* driver merged). You are the maintainer of the new stuff and (almost) everything will work as you want.
It might not be the cleanest or most elegant solution in the world, but if it works, who cares and why?
Where is now the real problem?
I can't see one.

Bernd

Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

SAS is ultimately SCSI. I'll just have to write my own SCSI core. _We_ together can do this in parallel to the old SCSI Core.

This is the whole idea.

You see, at my age and my situation, I no longer see this as "my balls - your balls". What matters to me is good design, quality code, customer satisfaction, bottom line.

E.g. I'm quite a liberal person and I wouldn't block or stop new technologes from going into Linux on the basis and merit of my not understanidn that particular new technology.

The bottom line is not "my balls - your balls" but the wide spread use of Linux and "storage OS of choice". Not "hobbyist OS of choice" and not "let me play Robin Hood".

Yep, I've been saying this since 2002. On the linux-scsi ML.

And this is the problem: *you* and "the community" see things in *this* way: "your balls - my balls", "yours/mine".

While I see things like this: new technology, absolve, use, move on.

As to your comment above, it's not about how *I* see things. It's about how things _actually_ *are*:
http://www.t10.org/ftp/t10/drafts/sam4/sam4r03.pdf

Turn the table around: can _I_ pose this question to JB and Christoph?

(since they are the ones who think this of SAM/SPC)

Me neither.

Luben

You should have stated this plainly, from the start.

If you want to do your own SCSI layer, you need to do it at the block layer rather than poking around drivers/scsi/

Jeff

So now you are saying that I should _not_ poke at drivers/scsi? (as I haven't done)

Are you going to make up your mind?

Luben

Are you: are you going to rewrite the SCSI core, or work to improve the existing one?

Your choice, not mine. Your time, not mine.

Jeff

Luben, I think you are missing the distinction being made here and that distinction is very important.

It is *critical* to hardware vendors, to the linux community, and even at this point to some of your competitors in the HBA space that we find the best solutions that work well for *everyone*. The more often we wind up with independent, unique stacks and unrelated methods and mechanisms in the kernel, the more work we all have to do to support Linux. If every HBA out there that thought there were following a new and interesting technology and were going to code to the perfect ISO or T10 or IETF layering model, the linux kernel would be one huge, inconsistent, bloated stinking mess for the rest of us to support.

Worse, all of our customers would see different behaviors in semantics, functionality, and support matrices for every HBA, hardware component, or combination of platform/HBA/storage subsystem.

I believe that for you personally to find success in the Linux development community (much as you probably do in the standards or HBA community) is that you have to talk off your personal hat, you have to take off your employer's hat, and you must put on the Linux community hat for a while and see through their eyes.

In this case, Jeff and others are providing two options with a common goal. The common goal is to increase the overall amount of commonality and common code in Linux in this case.

You can do that two ways: Start with a new SCSI core library and show with code how it works for multiple HBA vendors (this includes working with your competitors!) OR help to evolve the current code to better address your needs while not breaking the needs of other consumers.

Either approach is valid. You can start sending patches to update the SCSI core code to simplify your driver and fit with the current community model, OR you can put forward not just a model but the libraries and proof-of-concepts (possibly while working with other HBA vendors) in the form of Linux kernel code to demonstrate the viability of a new stack which satisfies the wider community goals of ease of maintainence and ease of understanding over time.

In general, if you think you see a contraction coming from the community, I'd encourage you to look more closely - there are some strong guiding principles for the community.

I suggest reading through a few of these resources which might help:

http://lists.osdl.org/pipermail/os_drivers/attachments/20050426/261c7d9d/DeviceDriverDevelopmentPlan-v.02-0001.pdf

http://www.madrone.org/mentor/linux-mentoring.pdf

I'm sure others have simmilar materials floating around. You are not the first person to suffer from culture shock here but the sooner you understand the goals of the community and show how to help meet those goals (hopefully with code to substantiate your goals, and the ability to incorporate feedback and give feedback) the sooner we'll have a working driver in mainline.

gerrit

[...]

The question is: Who starts and leads seriously this effort (and takes in the course others with him)?
Just telling others where to go and waiting until they carry you there usually doesn't succeed if you are not the boss of them (and/or they are free to leave at any time without significant punishment).

[...]

Perhaps speaking is not enough and work should follow?
All people who really do the work like those others standing beneath (possibly doing their own thing) and telling them how to do their own work best.

It's at most (and actually exactly) "my work - your work", not "my balls
- your balls" (or "code" or whatever).
If you want to play "our work" (read: team[0] work), than you have to cope with others (and they with you), their - probably historically grown - design (even if it is wrong), etc.
If this doesn't work (and ATM I have this impression) for whatever reason, the second step is "my work, your work".
And if two (or more) people have different design opinions (and different opinions about "quality" and/or "correct vs wrong" - I can't decide since I have virtually no knowledge about SCSI core internals, discussions on the Linux-SCSI-ML, etc. to decide - not even for me -, who is "right" and/or "wrong" in which aspect or in general), it comes down to "better the not-so-good design and working code than the best design and no code".
So just copy the old core, throw out what you don't want, need and/or like and voila. If it *is* "better", it will succeed and people will come and help.
Of course it is probably more work in the short and in the long run to maintain a separate block driver for SAS storage, but it is at least a possible solution.
Perhaps all relevant people will see that the difference is small enough to converge to some point in between.

Bernd

[0]: Not in the ironic interpretation in German which translates roughly
to "great, another one does it".

Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Ok.

That sounds very fine, _but_ "the community" isn't a "corporation".

"The community" is by far and wide very close friends, so if you point out to one of them that his code is wrong, even if all other see that you're indeed correct, no one would say anything.

Why? Because he doesn't want to same thing done to him.

So it doesn't matter who is right or who is wrong. What matters is who has political power to have it his way.

Blesses art thou, who believeth.

I wish it worked like this, I really wished. Sadly its doesn' work like this.

James is a strong political figure. He keeps people who he thinks know more than him at bay. I can quote several names here, who are active and some who were active at one point or another, all very well versed in the T10 ways, but it wouldn't be fare to them.

So what you have is a strong political game: they don't care what is right or wrong, they'll implement it the way _they_ think is right, in effect alongside _their_ code.

I'm not sure how much History the readers have studied, but such politics have always yielded to extinction.

The reason is _not_ because you reject something, but because you end up always doing it always _your_ way. Eventually you become unfit to compete when the world has changed _so much_ around you, that "your way" is no longer relevant and the change to make you fit is so radical, that it is impossible to do.

So you see, it is _not_ about accepting code, it is about accepting _ideas_ and _innovation_.

James can still do everything _his_ way. The question is how many _years_ would this be relevant?

Yes, but in "the community" people want "great, he does it _my_ way".

Luben

Is your cat sleeping on your underscore key?

Joel

"There is no more evil thing on earth than race prejudice, none at 
 all.  I write deliberately -- it is the worst single thing in life 
 now.  It justifies and holds together more baseness, cruelty and
 abomination than any other sort of error in the world." 
        - H. G. Wells

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127

Luben,

I'd really prefer if you'd just stop on your tirade and just send in a 10 line patch for the existing linux SCSI subsystem to fix something you think is wrong.

Code talks, bullshit walks.

Sure, Linux SCSI might not be ideal, but how many people do you know have SAS storage on their home PCs right now? Heck, I don't have SATA or PIV on my home system!

And as a customer of Adaptec hardware, I'm getting to the point of seriously taking my money and going elsewhere for my storage needs. If you are a general example of how Adaptec works with its customers, then I want nothing to do with you or your products.

Sure, I know you think Linux is stuck in the past, so help us move to the future in small baby steps. It doesn't require big huge leaps like you're proposing. I mean, have you actually tried to your old legacy SCSI controllers in a system with your new hardware? How did your testing go?

Now I'm not a programmer, and I can't talk like one, but in my real life job I'm a SysAdmin and knowing what I know about Adaptec's interactions on this list will certainly color my perceptions of your hardware and trying to make it work with my company.

John

Well said John!

For that matter get us a driver for the embedded 7902 (is that the one?
I forget now) with host raid support.

Luben,

I have a vested interest in the improvement of the Linux SCSI Core and wider adoption and support for SATA II and SAS controllers with their associated domains and transport.

Proving a better design with a migration path for integration is the key for success; however, I am not the person to be the political voice in the process. People will disagree in the process and the only counter to remove blockage/adoption is in code.

James is king of the hill, and is reasonable to a point. James also follows a model of generalization v/s specific design. Argh, this is not going to be an easy one to explain or back away from now. Erm, inclusive API design is where I am wanting to go with this thought.

Have you and company considered the approach of mapping to a library of sorts?

Cheers,

Andre

Us and other companies too.

Yes, _it is_ the key to success.

Which "hill" is the question. If he were reasonable, he'd understand that the two technologies _are_ completely different and distinct and he'd not try to shove HIS solution down my throat. Just as I do not shove my solution in his throat.

The whole point is that MPT-based (IOP on package) solutions _do not_ need a Transport Layer, since that Transport Layer _is_ already present: in the firmware _and_ access to it is _firmware_ dependent.

All the while open transport solutions (ours and apparently Broadcoms') _expose_ the whole infrastructure and _give_ you the chip as an interface to the interconnect. So you actually _need_ a transport management layer. Now that transport management layer sits _below_ SCSI Core and SCSI Core is _unaware_ of the transport.

You can have inclusive API design for chips like AIC-94xx and BCM8603, because they <see paragraph above>.

MPT-based ones <see paragraph above> would also use inclusive design _for MPT-based chips_.

Hmm, it is not a library.

It is a layer, again, because of what the chip actually is, and because of what it implements.

Take a look at the announcement text, I do give some description there and in the code the drivers/scsi/sas-class/README file describes the event/managment infrastructure. Also you can take a look at the code.
http://linux.adaptec.com/sas/

What you'll see in the code is:

hardware implementation (interconnect, SAM 4.15, 1.3)
firmware implementation (interconnect, SDS, SAM 4.6, 1.3)
LLDD (SAM, section 5, 6, 7)
Transport Layer (SAM 4.15, SAS)
SCSI Core (SAM section 4,5,8)
Commmand Sets (SAM section 1)

A very nice explanation in latest SAM4r03,
section 4.15 The SCSI model for distributed communications.

Now for MPT based solutions you have:

LLDD (SAM, section 5, 6, 7)
SCSI Core (SAM section 4,5,8)
Commmand Sets (SAM section 1)

You see? No Transport Layer between LLDD and SCSI Core! Why? Because all this work is done in FIRMWARE!

That is, the LLDD exports the LUs right into SCSI Core, So in effect there is _no_ need for a software Transport Layer.

Can we have a video/tele conference on this?

Luben

Transport class + libsas achieves this.

Maybe I will have to demonstrate using code...

Jeff

This is *WRONG*. (see below)

And it doesn't "achieve" this. Stop the FUD.
There is a _reason_ why it is the way it is.

Jeff,

There is a _reason_ why technical people separate concepts in _layers_.

There is a _reason_ why technical people use Object Oriented Paradigms describing models and design.

Do you know _what_ that reason is?

Or should I leave you to "demonstrate with code"?

Seeing that you keep _persisting_ in your ways,
I'll leave it for you to "enrich" Linux SCSI in
your "demonstrate with code".

Luben

BTW, Linux' implementations of transports like USB storage and SBP-2 have always been similarly layered. (Actually they come with at least one more layer between LLDD and SCSI core.) Needless to say that these transports need their specific managing infrastructures. So this layering is not at all new to Linux.

Stefan Richter
-=====-=-=-= =--= ===-=
http://arcgraph.de/sr/

True, true.

But those subsystems are shielded from SCSI Core. Plus SCSI Core is managed by people unaware of SAM or the layering infrastructure.

Luben

Hmm.. 20 years ago I was hooking Fujitsu Super-Eagles to Sun3/280 servers. If you're going to claim that the current SCSI core is *that* far behind, you're going to have to back it up. Remember that making exaggerated claims is a good way to make people not listen to the *rest* of your message.

Seen in include/scsi/scsi.h:

/*
* FIXME: bit0 is listed as reserved in SCSI-2, but is
* significant in SCSI-3. For now, we follow the SCSI-2
* behaviour and ignore reserved bits.
*/

So obviously, it's at least the number of years since SCSI-3 was defined,
but no more than the time since SCSI-2. According to http://home.comcast.net/~SCSIguy/SCSI_FAQ/scsifaq.html
SCSI-2 devices started showing up in 1988, and X3.131-1994 came out in 1994.
1996 saw the first SCSI-3 proposals.

I'll give you *one* decade, but not two. :)

Ok, I'll take that.

BTW, I was referring to the _architecture_ of SCSI Core.

It hasn't seen any _innovation_ for the last 5 years as far as SCSI or Storage is concerned.

Luben

Incorrect, just look in the latest upstream kernel.

That's the problem.

libata-scsi.c is the SAT translation layer.

That's something that needs fixing, for SAS.

Because it has the problems listed time and again.

The SAS transport class is designed to support both firmware-based devices like MPT, and non-firmware devices such as Adaptec.

Sure it might need patches -- send patches, work with people, rather than ignoring existing work.

We've been over the technical stuff time and again. That's the maintainer problem. We need someone who will listen to the community.

It means not taking feedback, and working around rather than with the SCSI core.

Then you don't understand the ->qc_{prep,issue} hooks. That should get you 90% of the way there, if not 99%.

Re-read libata-scsi.c, and submit any patches you feel are needed.

That's one of the problems: It should update the SCSI core.

Jeff

Yes, because it was Christoph's code submitted right after I submitted and JB took his code right in.

Again, I had the infrastructure ready _before_ OLS.

Not quite.

It has potential to become SATL but at its current form it cannot be:

int ata_scsi_queuecmd(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))

{
	struct ata_port *ap;
	struct ata_device *dev;
	struct scsi_device *scsidev = cmd->device;

	ap = (struct ata_port *) &scsidev->host->hostdata[0];

If I want to use that I have to _simulate_ ata_port. Furthermore, scsidev->host->hostdata may not point to ata_port for SATA devices over a different transport.

Ideally SATL is just a _data transformation function_ and getting to the ATA device is transport dependent.

Jeff, would you be accepting patches?

Would you be accepting patches for the creation,
and use of "struct scsi_domain_device { ... }"?

This would be a "SCSI Device with a Z Port". Z could be target or initiator (most likely just T).

Then for each scsi_domain_device, SCSI Core does REPORT LUNS and then INQUIRY for each LU it found.

The old (current) code would still say as is, unchanged, since it supports older, broken hardware.

Would you be accepting patches for this?

What problems when there was no other code around.

Look at it this way: _their_ code doesn't integrate with ours. See?

I simply cannot take an argument like this:
"Because it has the problems listed time and again."

You have to be specific.

No, it never has been. It is an _attribute_ exporting framework only.

If you understood how different those architectures are, you'd understand what I mean.

I do not _know_ how to consolidate the completely broken "design" set forth by JB and company.

It is _completely_ not to SAM spec.

Exporting attributes from MPT-based drivers is not the same as managing a transport.

I repeat again:
* MPT _hides_ the transport and the managment
of the transport from you -- so in effect there is nothing to manage.
* MPT gives you Logical Units to work with, NOT ever domain devices or anyhing domain like.
* MPT gives you a SAM/SPC hook to hook _right_ into SCSI Core.

_This_ is why their LLDD _can_ use the host template.

But an AIC-94xx and BCM8603 _is_ NOT a scsi_host material. It is just an interface to the interconnect.

To convince yourself of this: take a look at the _members_ of the scsi host/scsi host template: nothing in that structure is presented in the chip -- UNLIKE old Parallel SCSI device drivers.

The scsi host template was written to cater to _old_ (then new) Parallel SCSI drivers! Times have changed! Time to evolve.

No we haven't been over the technical stuff time and again.

I bet you're melting people's hearts when they read this.

So to summarize for corporate management what you're saying here is:
- you're saying that I do not listen to "the community",
- you're saying that I'm an _outsider_ to "the community",
- you're saying that I'm no good to work with you, since you are part of that community but not me.

That is you cannot take it that someone will tell "the community" anything. "The community" knows all and it knows best.

So in effect there are no good and knowlegable engineers anywhere but in the "Linux community".

That is there is no people with new ideas, better ideas, innovative ideas, more knowlege about certain subject matter, _anywhere_ but in the "Linux community".

So in effect, there will never be an "outsider" who will come in to the "linux community" and change things around, no fresh ideas, no better (right?) way to do things.

"The community" is not going to listen to anyone but only to already politically established members on _any_ subject matter, even technical, even from "outsiders" who work with the technology every day.

So does this mean, that if I submit patches "fixing" (oops, not meant to hurt anyone's bal^w^w^wpride) SCSI Core, they will be accepted.

Or do you want me to do "your way of SAS"? Maybe JB et al, should write the "Linux SAS spec" and then we can recommend this to T10.org, so they can scrap their version and use "the community's"?

I hear there was such a suggestion on the mailing lists for ATA and T13.org, but I'm sure I'm misunderstanding something here as usual.

But I have to simulate struct ata_port, Jeff.

Which isn't so bad, but speaks about the design.

Will do.

Thank you for admitting this -- you're the first and only member of "the community" (since I'm not a member apparently) to admit this.

Luben

Incorrect. It needs to issue multiple ATA commands to emulate a single SCSI command, cache some state, and other details. Not purely data transformation.

Yes. That's what I mean when I say "submit patches".

What needs to happen is that SPI-specific stuff in the SCSI core needs to be moved to SPI-specific transport code.

Then all transports will be at an equal level, and the SCSI core will be fully transport-agnostic.

"time and again" means that we have been specific multiple times. Re-read your emails from James and Christoph :)

James is wrong here. The "transport class" in reality winds up as a transport lib, in addition to simply exporting attributes.

There will always be functions that are common to a transport. Christoph calls this "libsas", since software-driven SAS implementations will share this transport code, but not necessarily all SAS implementations (MPT, qla etc.).

Not true. Just because SCSI core lacks an explicit "execute SCSI command" RPC hook, does not imply that that capability is non-existent.

Stare at the Scsi_Host_Template some more and you'll see it.

A scsi_host is simply a container. You're being too literal.

Yes. We need to evolve the SCSI core to separate out SPI-specific pieces, to make it more transport-agnostic.

correct

irrelevant

irrelevant

overreaction

James and Christoph have been asking you to submit patches for a long time now.

You're over-reacting.

You have to provide a means to submit ATA commands and receive results, no more or less.

James, Christoph and the rest of linux-scsi have been saying this over and over again.

You need to update the SCSI core properly, rather than working around it.

Everybody knows the SCSI core is too SPI-centric, and you have been given a recipe for fixing this.

Jeff

Yes, this is right and I'm aware of it. I really, really, had some wishful thinking.

Yeah, I've been saying this for ages:
Read half way through my message from 2003 here:
http://marc.theaimsgroup.com/?l=linux-scsi&m=104257405408078&w=2

So you're saying that they are right and I'm wrong, and I should listen to them.

Which is _exactly_ what I've been pointing out:
"the community" thinks that only they are the experts on the planet regarding new technologies, thus they are right, engineers "outside the community" are dead wrong and they should listen to what "the community" says.

Shall I point out a multitude of emails whereby some "community members" go out on _technology_ public lists and start a flame war, until they are warned to behave or be expelled? Is this what you mean the "they" are right, and "we" are wrong?

The community calls specs "techno-gibberish" and think that LUNs can be u64, that REQUEST SENSE clears ACA, and that HCIL is here for ever, etc.

Excuse me if I simply ignore their "SCSI storage advice".

Either you meant "Luben" here or you've been banned forever from "the community" and your patches would never ever be accepted.

Look at the pictures (since its easier) in SAM/SAS/SPC. It is not a "lib" it is a _layer_ in its own right, which is completely and fully implemented in the FW of an MPT-like (IOP in package) solution.

Access to _anything_ transport specific is _FIRMWARE_ specific and doesn't yield to unification as does a SAS Transport Layer management.

The only unification you get is: "Here is a LLDD giving you LUs to manage, and oh yeah, here is some FIRMWARE dependent way to peek at the transport and transport attributes".

Well, then, how can I reply to this?

I say "1+1=2", you say "Not true. 1+1=3."

Show me the equivalence between the Scsi_Host_Template and SAM, give me section references as well.

What I meant is that the place and design of JB's transport attribute "blessing" is completely out of whack for SAM abiding implementation.

Look at the pictures: the transport layer is _below_ the application client and the application client is completely unaware of the transport.

Now lets translate (http://www.t10.org/scsi-3.gif) :

    Command set drivers  <-->  sd, st, etc (application client)
                SAM/SPC  <-->  SCSI Core to be
        Transport layer  <-->  SAS (all others implemented in LLDD)
                    SDS  <-->  LLDD + Firmware
           Interconnect  <-->  Firmware + hardware.

It is _this_ SAM architecture which allows you to say, send SATA commands over SAS links (via STP), and do other interesting things.

I guarantee you that any _new_ SCSI transport would yield to a Transport Layer abstraction just as SAS does.

Since, this is what SAM _is_ (all about).

I don't mind James Bottomley entertaining his "transport attribute" code, as long as he's not shoving it down my throat (again because of pictures like the one above).

As I've said before both implementations are _orthogonal_ and can coexist.

By "too literal" do you mean "following specs too closely", or do you mean "being realistic without tweaking things".

2003:
http://marc.theaimsgroup.com/?l=linux-scsi&m=104257405408078&w=2

Not patches to fix SCSI Core or to get "struct scsi_domain_device" into SCSI Core or to get 64 bit LUNs into the kenrel.

They've been asking me for me to abandon the complete SAS transport layer which I have, which is 1000 years ahead of theirs and ahead of SCSI Core and to go and work on LSI's and on "transport attributes".

Sorry, but SAM seems to disagree.

They've never said it: why else do you think we do not have 64 bit LUNS or why else do you think we have HCIL in SCSI Core.

Instead, what James does is he goes off to implement "transport attributes" and to submit patches to IDR after I myself submitted patches to IDR? Why? So he can undermine my patches? Well, as you can see I completely removed IDR's usage from aic94xx. Now "the community" is happy.

I "have been given recipe for fixing this"?

Are you all right Jeff?

Yep, the recipe was given to me by people who think that we should stay with HCIL, who have found purpose for the "second channel" in HCIL, who think that REQUEST SENSE clears ACA, who think that 64 bit LUN is just a waste, and so forth with their antics.

So excuse me if I don't see or recognize the recipe given to me. Mind pointing out a link? This way we will have a hard coded evidence of what that recipe is. And we can see the exact steps it outlines.

Thank you,
Luben

How many times do we have to repeat "HCIL dependencies should get moved to SPI-specific transport code" before you listen?

Quit being silly. I meant what I wrote. Though it turns out that your interpretation of James's ideas was incorrect. James agrees that "transport code" includes both lib and attributes, not attributes only.

By focusing on attributes, you miss the big picture. This is not about attributes.

This is fundamental: SCSI specs dictate how this functions, not necessarily what Linux code looks like.

The big picture is

	device class <-> transport class

and everything falls out from there, including SAM.

Moving SPI (parallel SCSI, for those unfamiliar with the acronym) to transport-specific code will eliminate legacy assumptions that you keep complaining about.

Linux is about getting things done, not being religious about specifications. You are way too focused on the SCSI specs, and missing the path we need to take to achieve additional flexibility.

With Linux, it's all about evolution and the path we take.

As long as you think James is merely talking about attributes, you're just not getting it.

Transport classes are an abstraction which allows SAS to exist as a peer to parallel SCSI, FC, and other acronyms.

I mean, paying too much attention to specs at the expense of understanding how Linux code needs to be shaped.

They are asking you to help with the task of eliminating SPI dependencies in the core, so that SAS can exist as a peer to other transports.

Repeating myself (and others): everybody wants HCIL stuff to move to SPI-specific transport code.

That is the task that must be completed BEFORE transport layer for SAS can be fully merged.

For the nth time, everybody agrees HCIL should move to transport-specific code.

And from past threads, everybody seems OK with moving to a 64-bit lun representation.

For the nth time: http://marc.theaimsgroup.com/?l=linux-scsi&m=112487476527470&w=2

Jeff

Hmmm... I'm fine with "not being religious about specs", but I hope we try to respect them as much as possible, because it's the only way to get everything working and not get the finger pointed to when there's a problem. When I put netfilter + window tracking in production 2 years ago, there were a lot of drops (about 2% of sessions = ~40/s), because shortcuts had been taken WRT rfc793 ("the specs"). Then, printing the whole RFC+erratas was the only way to get the code working and supporting the most bizarre cases that had previously been considered stupid or impossible by the developpers (including me during the first phase of the fixes).

I prefer that we stick to specs even if we just implement what we *need* from them, leaving lots of blank boxes on the diagram, rather than interprete them as we think would be better and get annoyed everytime a vendor tries to adapt it to support his hardware.

Both are true : Linux code needs to be shaped to match specs. We must not spend time on what we don't need but we must respect the model and layering, and that's true in any subsystem. Networking for example, is very clean in this area. Even accelerations do not break layering, it's just that some of the work can be pushed down from layer to layer until one can process it (eg: checksums). And when I worked on bonding, I did not have problems with it breaking upper layers. I really believe that's important, it's the first step to avoid spending time in debugging.

Cheers,
Willy

a spec describes how the hw works... how we do the sw piece is up to us ;)

(I know the scsi stuff also provides sort of a reference "here is how you can do it in sw" but I see that more as you "you need this functionality" not "you need this exact architecture in your software")

Arjan, I'll be your best friend here:
Never say this in public or in an intervew.

Hardware folks needs to work with software folks and software folks need to work with hardware folks.

This is what makes good design.

Luben

It's hard-earned experience. We constantly have to teach hardware vendors how to write good drivers.

At some point you have to step away from the spec, and ask yourself what makes sense for Linux. I've already had to poke T10 when they put silly things in the SAT spec.

As a tangent, I already have a design for a Linux filesystem that makes use of SCSI object-based storage (to James's horror, no doubt :)). It's a fun thing to ponder.

Certainly. The historical disconnect is where hardware vendors tend to presume They Know Best, when in reality it needs to be an equal tradeoff. Hardware vendors must admit they don't know Linux, and Linux developers must admit that hardware vendors know their own hardware better than anyone else.

Jeff

I'm sure you have. Hardware vendors are lost without Jeff, James Bottomley and Christoph.

You see, it is because of your _enormous_ ego as shown above, that the code is being blocked.

I'm sure -- flush interoperability down the drain.

Surely they are lost without you.

Ok, so the way I see it you want to show who has got the bigger balls?

Jeff, I have *worked* on a Linux OBD-based filesystem.

Are you going to stop this self-gratifying stuff?

Reflection of above:
The historical disconnect is where "the community" tend to
presume They Know Best, when in reality it needs to be an equal tradeoff. "The community" must admit they don't know hardware, and hardware developers must admit that "the community" know their own code better than anyone else.

Jeff, if you had started looking at the design and firmware of any new SCSI storage chip, you'd see how incredibly similar it is to the transport it defines, and thus to SAM, since the transport itself has to comply with SAM for interoperability (TMF and all).

Linux SCSI does _not_ need to do "its own thing". There are perfectly well defined specs, telling you how things are conceptually _and_ in the physical world.

In order to control those objects, you need to represent them internally (you can learn this either in neuroscience class or in OOD & OOP comp sci classes) as you can see has been done in the SAS Transport Layer code.

So if you want _better control_, higher quality you need to invent _your own_ stuff as _little as possible_ and represent things as they are.

Luben

No, I was referring to things such as, e.g.
http://people.redhat.com/arjanv/olspaper.pdf
http://people.redhat.com/arjanv/OLS.pdf

It has nothing to do with ego, just hard-won experience.

There are bunches of hardware vendors who have their patches merged immediately after posting. They get it. They have internalized the reasons why Linux drivers look the way they do.

Oh good grief. It was an example, silly. Trying to lighten the mood, even.

Jeff

This is so true, and so often overlooked when hardware is being designed and built. A lot of it can be attributed to mismanagement and bad communication between departments ... But, that's the drawback of the corporate system

AFAIU, demands to get our concepts closer to SAM concepts are actually motivated by this: To achieve additional flexibility. (In particular, to ease integration of the various transports.)

We implement drivers of initiators. (Well, targets too.) The specs describe _what_ initiators do. Thereby they partly describe _how_ drivers of initiators (our sw pieces) work.

This is correct. The SAM is what it is --- the SCSI-3 Architecture Model, not the architecture model of Linux' SCSI stack.

However it is very desirable to reflect layers and concepts of the SAM in our stack. One class of reasons is maintainability. No person is able to understand the stack; nor is a person able to consume and understand all or even half of the SCSI specs. No person is able to construct a mapping between the whole stack and the whole SCSI-3 Architecture Model (in his mind or with pencil and paper...). Therefore there have to be _components_ of the stack's architecture model which map 1:1 to _components_ of the SCSI-3 Architecture Model.

Fortunately, SAM layers and concepts are already there in the stack, although incomplete and incoherent. It will always be disputable _how_ complete and coherent our software should be in this respect. However it is out of a question that our software's architecture model might arbitrarily differ from the SAM.

Stefan Richter
-=====-=-=-= =--= ===-=
http://arcgraph.de/sr/

That's what transport classes help achieve.

We just gotta move gunk from the core (HCIL etc.) to scsi_transport_spi before we get there.

Mostly agreed. Some of the firmware-based and emulated SCSI stuff is a bit of an I_T mix.

I think just about everybody agrees with this.

The current thread is more about the path to take, to get there...

The general point about specs is that you gotta think about them, not just blindly implement them. SNIA for example is notorious for generating silly storage-related specifications. And some of the T10 stuff is... well... obviously designed by committee :)

Jeff

No Arjan, you cannot say that ! (well, of course you can but in this case you may be wrong). A spec describes any process, whether it's soft or hard, and BTW, the frontier between soft and hard is diminishing. When I designed a PI-Bus-PCI bridge 10 years ago, I used PCI 2.1 Specification. And it was more related to software than hardware (FSMs, config registers, etc...).

It's the *implementation* which is up to us, not the spec. A spec will never tell you that you have to be compliant with 4k stacks or things like this. This is implementation. What it tells you is when interrupt X strikes and you read bit Y from reg Z, then you must reset bit Y before leaving. And this is software specs, not hardware.

Keeping close to an accepted standard model makes it far easier to upgrade later, but you're right, the spec does not tell you what your implementation must look like.

I think we agree but just don't give the exact same meaning to words.

Regards,
Willy

How we do the SW is indeed up to us, but I want to step in on your first point.

Again.

A "spec" is close to useless. I have _never_ seen a spec that was both big enough to be useful _and_ accurate.

And I have seen _lots_ of total crap work that was based on specs. It's _the_ single worst way to write software, because it by definition means that the software was written to match theory, not reality.

So there's two MAJOR reasons to avoid specs:

- they're dangerously wrong. Reality is different, and anybody who thinks specs matter over reality should get out of kernel programming NOW. When reality and specs clash, the spec has zero meaning. Zilch. Nada. None.

It's like real science: if you have a theory that doesn't match experiments, it doesn't matter _how_ much you like that theory. It's wrong. You can use it as an approximation, but you MUST keep in mind that it's an approximation.

- specs have an inevitably tendency to try to introduce abstractions levels and wording and documentation policies that make sense for a written spec. Trying to implement actual code off the spec leads to the code looking and working like CRAP.

The classic example of this is the OSI network model protocols. Classic spec-design, which had absolutely _zero_ relevance for the real world. We still talk about the seven layers model, because it's a convenient model for _discussion_, but that has absolutely zero to do with any real-life software engineering. In other words, it's a way to _talk_ about things, not to implement them.

And that's important. Specs are a basis for _talking_about_ things. But they are _not_ a basis for implementing software.

So please don't bother talking about specs. Real standards grow up _despite_ specs, not thanks to them.

Linus

<fun_but_off_topic>

[Removing most of the CC. Because they've got real work to do]

[...]

The only usefulness of a spec is to allow to differ between morons and assholes.

http://diveintomark.org/archives/2004/08/16/specs

Couldn't resist.

</fun_but_off_topic>

J

A spec defines how a protocol works and behaves. All SCSI specs are currently very layered and defined by FSMs.

This is _the reason_ I can plug in an Adaptec SAS host adapter to Vitesse Expander which has a Seagate SAS disk attached to phy X... And guess what? They interoperate and communicate with each other.

Why? Because at each layer (physical/link/phy/etc) each one of them follow the FSMs defined in the, guess where, SAS spec.

If you take a SAS/SATA/FC/etc course, they _show you_ a link trace and then _show_ you how all of it is defined by the FSM specs, and make you follow the FSMs.

Ok, then I accept that you and James Bottomley and Christoph are _right_, and I'm wrong.

I see we differ in ideology.

But this is _the_ definition of a theory. No one is arguing that a theory is not an approximation to observed behaviour.

What you have here is interoperability. Only possible because different vendors follow the same spec(s).

Ok, I give up: I'm wrong and you and James B are right.

Yes, it is a _classic_ example and OSI is _very_ old.

_But_ the tendency of representing things in a _layered_, object oriented design has persisted.

Ok.

Ok. Let's forget about maintenance and adding _new_ functionality.

Yes, you're right. Linus is always right.

Now to things more pertinent, which I'm sure people are interested in:

Jeff has been appointed to the role of integrating the SAS code with the Linux SCSI _model_, with James Bottomley's "transport attributes". So you can expect more patches from him.

Regards,
Luben

P.S. I have to get this 8139too.c network card here working.

Luben,

The role of standard bodies is to primarily enforce interoperability but while they suggest FSMs and layering, those directives are not mandatory.

I have also seen industrial SCSI Core implementations from various sources to come to the following conclusions (i) they do not implement all the manadatory stuff (ii) they implement just enough to get by with interoperability [who has the time] (iii) any layering design is evolutionary and (iv) none of them come close to the T10 FSMs.

You may disagree, but there needs to be a balance between standards and implementations.

Luben Tuikov
<[email protected]
m> To
Sent by: Linus Torvalds <[email protected]>,
linux-scsi-owner@ Arjan van de Ven
vger.kernel.org <[email protected]>
cc
Willy Tarreau <[email protected]>,
09/29/2005 04:20 SCSI Mailing List
PM <[email protected]>,
Andrew Morton <[email protected]>, Linux Kernel Mailing List
Please respond to <[email protected]>,
[email protected] Luben Tuikov
<[email protected]>, Jeff Garzik <[email protected]>
Subject
Re: I request inclusion of SAS Transport Layer and AIC-94xx into the kernel

big

A spec defines how a protocol works and behaves. All SCSI specs are currently very layered and defined by FSMs.

This is _the reason_ I can plug in an Adaptec SAS host adapter to Vitesse Expander which has a Seagate SAS disk attached to phy X... And guess what? They interoperate and communicate with each other.

Why? Because at each layer (physical/link/phy/etc) each one of them follow the FSMs defined in the, guess where, SAS spec.

If you take a SAS/SATA/FC/etc course, they _show you_ a link trace and then _show_ you how all of it is defined by the FSM specs, and make you follow the FSMs.

Ok, then I accept that you and James Bottomley and Christoph are _right_, and I'm wrong.

I see we differ in ideology.

But this is _the_ definition of a theory. No one is arguing that a theory is not an approximation to observed behaviour.

What you have here is interoperability. Only possible because different vendors follow the same spec(s).

the

Ok, I give up: I'm wrong and you and James B are right.

Classic

Yes, it is a _classic_ example and OSI is _very_ old.

_But_ the tendency of representing things in a _layered_, object oriented design has persisted.

Ok.

But

Ok. Let's forget about maintenance and adding _new_ functionality.

Yes, you're right. Linus is always right.

Now to things more pertinent, which I'm sure people are interested in:

Jeff has been appointed to the role of integrating the SAS code with the Linux SCSI _model_, with James Bottomley's "transport attributes". So you can expect more patches from him.

Regards,
Luben

P.S. I have to get this 8139too.c network card here working.

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Prasenjit,

The role of a standards body is to self promote and backslap each other amd figure out how to divide the market and make money and not cause problems.

The problem here is T10/T11 is the protocol is "ends justify the means".

This is where the clash of technical and design happens.

If the standards body was going to "primarily enforce interoperability", then why the heck did the T10 proposal for "domain validation" become an empty file and was withdrawn? IIRC it was Adaptec who helped to kill Domain Validation on T10, but I could be wrong and correction is warrented here if needed.

"(iv) none of them come close to the T10 FSMs" is the answer, and this justifies Linux's position to operate under "ends justify the means".

I can tell you why the T13 proposal died. I killed when I withdrew it as a direct result of T10's actions or lack of such.

As much as I believe in NCITS standards, they are worthless if no one is there to assist in the process of creation, model, and adoption.

Regardless of anything stated above, the problem with creation of a scsi standard core is most of the VOODOO of SCSI is hidden in the firmware. James knows this fact and points it out on occassion. SCSI is a FUZZY State Machine and nowhere close to being consider finite.

Not sure if there is a good solution to date; however, James does have an strong understanding of the mess and Christop is a good right hand enforcer regardless that he and I disagree more often than agree.

Cheers,

Andre Hedrick
LAD Storage Consulting Group

No.

A scientific theory is an approximation of observed behaviour WITH NO KNOWN HOLES.

Once there are known holes in the theory, it's not a scientific theory. At best it's an approximation, but quite possibly it's just plain wrong.

And that's my point. Specs are not only almost invariably badly written, they also never actually match reality.

At which point at _best_ it's just an approximation. At worst, it's much worse. At worst, it causes people to ignore reality, and then it becomes religion.

And that's way _way_ too common. People who ignore reality are sadly not at all unusual.

"But the spec says ..." is pretty much always a sign of somebody who has just blocked out the fact that some device doesn't.

So don't talk about specs.

Talk about working code that is _readable_ and _works_.

There's an absolutely mindbogglingly huge difference between the two.

Linus

You are right about scientific theory, but specs are not just a theory. You are mixing "discovery" and "invention".

A scientific theory has to match reality, because the universe deveops independently. There is no way you can enforce your theory down the throat on the "nature".

But the roles of specs are more than that. There are two parts of it:
1. unify/summarize the reality
2. guide future implementations on a unified road

It might do job 1 poorly (simply because the reality is a mess), but if everyone from the point on puts the effort to follow it, job 2 can be done, and it is the real goal. It can do this simply because *humans* can collaborate and be influenced for a goal that could eventually benefit everybody.

Let me add more to the moron/asshole argument:

Anyone that thinks specs are reality is a moron.

Anyone that thinks specs are useless and refuses to collaborate is an asshole. :)

Since "approximation" is equivalent to a "known hole", there is no single scientific theory without known holes out there?! Well that's a segfault in brain - unless you don't consider math science of course.

A scientific theory is just a set of axioms and deduction rules. Not much more by definition...

I think a better explanation of 'scientific theory' is:

an explanation or definition of observed behaviour with no known error

which, is what I believe Linus was trying to say

[cc trimmed]

...that is testable. (Not that this has anything to do with the point.)

While jumping on the thread for no good reason, I might point out (to Luben) that ^w is erase word and ^h is erase character. Your stty settings may vary.

Why not?

There are good, bad and ill-timed specs. The bad ones are ignored. Ill-timed ones gather dust, for example the SCSI Ultra 640 standard (SPI-5) since the industry has bet on SAS; hope they weren't expecting timely driver support from Linux.

As for a good spec how about the Multi Media Command (MMC) set which allows us to read music and data CDs written almost twenty years ago plus many different formats since then. It is currently (in MMC-5) being enhanced to cope with the next big bun fight in that space: Blue ray and HD-DVD (and their various content protection systems).

In the t10.org hierarchy of specs, MMC is subservient to the primary commands (SPC) and the general architecture (SAM). The companies that work in the MMC space (and tend to define that standard) have "bent the rules" over the years. The response from the (different set of) companies that are behind SPC and SAM was to make allowance for MMC.
The process seems to work pretty well and I am unaware of any proposed alternate interfaces. Transports have come and gone but the logical interface has remained.

For SAS we have a well thought out definition for what the interface should be to hardware specific drivers IMO. It is called CSMI, rechristened SDI. The editor of SDI is also the editor of SAS, SAS-1.1 and SAS-2. Judging from his work, he knows his stuff. Unfortunately SDI uses ioctls to define its interface, which is mainly between two kernel layers (if one ignores pass throughs). Sorry, did I mention "ioctl", oh that makes SDI unacceptable. Almost a year ago that is what happened to the first proposed SAS driver for Linux. That decision has pushed Luben (amongst others) into the position he is in now: come up with a "better" design, produce code and then watch it get rejected. So please cut Luben a bit of slack.

Just in case people think that I agree with Luben on using sysfs to represent the whole SAS topology and interesting bits suspended in it (i.e. SAS expanders), then I don't. I am, however, prepared to argue the detail. Since the days of Eric Youngdale I have believed that SCSI Host Bus Adapters (HBAs) should be addressable devices, just like network interfaces. IMO it is the block layer and its associated end devices that are the legacy thinking.

It is ironic that as the author of the SG_IO
passthrough ioctl the "specs" that I am
often asked to help circumvent are the "we
know better" variety built into various layers
of the linux kernel :-)

Doug Gilbert

Hi Doug,

Almost all of the SDI stuff can be extracted by a user space library reading the SAS sysfs domain tree (which is the result of domain discovery).

Pictures of can be found here:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112629509826900&w=2

Since MPT-based hardware does domain discovery in the FIRMWARE, this is why you do not have the domain picture as easily.

What James Bottomley et al. complained back then is that "common code" to SAS should be pulled out in a "common layer".

This layer is the SAS Transport layer.

But in the mean time, LSI came out with SAS, whose driver as you can see had nothing to do with SAS -- it was an MPT driver after all, so James Bottomley decides that whatever else comes along, it would use _his_ design. Whether it works for open transport, he doesn't want to listen. He is using his political power to enforce it.

Sorry, since the discover process keeps an internal representation of "what is out there" via discovery, I just tacked a kobject to each structure I have anyway, and showed it in sysfs.

I thought that was the whole purpose of sysfs -- to show kernel internal structures and dependencies.

Plus, this is what it _actually_ looks in the physical world.

Also you have to admit -- it looks cool:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112629509826900&w=2

Host Adapters are addressable -- if there's an initiator on the domain, the discover process will discover it and it will show up in the SAS sysfs tree.

Indeed.

Luben

On Fri, 2005-09-30 at 17:29 +1000, Douglas Gilbert wrote:

snippage.

I talked to one of the authors of these specs. SDI is an attempt to create an open standard for the somewhat proprietary CSMI spec developed by HP. It is currently languishing in t10 due to the IOCTL problem you describe below (the "no new IOCTL's" doctrine caught them by surprise). There is some thought to going to a write()/read() on a character device model, but this has various problems as well.

That was one of the reasons the MPT Fusion and Adaptec drivers were rejected. There were others as well, such as lack of a SAS transport class.

Andrew

I think that read/write from a char device is going away too.

For this reason I showed the whole picture of the SAS domain in sysfs _and_ created a binary file attr to send/receive SMP requests/responses to control the domain and get attributes ("smp_portal" binary attr of each expander).

It is completely user space driven and a user space library is simple to write.

See drivers/scsi/sas/expander_conf.c which is a user space utility. For the output see Announcement 3:
http://linux.adaptec.com/sas/Announce_2.txt or here:
http://marc.theaimsgroup.com/?l=linux-scsi&m=112629509318354&w=2

The code which implements it is very tiny, at the bottom of drivers/scsi/sas/sas_expander.c

You mean the first Adaptec SAS "adp94xx" driver.

BTW, neither the original "adp94xx", nor the subsequent "aic94xx" Adaptec drivers implmented _any_ ioctls for CSMI/SDI.

Luben

Linus, the world has changed around you.

Take a look at the SAS spec and then at a SAS chip implementation, for example.

(We're talking abou T10 specs, right?)

You are saying I ignored reality?

http://linux.adaptec.com/sas/

Luben

A spec defines how a protocol works and behaves --- *if* it is well-specified and unambiguous, and *if* vendors actually implement the spec correctly. (And sometimes vendors have major economic incentives to cheat and either intentionally violate the specification, or simply not bother to test to make sure whether or not they implemented their hardware correctly.)

Computing history has been literred with specifications that were incompentently written and/or incompentently implemented --- from the disaster known as ACPI, to FDDI (early FDDI networking gear was interoperable only if you bought all of your gear from one vendor, natch), consumer-grade disks which lied about when data had been safely written to iron oxide to garner better Winbench scores, and many, many, many others.

This is one of the reasons why the IETF doesn't bless a networking standard until there are multiple independent, interoperable implementations --- and even _then_ there will be edge cases that won't be caught until much, much later.

In those cases, if you implement something which is religiously adherent to the specification, and it doesn't interoperate with the real world (i.e., everybody else, or some large part of the industry) --- do you claim that you are right because you are following the specification, and everyone else in the world is wrong? Or do you adapt to reality? People who are too in love with specifications so that they are not willing to be flexible will generally not be able to achieve complete interoperability. This is the reason for the IETF Maxim --- be conservative in what you send, liberal in what you will accept. And it's why interoperability testing and reference implementations are critical.

But it's also important to remember when there is a reference implementation, or pseudo-code in the specification, it's not the only way you can implement things. Very often, as Linus has pointed out, there are reasons why the pseudo-code in the specification is wholely inappropriate for a particular implementation. But that's OK; the implementation can use a different implementastion, as long as the result is interoperable.

Regards,

- Ted

Linus,

I have to tip my hat to you sir.

As much as I wanted to believe and tried to make it happen ... ATA/IDE was forced to design many exception case events. Regardless how hard I an others tried to invoke/create a driver to mimic the "SPEC", the hardware people broke most of the rules and each chipset was littered with exception cases.

It has been 7 years since you and I started butting heads, and in the end both of us were right. A driver could be written to follow the standard exactly, and it would never work (alone, as-is) because the hardware was not paying attention the rules book.

Hope you can kick back and laugh about the history, too!

Have a great Day!

Andre Hedrick
LAD Storage Consulting Group

Jeff Garzik wrote:
[...]

So it could at least be _halfway_ merged, like it was done with the other non-SPI transport layers long ago, couldn't it?

Stefan Richter
-=====-=-=-= =--= ===-=
http://arcgraph.de/sr/

Not just for SAS.

Sure, but the same is true for the other transports except for SPI.

Stefan Richter
-=====-=-=-= =--= ===-=
http://arcgraph.de/sr/

BTW, other SCSI transport layers in Linux do this too, and have been doing so for some time. So this is hardly a valid reason not to include the new SAS layer.

Stefan Richter
-=====-=-=-= =--= ===-=
http://arcgraph.de/sr/

These patches implement NUMA memory node emulation for regular i386 PC:s.

NUMA emulation could be used to provide coarse-grained memory resource control using CPUSETS. Another use is as a test environment for NUMA memory code or CPUSETS using an i386 emulator such as QEMU.

A similar feature was accepted for x86_64 back in 2.6.9. These patches use the same config options and kernel command line parameters as the x86_64 code.

Patches that depend on 2.6.14-rc2:

[PATCH 01/07] i386: srat non acpi
[PATCH 02/07] i386: numa on non-smp
[PATCH 03/07] cpuset: smp or numa

Patches that depend on 2.6.14-rc2 plus two patches written by Dave Hansen and posted to lkml and linux-mm at Sep 13 2005:

i386: consolidate discontig functions into normal ones
i386: move NUMA code into numa.c

[PATCH 04/07] i386: numa warning fix
[PATCH 05/07] i386: sparsemem on pc
[PATCH 06/07] i386: discontigmem on pc
[PATCH 07/07] i386: numa emulation on pc

To test, configure your i386 kernel with CONFIG_X86_PC, CONFIG_NUMA_EMU and CONFIG_NUMA all set and pass "numa=fake=2" to the kernel to emulate two nodes.

Feedback is very appreciated.

Thanks,

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch adds code to check the return value of acpi_find_root_pointer(). Without this patch systems without ACPI support such as QEMU crashes when booting a NUMA kernel configured with CONFIG_ACPI_SRAT=y.

---

 srat.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletion(-)

@@ -327,7 +327,12 @@ int __init get_memcfg_from_srat(void)
 	int tables = 0;
 	int i = 0;
 
-	acpi_find_root_pointer(ACPI_PHYSICAL_ADDRESSING, rsdp_address);
+	if (ACPI_FAILURE(acpi_find_root_pointer(ACPI_PHYSICAL_ADDRESSING, 
+						rsdp_address))) {
+		printk("%s: System description tables not found\n",
+		       __FUNCTION__);
+		goto out_err;
+	}
 
 	if (rsdp_address->pointer_type == ACPI_PHYSICAL_POINTER) {
 		printk("%s: assigning address to rsdp\n", __FUNCTION__);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch makes it possible to compile and use CONFIG_NUMA without CONFIG_SMP. Useful for NUMA emulation on real or emulated UP hardware.

---

 asm-i386/topology.h |    7 ++++++-
 linux/topology.h    |    2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)

@@ -29,8 +29,9 @@
 
 #ifdef CONFIG_NUMA
 
-#include <asm/mpspec.h>
+#ifdef CONFIG_SMP
 
+#include <asm/mpspec.h>
 #include <linux/cpumask.h>
 
 /* Mappings between logical cpu number and node number */

@@ -88,6 +89,10 @@ static inline int node_to_first_cpu(int 
 	.nr_balance_failed	= 0,			\
 }
 
+#else
+#include <asm-generic/topology.h>
+#endif
+
 extern unsigned long node_start_pfn[];
 extern unsigned long node_end_pfn[];
 extern unsigned long node_remap_size[];

@@ -158,7 +158,7 @@
 	.nr_balance_failed	= 0,			\
 }
 
-#ifdef CONFIG_NUMA
+#if defined(CONFIG_NUMA) && defined(CONFIG_SMP)
 #ifndef SD_NODE_INIT
 #error Please define an appropriate SD_NODE_INIT in include/asm/topology.h!!!
 #endif

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch for makes it possible to compile and use CONFIG_CPUSETS without CONFIG_SMP. Useful for NUMA emulation on real or emulated UP hardware.

---

 init/Kconfig    |    2 +-
 kernel/cpuset.c |    2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

@@ -245,7 +245,7 @@ config IKCONFIG_PROC
 
 config CPUSETS
 	bool "Cpuset support"
-	depends on SMP
+	depends on SMP || NUMA
 	help
 	  This option will let you create and manage CPUSETs which
 	  allow dynamically partitioning a system into sets of CPUs and

@@ -657,6 +657,7 @@ static int validate_change(const struct 
 
 static void update_cpu_domains(struct cpuset *cur)
 {
+#ifdef CONFIG_SMP
 	struct cpuset *c, *par = cur->parent;
 	cpumask_t pspan, cspan;
 

@@ -694,6 +695,7 @@ static void update_cpu_domains(struct cp
 	lock_cpu_hotplug();
 	partition_sched_domains(&pspan, &cspan);
 	unlock_cpu_hotplug();
+#endif
 }
 
 static int update_cpumask(struct cpuset *cs, char *buf)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch contains a warning fix for the NUMA patch written by Dave Hansen which was posted to lkml and linux-mm at September 13:th 2005.

[snip]
CC arch/i386/mm/numa.o
arch/i386/mm/numa.c: In function `remap_numa_kva':
arch/i386/mm/numa.c:85: warning: implicit declaration of function `set_pmd_pfn'
LD arch/i386/mm/built-in.o
[snip]

---

 pgtable-3level.h |    1 -
 pgtable.h        |    2 ++
 2 files changed, 2 insertions(+), 1 deletion(-)

@@ -65,7 +65,6 @@ static inline void set_pte(pte_t *ptep, 
 		set_64bit((unsigned long long *)(pmdptr),pmd_val(pmdval))
 #define set_pud(pudptr,pudval) \
 		(*(pudptr) = (pudval))
-extern void set_pmd_pfn(unsigned long vaddr, unsigned long pfn, pgprot_t flags);
 
 /*
  * Pentium-II erratum A13: in PAE mode we explicitly have to flush

@@ -327,6 +327,8 @@ static inline pte_t pte_modify(pte_t pte
 #define pmd_large(pmd) \
 ((pmd_val(pmd) & (_PAGE_PSE|_PAGE_PRESENT)) == (_PAGE_PSE|_PAGE_PRESENT))
 
+extern void set_pmd_pfn(unsigned long vaddr, unsigned long pfn, pgprot_t flags);
+
 /*
  * the pgd page can be thought of an array like this: pgd_t[PTRS_PER_PGD]
  *

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch for enables and fixes sparsemem support on i386. This is the same patch that was sent to linux-kernel on September 6:th 2005, but this patch includes up-porting to fit on top of the patches written by Dave Hansen.

---

 Kconfig        |    4 ++--
 kernel/setup.c |    1 +
 2 files changed, 3 insertions(+), 2 deletions(-)

@@ -762,7 +762,6 @@ config NUMA
 	depends on SMP && HIGHMEM64G && (X86_NUMAQ || X86_GENERICARCH || (X86_SUMMIT && ACPI))
 	default n if X86_PC
 	default y if (X86_NUMAQ || X86_SUMMIT)
-	select SPARSEMEM_STATIC
 
 # Need comments to help the hapless user trying to turn on NUMA support
 comment "NUMA (NUMA-Q) requires SMP, 64GB highmem support"

@@ -801,7 +800,8 @@ config ARCH_DISCONTIGMEM_DEFAULT
 
 config ARCH_SPARSEMEM_ENABLE
 	def_bool y
-	depends on NUMA
+	depends on NUMA || (X86_PC && EXPERIMENTAL)
+	select SPARSEMEM_STATIC
 
 config ARCH_SELECT_MEMORY_MODEL
 	def_bool y

@@ -390,6 +390,7 @@ int __init get_memcfg_numa_flat(void)
 	/* Run the memory configuration and find the top of memory. */
 	node_start_pfn[0] = 0;
 	node_end_pfn[0] = max_pfn;
+	memory_present(0, 0, max_pfn);
 
         /* Indicate there is one node available. */
 	nodes_clear(node_online_map);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'll post a more comprehensive way to do this in just a moment.

Subject: memhotplug testing: hack for flat systems

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Looks much better, will compile and test on Monday. Thanks.

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch enables and fixes discontigmem support for i386.

---

 arch/i386/Kconfig         |    8 ++++++--
 include/asm-i386/mmzone.h |    3 ++-
 include/linux/mmzone.h    |    5 +++++
 include/linux/numa.h      |    2 +-
 mm/Kconfig                |    2 +-
 5 files changed, 15 insertions(+), 5 deletions(-)

@@ -790,9 +790,13 @@ config HAVE_ARCH_ALLOC_REMAP
 	depends on NUMA
 	default y
 
+config ARCH_FLATMEM_ENABLE
+	def_bool y
+	depends on X86_PC
+
 config ARCH_DISCONTIGMEM_ENABLE
 	def_bool y
-	depends on NUMA
+	depends on NUMA || (X86_PC && EXPERIMENTAL)
 
 config ARCH_DISCONTIGMEM_DEFAULT
 	def_bool y

@@ -812,7 +816,7 @@ source "mm/Kconfig"
 config HAVE_ARCH_EARLY_PFN_TO_NID
 	bool
 	default y
-	depends on NUMA
+	depends on NUMA || DISCONTIGMEM
 
 config HIGHPTE
 	bool "Allocate 3rd-level pagetables from highmem"

@@ -75,7 +75,7 @@ static inline int pfn_to_nid(unsigned lo
 #endif
 }
 
-#define node_localnr(pfn, nid)		((pfn) - node_data[nid]->node_start_pfn)
+#define node_localnr(pfn, nid)		((pfn) - NODE_DATA(nid)->node_start_pfn)
 
 /*
  * Following are macros that each numa implmentation must define.

@@ -106,6 +106,7 @@ static inline int pfn_to_nid(unsigned lo
 ({									\
 	unsigned long __pfn = pfn;					\
 	int __node  = pfn_to_nid(__pfn);				\
+	int foo = (&foo == &__node); /* disable unused warning */	\
 	&NODE_DATA(__node)->node_mem_map[node_localnr(__pfn,__node)];	\
 })
 

@@ -414,7 +414,12 @@ extern struct pglist_data contig_page_da
 #define NODE_DATA(nid)		(&contig_page_data)
 #define NODE_MEM_MAP(nid)	mem_map
 #define MAX_NODES_SHIFT		1
+
+#ifdef CONFIG_DISCONTIGMEM
+#include <asm/mmzone.h>
+#else
 #define pfn_to_nid(pfn)		(0)
+#endif
 
 #else /* CONFIG_NEED_MULTIPLE_NODES */
 

@@ -3,7 +3,7 @@
 
 #include <linux/config.h>
 
-#ifndef CONFIG_FLATMEM
+#ifdef CONFIG_NUMA
 #include <asm/numnodes.h>
 #endif
 

@@ -84,7 +84,7 @@ config FLAT_NODE_MEM_MAP
 #
 config NEED_MULTIPLE_NODES
 	def_bool y
-	depends on DISCONTIGMEM || NUMA
+	depends on NUMA
 
 config HAVE_MEMORY_PRESENT
 	def_bool y

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch adds NUMA emulation for i386 on top of the fixes for sparsemem and discontigmem. NUMA emulation already exists for x86_64, and this patch adds the same feature using the same config option CONFIG_NUMA_EMU. The kernel command line option used is also the same as for x86_64.

Pass "numa=fake=N" to the kernel where N is the number of nodes to emulate.

---

 arch/i386/Kconfig           |   20 +++++++-
 arch/i386/kernel/setup.c    |   34 +++++++++-----
 arch/i386/mm/numa.c         |  100 ++++++++++++++++++++++++++++++++++++++++++++ include/asm-i386/mmzone.h   |    7 +++
 include/asm-i386/numnodes.h |    2
 5 files changed, 145 insertions(+), 18 deletions(-)

@@ -134,7 +134,7 @@ endchoice
 config ACPI_SRAT
 	bool
 	default y
-	depends on NUMA && (X86_SUMMIT || X86_GENERICARCH)
+	depends on NUMA && (X86_SUMMIT || X86_GENERICARCH || NUMA_EMU)
 
 config X86_SUMMIT_NUMA
 	bool

@@ -756,12 +756,21 @@ config X86_PAE
 	depends on HIGHMEM64G
 	default y
 
+config NUMA_EMU
+	bool "Numa Memory Nodes Emulation"
+	depends on X86_PC
+	default n
+	help
+	  Enable NUMA emulation. A regular single-node PC machine will be
+	  split into virtual nodes when booted with "numa=fake=N", where
+	  N is the number of nodes.
+
 # Common NUMA Features
 config NUMA
 	bool "Numa Memory Allocation and Scheduler Support"
-	depends on SMP && HIGHMEM64G && (X86_NUMAQ || X86_GENERICARCH || (X86_SUMMIT && ACPI))
+	depends on (NUMA_EMU && ACPI && HIGHMEM) || (SMP && HIGHMEM64G && (X86_NUMAQ || X86_GENERICARCH || (X86_SUMMIT && ACPI)))
 	default n if X86_PC
-	default y if (X86_NUMAQ || X86_SUMMIT)
+	default y if (X86_NUMAQ || X86_SUMMIT || NUMA_EMU)
 
 # Need comments to help the hapless user trying to turn on NUMA support
 comment "NUMA (NUMA-Q) requires SMP, 64GB highmem support"

@@ -770,6 +779,9 @@ comment "NUMA (NUMA-Q) requires SMP, 64G
 comment "NUMA (Summit) requires SMP, 64GB highmem support, ACPI"
 	depends on X86_SUMMIT && (!HIGHMEM64G || !ACPI)
 
+comment "NUMA (Emulation on PC) requires highmem support and ACPI"
+	depends on X86_PC && (!HIGHMEM || !ACPI)
+
 config HAVE_ARCH_BOOTMEM_NODE
 	bool
 	depends on NUMA

@@ -916,7 +928,7 @@ config IRQBALANCE
 # Summit needs it only when NUMA is on
 config BOOT_IOREMAP
 	bool
-	depends on (((X86_SUMMIT || X86_GENERICARCH) && NUMA) || (X86 && EFI))
+	depends on (((X86_SUMMIT || X86_GENERICARCH || NUMA_EMU) && NUMA) || (X86 && EFI))
 	default y
 
 config REGPARM

@@ -931,6 +931,13 @@ static void __init parse_cmdline_early (
 			elfcorehdr_addr = memparse(from+11, &from);
 #endif
 
+#ifdef CONFIG_NUMA_EMU
+		// virtual numa setup
+		else if (!memcmp(from, "numa=", 5)) {
+			extern void numa_setup(char*, char**);
+			numa_setup(from+5, &from);
+		}
+#endif
 		/*
 		 * highmem=size forces highmem to be exactly 'size' bytes.
 		 * This works even on boxes that have no highmem otherwise.

@@ -1211,26 +1218,22 @@ static inline unsigned long  nid_size_pa
 {
 	return node_end_pfn[nid] - node_start_pfn[nid];
 }
-static inline int nid_starts_in_highmem(int nid)
-{
-	return node_start_pfn[nid] >= max_low_pfn;
-}
-
 void __init nid_zone_sizes_init(int nid)
 {
 	unsigned long zones_size[MAX_NR_ZONES] = {0, 0, 0};
-	unsigned long max_dma;
+	unsigned long max_dma = min(max_hardware_dma_pfn(), max_low_pfn);
 	unsigned long start = node_start_pfn[nid];
 	unsigned long end = node_end_pfn[nid];
 
 	if (node_has_online_mem(nid)){
-		if (nid_starts_in_highmem(nid)) {
-			zones_size[ZONE_HIGHMEM] = nid_size_pages(nid);
-		} else {
-			max_dma = min(max_hardware_dma_pfn(), max_low_pfn);
-			zones_size[ZONE_DMA] = max_dma;
-			zones_size[ZONE_NORMAL] = max_low_pfn - max_dma;
-			zones_size[ZONE_HIGHMEM] = end - max_low_pfn;
+		if (start < max_dma) {
+			zones_size[ZONE_DMA] = min(end, max_dma) - start;
+		}
+		if (start < max_low_pfn && max_dma < end) {
+			zones_size[ZONE_NORMAL] = min(end, max_low_pfn) - max(start, max_dma);
+		}
+		if (max_low_pfn <= end) {
+			zones_size[ZONE_HIGHMEM] = end - max(start, max_low_pfn);
 		}
 	}
 

@@ -1270,7 +1273,12 @@ void __init setup_bootmem_allocator(void
 	/*
 	 * Initialize the boot-time allocator (with low memory only):
 	 */
+#ifdef CONFIG_NUMA_EMU
+	bootmap_size = init_bootmem(max(min_low_pfn, node_start_pfn[0]),
+				    min(max_low_pfn, node_end_pfn[0]));
+#else
 	bootmap_size = init_bootmem(min_low_pfn, max_low_pfn);
+#endif
 
 	register_bootmem_low_pages(max_low_pfn);
 

@@ -18,6 +18,9 @@ extern struct pglist_data *node_data[];
 	#include <asm/srat.h>
 #endif
 
+#ifdef CONFIG_NUMA_EMU
+extern int get_memcfg_numa_emu(void);
+#endif
 extern int get_memcfg_numa_flat(void );
 /*
  * This allows any one NUMA architecture to be compiled

@@ -33,6 +36,10 @@ static inline void get_memcfg_numa(void)
 	if (get_memcfg_from_srat())
 		return;
 #endif
+#ifdef CONFIG_NUMA_EMU
+	if (get_memcfg_numa_emu())
+		return;
+#endif
 
 	get_memcfg_numa_flat();
 }

@@ -8,7 +8,7 @@
 /* Max 16 Nodes */
 #define NODES_SHIFT	4
 
-#elif defined(CONFIG_ACPI_SRAT)
+#elif defined(CONFIG_ACPI_SRAT) || defined(CONFIG_NUMA_EMU)
 
 /* Max 8 Nodes */
 #define NODES_SHIFT	3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

...

Ewwwwww :) No real need to put new function in a big #ifdef like that. Can you just create a new file for NUMA emulation?

Hi again Dave,

Break it out, sure! I'm not sure which patch to look at, though.

Do you mean "discontig-consolidate0.patch"? Maybe I'm misunderstanding.

Hehe, what is this, a beauty contest? =) I agree, but I guess the reason for this code to be here is that a similar arrangement is done by x86_64...

I will create a new file. Is arch/i386/mm/numa_emu.c good?

No biggie, I will add a config option.

But first, you have written lots and lots of patches, and I am confused. Could you please tell me on which patches I should base my code to make things as easy as possible?

Many thanks,

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This one, I believe:

http://sr71.net/patches/2.6.14/2.6.14-rc2-git8-mhp1/broken-out/B2.1-i386-discontig-consolidation.patch

If that's really the case, can they _actually_ share code? Maybe we can do this NUMA emulation thing in non-arch code. Just guessing...

This is the staging ground for my memory hotplug work. But, it contains all of my work on other stuff, too. If you build on top of this, it would be great:

http://sr71.net/patches/2.6.14/2.6.14-rc2-git8-mhp1/

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'd like to avoid duplication as much as you, but at a quick glance the x86_64 and i386 architecture looked pretty different. But I will see what I can do.

I will build on top of that then.

Thanks,

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

It seems like you've forgot to bind cpus with emulated nodes as linux for x86_64 does. I don't think it's your intention.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

True, not my intention. I will have a look at that. Thanks.

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch set basically allows the "NUMA depends on SMP" dependency to be removed. I'm not sure this is the right approach. There will likely never be a real-world NUMA system without SMP. So, this set would seem to include some increased (#ifdef) complexity for supporting SMP && ! NUMA, which will likely never happen in the real world.

Also, I worry that simply #ifdef'ing things out like CPUsets' update means that CPUsets lacks some kind of abstraction that it should have been using in the first place. An #ifdef just papers over the real problem.

I think it would likely be cleaner if the approach was to emulate an SMP NUMA system where each NUMA node simply doesn't have all of its CPUs online.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, this patch set removes "NUMA depends on SMP". It also adds some simple NUMA emulation code too, but I am sure you are aware of that! =)

I agree that it is very unlikely to find a single-processor NUMA system in the real world. So yes, "[PATCH 02/07] i386: numa on non-smp" adds _some_ extra complexity. But because SMP is set when supporting more than one cpu, and NUMA is set when supporting more than one memory node, I see no reason why they should be dependent on each other. Except that they depend on each other today and breaking them loose will increase complexity a bit.

Maybe. CPUSETS has two bitmaps, one for cpus and one for mems. So depending on SMP or NUMA seems logical to me. Regarding the #ifdef, it was added because partition_sched_domain() is only implemented for SMP. That symbol has no prototype or implementation when CONFIG_SMP is not set. Maybe it is better to add an empty inline function in linux/sched.h for !SMP?

Absolutely. And that removes the need for some of my patches. QEMU runs SMP kernels. It is possible to run SMP kernels on UP hardware. But there is of course a certain performance loss introduced by all the SMP locks. I'd rather not force !SMP users to run SMP kernels if they want coarse-grained memory resource control.

Thanks for your input!

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

In the abstract, cpusets should just assume that the system has one or more CPUs, and one or more Memory Nodes. Ideally, it should not require either SMP nor NUMA. Indeed, if you (Magnus) can get it to compile with just one or the other of those two:

config CPUSETS
bool "Cpuset support"
- depends on SMP
+ depends on SMP || NUMA

then I would hope that it would compile with neither. The cpuset hierarchy on such a system would be rather boring, with all cpusets having the same one CPU and one Memory Node, but it should work ... in theory of course.

In practice of course, there may be details on the edges that depend on the current SMP/NUMA limitations, such as:

An empty inline partition_sched_domain() would be better than ifdef's in cpuset.c, yes. Or at least, that's usually the case. Probably here too.

In theory at least, I applaud Magnus's work here. The assymetry of the SMP/NUMA define structure has always annoyed me slightly, and only been explainable in my view as a consequence of the historical order of development. I had a PC with a second memory board in an ISA slot, which would qualify as a one CPU, two Memory Node system.

Or what byte us in the future (that PC was a long time ago), the kinks in the current setup might be a hitch in our side as we extend to increasingly interesting architectures.

Aside - for those reading this thread on lkml, it originated on linux-mm. It looks like Dave added lkml to the cc list.

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I just tested this on top of my patches:
@@ -245,7 +245,6 @@ config IKCONFIG_PROC

config CPUSETS
bool "Cpuset support"
- depends on SMP || NUMA
help

and it seems to work ok in practice too. On a regular !SMP !NUMA PC anyway. As you note, the hierarchy is not that exciting. =) Anyway, both SMP || NUMA or nothing seems to work as dependencies. After partition_sched_domain() gets fixed that is.

I agree.

Nice to hear that you like the idea.

Maybe I should have broken down my patches into three smaller sets:

1) i386: NUMA without SMP
2) CPUSETS: NUMA || SMP
3) i386: NUMA emulation

If people like 1) then it's probably a good idea to convert other architectures too. Both 2) and 3) above are separate but related issues. And now seems like a good time to solve 2).

So, Paul, please let me know if you prefer SMP || NUMA or no depencencies in the Kconfig. When I know that I will create a new patch that hopefully can get into -mm later on.

Huh? I sent my patches both to lkml and linux-mm...

Thank you for the feedback!

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

The latter seems a good idea to me if you're going to enhance CPUSETS acceptable for CPUMETER or something like that.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

In theory, I prefer none. But the devil is in the details here, and I really don't care that much.

So pick whichever you prefer, or whichever provides the nicest looking code or patch, or flip a coin ;).

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm tempted to consult the magic eight-ball, but I think I will stick with the advice from Takahashi-san instead. =) So, the dependency will be removed.

/ magnus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes - Takahashi-san gives much better advice than an eight-ball.

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>