2.2.18Pre Lan Performance Rocks!

Alan,

I don't know what all changes you guys did to 2.2.18pre, but the LAN I/O performance absolutely rocks on our tests here vs. NetWare. Good job. It's still got some problems with NFS (I am seeing a few RPC timeout errors) so I am backreving to 2.2.17 for the Ute-NWFS release next week, but it's most impressive.

:-)

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Yes. I just went home, so I am emailing from my house. I'll post late tonight or in the morning. Performance on 100Mbit with NFS going Linux->Linux is getting better throughput than IPX NetWare Client -> NetWare 5.x on the same network by @ 3%. When you start loading up a Linux server, it drops off sharply and NetWare keeps scaling, however, this does indicate that the LAN code paths are equivalent relative to latency vs. MSM/TSM/HSM in NetWare. NetWare does better caching (but we'll fix this in Linux next). I think the ring transitions to user space daemons are what are causing the scaling problems Linux vs. NetWare.

:-)

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

There are no user space daemons involved in the knfsd fast path, only in slow paths like mounting.
The main problem I think in knfsd are the numerous copies of the data (e.g. 2+checksumming for RX with fragments, upto 4 in some specific configurations). They're unfortunately not trivial to fix. TX is a bit better, it does only one copy usually out of the page cache. For RX it also helps to have a network card that supports hardware checksumming.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

So why is it spawning off nfsd servers in user space? I did notice that all the connect logic is down in the kernel with the RPC stuff in xprt.c, and this looks to be pretty fast. I know about the checksumming problem with TCPIP. But cycles are cheap on todays processors, so even with this overhead, it could still get faster. IPX uses small packets that are less wire efficient since the ratio of header size to payload is larger than what NFS in Linux is doing, plus there's more of them for an equivalent data transfer, even
with packet burst. IPX would tend to be faster if there were multiple
routers involved, since the latency of smaller packets would be less and IPX never has to deal with the problem of fragmentation.

Is there any CR3 reloading or tasking switching going on for each interrupt that comes in you know of, BTW? EMON stats show the server's bus utilization to get disproportiantely higher in Linux than NetWare 5.x and when it hits 60% of total clock cycles, Linux starts dropping off. NetWare 5.x is 1/8 the bus utilitization, which would suggest clocks are being used for something other than pushing packets in and out of the box (the checksumming is done in the tcp code when it copies the fragments, which is very low overhead).

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

They just provide a process context, the actual work is only done in kernel mode.

No, interrupts do not change CR3.

One problem in Linux 2.2 is that kernel threads reload their VM on context switch (that would include the nfsd thread), this should be fixed in 2.4 with lazy mm. Hmm actually it should be only fixed for true kernel threads that have been started with kernel_thread(), the "pseudo kernel threads" like nfsd uses probably do not get that optimization because they don't set their MM to init_mm.

I think that can be explained by the copying.

To be sure you could e.g. use the ktrace patch from IKD, it will give you cycle accurate traces of execution of functions.

No, unfortunately not. NFS uses UDP and the UDP defragmenting doesn't do copy-checksum currently.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

yes, but for this there is an explicit mechanizm to lazy-MM during lengthy system calls, an example is in buffer.c:

                user_mm = start_lazy_tlb();
                error = sync_old_buffers();
                end_lazy_tlb(user_mm);

yes. Constant copying contaminates the L1/L2 caches and creates dirty cachelines all around the place. Fixed in 2.4 + TUX ;-)

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ingo, we need a build option to completely disable multiple address spaces for a start, and just map everything to a linear address space. This will eliminate the overhead of the CR3 activity. The use of segment registers for copy_to_user, etc. causes segment register reloads, which are very heavyweight on Intel.

Is there an option to map Linux into a flat address space like NetWare so I can do an apples to apples comparison of raw LAN I/O scaling?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

nope, Linux is fundamentally multitasked.

what you can do to hack around this is to not switch to the idle thread after having done work in nfsd. Some simple & stupid thing in schedule:

	if (next == idle_task) {
		while (nr_running)
			barrier();
		goto repeat_schedule;
	}

(provided you are testing this on a UP system.) This way we do not destroy the TLB cache when we wait a few microseconds for the next network interrupt.

we do this in 2.4 already - ie. nfsd doesnt have to mark itself lazy-MM, the idle thread will automatically 'inherit' the MM of nfsd, and is going to switch CR3 only if the next process is not nfsd. So you can get an apples to apples comparison by using 2.4.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

uClinux may be able to do this, at the cost of a dramatically reduced userspace functionality.

dwmw2


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

David/Alan,

Andre Hedrick is now the CTO of TRG and Chief Scientist over Linux Development. After talking
to him, we are going to do our own ring 0 2.4 and 2.2.x code bases for the MANOS merge.
the uClinux is interesting, but I agree is limited.

MANOS schedules should be unaffected. The current DLL prototype of Linux 2.2 is ring 0, but I shudder at trying to merge all the changes I've done to it into core 2.2.X as a .config
option. There's also the gravity well forces of different views to this effort. With Andre
on the job, I am more confident in co-opting the Linux drivers and just biting the bullet
on the support issues, and doing a full fork of Linux.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Jeff,

What would be missed out in this approach:
* Use Montavista "fully" preemtible kernel.
* Using Kernel threads for all services (File, Print, Web, etc.).

/RogerL

--
Home page:
http://www.norran.net/nra02596/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ingo, I will attempt this, but I doubt seriously it will allow Linux to defeat NetWare 5.x on LAN I/O scaling. ALl protection has to go away in all LAN paths for this to happen, and user space apps set to ring 0. NetWare 5.cx does support ring 3 applications, but the model is different than Linux. I will look at a potential
compromise between the two with the MANOS /arch merge. It may
allow some incarnation of Linux to smoke NetWare 5.x on LAN performance. Doing this would move MARS-NWE and SAMBA into the kernel without changing a line of code in either...

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

i found that this is not a requirement for good network scalability. We do not do a syscall for every packet, so the cost evens out. Sure, it does not hurt to not eat ~1 microsecond per system-call, but it causes no overhead or scalability limit otherwise. In the TUX webserver we have user-space modules doing context-switch-less webserving, and it scales quite well, and is generic.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

No argument here, but the overhead of reloading CR3 period will kill performance. 2.4 does not beat NetWare, BTW, it gets a little further, but still hits the wall, and Netware keeps going strong, and scales up to 5000 users on file and print, with a SINGLE processor at less < 40% utilization. Linux hits the wall at a few hundred file and print users. It's the overhead of CR3 switching at all that is causing this and the heavy usage of Intel's protection model.

For example, if you put a MOV EAX, CR3; MOV CR3, EAX; in a context switching path, on a PPro 200, you can do about 35,000 context switches/second (I did this in NetWare 4.x in 1994) with the same code without the CR3 reload, I could do over 2,000,000 context switches/second without the CR3 load. INVL Page[0] in the same path is less heavy, but still reduced context switches to 135,000/second. It's the CR3 activity period that kills performance. There's also the use of segment registers all over the place to copy from kernel to user and user to kernel space memory. Having the fast paths you mention does help a lot, but it's the fact that this goes on at all that will make it tough to walk into a NetWare shop with Linux and rip out NetWare servers and replace them unless we look at a NetWare vs. NetLinux (that's what we call it! a NetWare-like Linux platform).

If we had this, MS would be running for the trenches. Coupling the internet application base of Linux with the speed of NetWare would be Mr. Gate's worst nightmare. They've been trying to kill off NetWare for almost 15 years, and it's still out there and they are still slower, and everyone knows it. It's not Linux or NT that's killing off NetWare, right now, it's the incompetent management in San Jose who are trying to rape Novell's rich bank accounts
(Schmidt and Sonsini and friends), and the fact they have no IA64 NetWare (they would be shipping it not if I were still there ....).

It could be as easy as a compile option in .config (NETWARE_SERVER_MODE [Y/N]) with WTD (which is a more sophistacted method for doing lazy VM and controlling context switching activity for fast path I/O) and mapping of te Linux address space as linear -- you can still have protection with this model, just some apps would be able to load in the kernel address space, and run at ring 0).

:-)

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

2.4 does not reload CR3, unless you are using multiple user-space processes.

as i told you in the previous mail, the main overhead is not CR3, it's the copying & dirtying of all data, and the subsequent DMA-initiated dirty cacheline writeback. I can serve 100 MB/sec web content with 2.4 & TUX just fine - it relies on a zero-copying infrastructure.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Great Ingo, you've got the web server covered. What about file and print. I think this is great, but most web servers are connected to a T1 or T3 line, and all the fancy optimization means absolutely squat since about 99.999999% of the planet has a max baandwidth of a T1, ADSL, or T3 Line, this is a far cry from Gigabit ethernet, or even 100Mbit ethernet.

How many users can you put on the web server? Web servers are also read only data, the easiest of all LAN cases to deal with. It's incoming writes that present all the tough problems, as Andi Klein wisely observed. Not to knock Tux, I think it's great, but it does not solve the file and print scaling problem, no does it?

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

a web server, as you probably know, is a read-mostly fileserver that serves files via the HTTP protocol. The rest is only protocol fluff.

Your argument is curious - first you cry for performance, then you say 'nobody wants that much bandwidth'. Of course, if the network is bandwidth limited then we cannot scale above that bandwidth. But thats not the point. The point is to put 10 cards into a server and still being able to saturate them. The point is also to spend less cycles on saturating available bandwidth. The point is also to not flush the L1 just because someone requested a 10K webpage.

tens of thousands, on a single CPU. Can probably handle over 100 thousand users as well, with IP aliasing so the socket space is spread out.

reads dominate writes in almost all workloads, thats common wisdom. Why write if nobody reads the data? And while web servers are mostly read only data, they can write data as well, see POST and PUT. The fact that incoming writes are hard should not let you distract from the fact that reads are also extremely important.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

It's not curious, it's not about bandwidth, it's about latency, and getting packets in and out of the server as fast as possible, and ahead of everything else. Cache affinity and all the Tux stuff is a great piece of work. Let's talk about file and print, that's the present problem, not how to pump out read-only data from a web server.

Web servers don't do writes, unless a CGI script is running somewhere or some Java or Perl or something, then this stuff goes through a wrapper, which is slow, or did I miss something.

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

yes, you missed TUX modules.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Great. I can load a TUX module and use it with my T1 line. If I could spare an extra $100,000/month, perhaps I can lease an SDM-172 or TAT-8, or even an OC-172 then I would be able to take advantage of it in the real world.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

TUX prepares a HTTP reply in about 30 microseconds (plus network latency), good enough? Network latency is the limit, even on gigabit - not to talk about T1 lines.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Great. Now how do we get the smae numbers on SAMBA and MARS-NWE? THat's
the question, not whether your baby TUX is pretty. I already said it was pretty, focus on the other issue.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

simple, write a TUX protocol module for it. FTP protocol module is on its way. Stay tuned.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I do not believe this approach will allow Linux to match NetWare's file and print performance, but I am willing to give it a whirl. Where is the TUX module for MARS. Let's start with this one. What help would you require. You understand these TUX modules, so you should take the lead.

I am listening. Instruct me....

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

TUX modules are kernel modules (I mean you have to write kernel space code for doing TUX ftp). Don't you agree that zero-copy sendfile like ftp serving would be able to perform equally well too? I mean: isn't better to spend the efforts to make an userspace API to run fast instead of moving every network functionality that needs high performance completly in kernel? People may need to write high performance network code for custom protocols, this way they will end creating kernel modules with system-crashing bugs, memory leaks and kernel buffer overflows (chroot+nobody+logging won't work anymore). (plus they will get into pain while debugging)

It's obvious kernel code runs faster and you can also do assumptions about scheduler and current CPU that you can't do in userspace, but is that so relevant in term of ftp server numbers compared to only skipping the memory copies?

About the TUX cgi module I had a fast look and I noticed cgis run by tux executes two clones(2) and one exec(2) and then they have to pay the startup of the interpreters for each cgi request. So at this stage of tux I guess that for perl/php pages tux would better redirected them to an apache. Maybe php and perl tux (kernel) modules are in your todo list? (I think they would be a bad idea though) One other way to handle efficiently the interpreted-cgi load without redirecting the cgi request to a full apache is to have a background php/perl interpreter listening to new cgis in input and filling the tux pipe in output.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I'm glad _someone_ is connected to reality with regards the security implications of throwing loads of servers into kernel space.

Cheers
Chris

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

If we implement a ring 0 Linux, all of this will remain intact with the need to
port modules into the kernel at all.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I don't understand what you mean sorry, could you elaborate?

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

For this to bw useful for ftp we need a sendfile() that can write from a socket to a diskfile also.

-Dan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi!

I had patch to fix sendfile this way... Sendfile is really ugly, as of now. (It basically falled back to read/write, giving only small performance advantage, but it made things cleaner).

Pavel

I'm [email protected]. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ingo's helping me get the info together on this for putting a MARS-NWE tux module in
the kernel. He had to go do some things this week he told me before he would be ready
to look at it. He did point me over to the info, and I agreed we would attempt to
implement it as something to look at. If it performs well enough, I will have
something reasonable to send out to Novell Resellers (CNEs) and Cutomers.

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

TUX modules are user-space, so i certainly cannot help you in 'putting MARS-NWE in the kernel'. While you (apparently) are trying to move server applications into ring0, i agree with Andrea and i'm trying to move kernel functionality out to user-space.

All i did was to inform you that the next release of TUX is imminent and that you might want to take a look at the new code. You interpreted that in a very interesting way. You are certainly free and welcome to take a look at any code and documentation released, but as visible in the past couple of email exchanges, our technical views about Linux networking scalability differ in fundamental ways.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I seem to remember a "don't post this email on the list here's what's up" message
from you I that I honored your instructions and did not post, nor will I, but I recall
some reasonable suggestions from it. :-)

You are certainly free and welcome to take a

I've been working on LAN scalability for almost 20 years, and NetWare LAN performance vs. Linux (or TUX) speaks for itself. I would agree that someone
coming from a Unix background would have radically different views than someone
coming from the group who created the Networking industry that made "LAN" a household
word to the world at large.

It's the differences of all those who work on Linux that make it such a potent
mix of skills and views. I think this is a very good thing, and I take your comment
as a compliment.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

in 2.4 & Xeons we can do more than 100,000 context switches/second, and that is more than enough. But the point is: network IO performance does not depend on context switching speed too much. Also, in Linux we are using global pages which makes kernel-space TLBs persistent even across CR3 flushes.

we do not use the fs segment register for user-space copies anymore, neither in 2.2, nor in 2.4. You must be reading old books and probably forgot to cross-check with the kernel source? :-)

the worst thing you can do is to mis-identify performance problems and spend braincells on the wrong problem. The problems limiting Linux network scalability have been identified during the last 12 months by a small team, and solved in TUX. TUX is a fileserver, it shouldnt be alot of work to enable it for (TCP-only?) netware serving. It's *done*, Jeff, it's not a hypotetical thing, it's here, it works and it performs.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

This is putrid. NetWare does 353,00,000/second on a Xenon, pumping out gobs of packets in between them. MANOS does 857,000,000/second. This is terrible. No wonder it's so f_cking slow!!!

ds: and es: are both used in copy-to-user and copy-from-user and they get reloaded.

NetWare is here too, and it handles 5000+ file and print users, Linux does not. Let's fix it. I know why NetWare is fast. Let's apply some of the same principles and see what happens. Love to have you involved.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

(no need to get emotional.) And please check your numbers, 857 million context switches per second means that on a 1 GHZ CPU you do one context switch per 1.16 clock cycles. Wow!

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

And they all share the same segment descriptor. Whats your point? ES is the default target segment for string operations. DS is the default data segment. Have you ever profiled how many cycles it takes to do a "mov __KERNEL_DS, %es" in entry.S, before making your (ridiculous) claim? I have.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

And please check your numbers, 857 million

Excuse me, 857,000,000 instructions executed and 460,000,000 context switches a second -- on a PII system at 350 Mhz. It's due to AGI optimization. Download MANOS and verify for yourself, it has a built in EMON in monitor. After I complete the port, not even NetWare will be able to touch it.

Your Tux web server will also run on it, at significantly increased performance.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

so it does 1.3 context switches per clock cycle? Wow! And i can type 100000000000000000000 characters a second, just measured it. Really!

as i told you in the previous mails, TUX does not depend on schedule() performance. schedule() cost does not even show up in the top 20 entries of the profiler.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Go download it and try it, then come back with that smirk wiped off your face. I'll enjoy it.....

:-)

And as I told, you, your code has nothing to do with it, it's the fact it goes on at all. Ingo, go get a copy of NetWare 3.12 (I'll even send you one -- I've got extra licensed copies), install it, put a load of 5000 connections on it, with 4 adapters. Dual boot Linux on it, and attempt the same with SAMBA or MARS-NWE, and watch it oink.

Go do it. What's your address to I can ship you the CD's for 3.12. Then come back and tell me how TUX is going to solve the File and Print performance issues in Linux.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

so in 0.53 clock cycles you are implementing things like address space separation, process priorities, fairness and other essential scheduling features? Truly awesome ...

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ingo, This original thread was regarding Linux vs. NetWare 5.x performance metrics and responses from Linux folks about how to affect and improve them, not a diatribe on the features of TUX.

I wrote the kernel in NetWare 4.x that later became 5.x. To date, my NOS kernel has grossed over 8 billion dollars. This is more money than all the Linux companies have made combined in their entire history, on your work, Linus's work and everyone else's. I don't have anything to prove to myself, or anyone else, which is why I do as I please in life, and no one's comments or snipes dissuade me from moving forward. I also am not "someone's employee".

If you have some ideas on how to improve file and print or help me get a linux incarnation that can stomp NetWare, I'd love to hear your ideas. I think TUX is great, BTW. Otherwise, end-of-line...

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

oh, i believe you misunderstand. TUX itself is quite simple. But it extended the Linux TCP stack and scalability to new levels (which has nothing to do with TUX itself, it's the scalability of the Linux networking stack that evolved gradually over the past 10 years - we spend 95% of the time outside of TUX!). And if you claim that Linux needs this and that for scalability, then i'd like to point you humbly towards those existing, generic and well-performing results. TUX is just a 5% 'HTTP protocol fluff' around the generic stuff.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

while beating netware in certain areas is certainly a noble goal, it is far from the only one.

[snip]

if netware users are happy with netware, i can't see why they'd want to switch to linux or a linux-netware-ish equivalent while netware are still around to provide support. "ain't broke, don't fix," etc.

j.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

SAMBA and Mars-nwe are running user space thats why. They have flexibility, protection and can run unpriviledged. If you want to put mars-nwe in the kernel then it will certainly be interesting

There's an http file system around 8)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Alan,

I've been studying Linux for the past two years the same way a diamond cutter studies a
prized and immensely valuable raw stone. I think I am nearing the
point I know where to
strike and I can cleave it into something Novell's installed base would like and could move
forward with. The easiest path I see for me to create a Linux NetWare hybrid is for you
guys to get a 2.2.X (not 2.4.X -- not just yet) tree that runs in a flat memory space,
then I can slip in our optimizations and we will be able to run all existing code in kernel
that's trusted. The changes needed to do this involve more than just the /arch area and
the asm macro includes, though this will cover about 80% of it, but the loader would need to
be able to create address domains around aps that choose to run protected.

If I attempt these changes without proper guidance, two things will happen. 1). people who
own some of this code will not be aware of all the aspects of the changes, and may not
longer be able to support it, and 2). it will fork away from the tree and become
unsupportable and diverge over time. This is the reason I have resisted making any
kernel changes myself other than code I write for Linux I own, and I have always
solicitied other folks to make the patches. This is about to change relative to this
project.

The first step I need is to be able to load Linux as a ring 0 OS in linear address space
with all protection disabled. Paging can still go on, just no separate CR3 reloads between
context switches. profiling Ring 0 Linux vs. NetWare will give me an
excellent idea of where
the optimizations will need to be inserted. A straight MARS-NWE port to kernel would just
happen, since we would be able to just load in kernel space and run it with no code
changes.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

There are one bunch of people running Linux on a flat memory space with no protection although their goal was to make Linux run on mmuless embedded hardware.

See www.uclinux.org; the uclinux guys started a 2.4 port recently. Basically the idea is to have a mm-nommu/ directory which implements a mostly compatible replacement for the mm layer (obviously stuff like mmap dont work without an mmu and fork is odd), and a set of binary loaders to load flat binaries with relocations.

That I think is the project that overlaps ..

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Thanks,

It will make merging the MANOS kernel happen faster. My DLL prototypes are using subsets
of Linux 2.2.16 for MANOS at present, and what I really need is for the support issues to dovetail into a supported effort. This one might fit the bill. I have no desire for TRG to support the 100's of LAN and disk drivers all by our little lonesome in a divergent code base.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

mmap works -- you just can't do MAP_PRIVATE. Everything has to be MAP_SHARED. There is no fork (though vfork works). brk doesn't work, and things like iopl, ioperm are pointless. etc...

Oh, and when you do malloc something (malloc is mmap based) you really want to remember to free it, since the system can't clean up after you,

-Erik

--
Erik B. Andersen email: [email protected]
--This message was written using 73% post-consumer electrons-- -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

You're up on that higher plane again. Send me some 'shrooms, please.

Linux has great internet connectivity, and a real application layer with lots of
apps. NetWare customers would love it. I am one and I love it already.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

No. I used a hardware analyzer to show me how many LOCK# assertions it does invisible to your software tools underneath. Try using EMON to profile, it gives hardware numbers and let's you watch the cache controllers issue non-cacheable memory references to fetch the descriptors.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi!

That's more than one context switch per clock. I do not think so. Really go and check those numbers.
Pavel

I'm [email protected]. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

yep, you cannot have 460 million context switches on that system, unless you have some Clintonesque definition for 'context switch' ;-)

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

The numbers don't lie. You know where the code is. You notice that there is a version of
the kernel hand coded in assembly language. You'l also noticed that it's SMP and takes ZERO LOCKS during context switching, in fact, most of the design is completely lockless.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

sure ;) I can do infinite context switches! You dont believe? See:

	#define schedule() do { } while (0)

[there is a small restriction, should only be used in single-task systems.]

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Actually, I think the compiler would optimize this statement completely out of the code.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

[Jeff Merkey]

[Ingo Molnar]

[Jeff]

That was Ingo's point. He is doing infinite context switches per second, where a context switch is defined as "schedule()", because he turned it into a noop.

I.e. the numbers can easily be made to lie, by playing with the rules of the game.

The point of confusion, Jeff, is that to *you* a context switch means stack switch, with no baggage like scheduling or reloading registers. Everyone else here thinks of a context switch as meaning a pre-emptive switch between two unrelated processes -- which as you know involves not only the stack, but MM adjustments, registers, floating point registers (expensive on pre-P6), IP, and some form of scheduling.

Obviously some of these can be optimized out if you can make assumptions about the processes: you might drop memory protection if you like the stability of Windows 95, floating point if you can get away with telling people they can't use it, maybe use FIFO scheduling if you don't care about fairness and you know the processes are more or less uniform. Linux cannot make any of these assumptions -- it is far too general-purpose.

In Linux, in fact, jumping from ring 3 to ring 0 (ie system call) is not considered a context switch. I suppose you would consider it one. So the real question is, how many gettimeofday() per sec can Linux do?

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Oh, about 3,531,073 on a 1Ghz AM thunderbird running Linux disks.bitmover.com 2.4.0-test5.

That's 283.2 nanoseconds per call, to save you the math.

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

[me]

[Larry McVoy]

So, at two "context switches" (Jeff's term) per syscall, we're somewhere around half the speed of Netware's longjmp.

Not bad. (:

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I suspect most of the confusion in this thread comes because you seem to use a different definition of context switch than Ingo and others. Could you explain what you exactly mean with a context switch ?

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

A "context" is usually assued to be a "stack". The simplest of all context switches
is:

mov x, esp
mov esp, y

A context switch can be as short as two instructions, or as big as a TSS with CR3 hardware switching,

i.e.

ltr ax
jmp task_gate

(500 clocks later)

ts->eip gets exec'd

you can also have a context switch that does an int X where X is a gate or TSS.

you can also have a context switch (like linux) that does

    mov    x, esp
    mov    esp, y
    mov    z, CR3
    mov    CR3, w

etc.

In NetWare, a context switch is an in-line assembly language macro that is 2 instructions long for a stack switch and 4 instructions for a CR3 reload -- this is a lot shorter than Linux.
Only EBX, EBP, ESI, and EDI are saved and this is never done in the kernel, but is a natural
affect of the Watcom C compiler. There's also strict rules about register assignments that re enforced between assembler modules in NetWare to reduce the overhead of a context switch. The code path is very complex in NetWare, and priorities and all this stuff exists, but these code paths are segragated so these types of checks only happen once in a while and check a pre-calc'd "scoreboard" that is read only across processors and updated and recal'd by a timer every 18 ticks.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

a very clintonesque definition indeed ;-)

what is relevant is the latency to switch from one process to another one. And this is what we call a context switch. It includes scheduling decisions and all sorts of other stuff. You are comparing stack & caller-saved register switching performance (which is just a small part of context switching and has no standing alone) against full Linux context switch performance (this is what i quoted), and thus you have won my 'Mindcraft benchmark of the day' award :-)

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ingo,

It kicks Linux's but in LAN I/O scaling. It would be nice for Linux to have an incarnation that's competitive. The only reason people are still buying NetWare is because nothing out there has come along to replace it. That is going to change...

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

brain cacheflush? Restart the same thread? Sorry i've got better things to do.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Really? So, since in a few messages back you claimed that it has a fully supported userland which implements all of P1003.1 as well as sockets, obviously, since it is a networking operating system, it should be the work of a few minutes to download LMbench, compile it, and come back with the lat_tcp performance numbers which "kicks Linux's but". Please do so.

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Larry,

What's your mailing address and I'll send you out a legally licensed copy of NetWare 3.12 and transfer the license to you then you can do the comparison and see for yourself.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I have this feeling that this is an April Fools joke. Unfortunately it's Halloween.

One could create a 'kernel' that does:

	for(;;)
        {
          proc0();
          proc1();
          proc2();
          proc3();
          etc();
        }

... and loop forever. All 'tasks' would just be procedures and no context-switching would even be necessary. This is how some network file-servers worked in the past (Vines comes to mind). Since all possible 'tasks' are known at compile-time, there isn't even any need for memory protection because every task cooperates and doesn't destroy anything that it doesn't own.

The only time you need to save anything is for interrupt handlers. This was some simple push/pops of only the registers actually used in the ISR.

Now, the above example may seem absurd, however it's not. Inside each of the proc()'s is a global state-variable that allows the code to start executing at the place it left off the last time through. If the code was written in 'C' it would be a 'switch' statement. The state-variable for each of the procedures is global and can be changed in an interrupt-service-routine. This allows interrupts to change the state of the state-machines.

This kind of 'kernel' is very fast and very effective for things like file-servers and routers, things that do the same stuff over and over again.

However, there techniques are not useful with a kernel that has an unknown number of tasks that execute 'programs' that are not known to the kernel at compile-time, such as a desk-top operating system.

These operating systems require context-switching. This requires that every register that a user could possibly alter, be saved and restored. It also requires that the state of any hardware that a user could be using, also be save and restored. This cannot be done in 2 instructions as stated. Further, this saving and restoring cannot be a side-effect of a particular compiler, as stated.

Cheers,
Dick Johnson

Penguin : Linux version 2.2.17 on an i686 machine (801.18 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of course you get it all back when you reboot..."; Actual explanation obtained from the Micro$oft help desk.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Dick,

In NetWare this:

would be coded like this (no C compiler):

proc0:

proc1:

proc2:

proc3:

etc:

label:
jmp proc0

I just avoided 5 x 20 bytes of pushes and pops on the stack ad optimized for a simple fall through case.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

oh, and what happens if it turns out that some other place wants to call proc3 as well? Recode the assembly - cool! Not.

FYI, GCC does not generate 5 x 20 bytes of pushes and pops. In fact in the above specific case it will not generate a single push (automatically - you dont have to worry about it).

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

They would load the registers to the proper values and jump to it. The return address would be stored in the ESI register, and when the routie completed, it would do

jmp esi

to return, with 0 stack usage.

If the compiler were set to optimize.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

yep, exactly. It simply optimizes the wrong thing and restricts architectural flexibility. It is very easy to optimize by making a system more specific. (this is fact is a more or less automatic engineering work) The real optimizations are the ones that do not take away from the generic nature of the system.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

One more optimization it has. NetWare never "calls" functions in the kernel. There's a template of register assignments in between kernel modules that's very strict (esi contains a WTD head, edi has the target thread, etc.) and all function calls are jumps in a linear space. layout of all functions are 16 bytes aligned for speed, and all arguments in kernel are passed via registers. It's a level of optimization no C compiler does -- all of it was done by hand, and most function s in fast paths are layed out in 512 byte chunks to increases speed. Stack memory activity in the NetWare kernel is almost non-existent in almost all of the "fast paths"

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

-----BEGIN PGP SIGNED MESSAGE-----

Jeff, one other thing. Linux is not x86 hand-crafted assembler, it's capable of running on many platforms. are you planning on giving up this capability or hand crafting the kernel for each chip?

Linux on x86 is nice (and I do use it a lot) but one of the things that makes it so useful is that when you do outgrow what youcan do on a x86 platform you cna move to a more powerful platform without having to change to a different OS.

David Lang

-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.2

iQEVAwUBOf9LYj7msCGEppcbAQFdZQgAocWtMwnNmmLnfSS+cGHZd8V0IFfmoVb7 fDRoNWMmOzU5g5W8aAudQFqGpGqizBR8/AA9ziqHRfKxwoo5/nuHtEMDpfw0nV5e ghsd7qtzv1kTk0l5zp6bN2qPlGgs7Ke72od10X6pTGDyuUDQK71YNQ9UUcCv8GEO 2PpPOnCHw3atuQ0hetMNcFfIdvvslTB2+pcVzYxWWGhCYIeWreF8w1qf8XDYQil9 Ih22vmu69LP03RwXkFioikVSK8F8m+31DUBA67exN+R4qXy8+U5ZtyPQ+onIeguh SnADBNjGjWK2mPyLNSVyAwH6EsIaGzk1QJ5hYULzVFi4zVl3pyRi8w==
=91LL
-----END PGP SIGNATURE-----
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

How about a NetWare like NOS with all the application support of Linux? I think a lot of people will love it. Novell's largest customers have told me they want to see it, and would deploy it. I do believe there's a market for such a beast. A very lucrative one...

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I don't doubt it. a port of netware that can run linux apps would be very useful to people who want to run netware, but this is not the same thing as what it has sounded like you were working on.

David Lang

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

this might be a win on a i486, but is a loss with any branch predicting, large-pipeline CPUs (think Pentium IV), which are optimized for CALLs, not for JMP *EAX instructions. This is the problem with assembly optimizations that try to compete with the compiler's work: hand-made assembly can only get worse over time (stay constant in the best case), while compilers are known to improve slowly but steadily. Plus hand-made assembly is a huge stone tied to your legs if you try to swim to other architectures. Eg. we quite often make use of GCC's register-based function parameter passing optimization. We do use hand-made assembly in a number of cases in Linux as well, and double-check GCC's assembly output in critical code paths, but we try to not make it an essential facility.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

It's hand optimized for all these cases to one of the highest degrees
that exist anywhere in the world. Intel and Novell have been in bed
together for a very long time...

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

What if I jump to an invalid address - does it crash ?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

The jumps are all to defined labels in the code, so there's never one that's invalid. The only exception are the jump tables in the MSM and TSM lan modules, and yes it does crash is a bad driver gets loaded, but that's the same as Linux. NetWare avoids using jump tables since they cause an AGI to get generated, which will interlock the processor pipelines. If you load an address on Intel, then immediately attempt to use it, the processor will generate and Address Generation Interlock (AGI) that will interlock the piplines so the request can be serviced immediately. This cost 2 clocks per address jump used, plus it shuts down the paralellism in the piplines.

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Nice! Now run that (i386 optimized?) beast on a machine that works different (latest K7s perhaps?), and many optimizations break.

When you got that fixed, would you please port it to Alpha?

Sure, using C (with a not-overly-bright compiler) has a non-negligible cost. But huge benefits too. The whole of software (including OS) design is an excercise in delicate juggling among conflicting goals. Had Linus gone down the "all-assembler, bummed to its limits" route, Linux would have been dead by 0.03 or so, depending on the stubborness of its creator to be sure.

Horst von Brand                             [email protected]
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ah ha ha ha!!!! Sure they do! You're just quoting statistics
measured under whatever conditions you imposed.

Numbers lying? I think the famous line has been variously
attributed to either Mark Twain or Disraeli (don't know which really coined the phrase) but it's been said that there are three kinds of lies "Lies, Damn Lies, and Statistics". Yes numbers do lie. Sometimes it's the GIGO law and sometimes its just the fact that if you abuse statistics long enough they will tell you ANYTHING. Sometimes it merely the person manipulating^H^H^H^H^H^H^H^H^H^H^H^Hproviding the numbers.

BTW... I was going to stay OUT of this rat trap, but since I'm
in for a dime, I might as well be in for a dolar...

<Minor Rant>

Comparisons have been made between the performance of Linux with
early (3.x, 4.x) versions of Novell. ANYONE who wants to compare Linux with that bug ridden, unreliable swamp of headaches and security holes (somewhere in my archives I have the virus launcher that BYPASSES the 3.x login program) should be beaten about the head with a good textbook on reliable coding techniques. Novell made its hayday by beating the bejesus out of TCP/IP and others primarily by disabling checksumming, memory protection, and other reliability techniques. Yes, they got better performance on low performace processors, but at what cost? Now we cover their performance with reliability and superior hardware. I remember one misguided soul wanting to run IPX over SLIP pleading for help on the Novell mailing list years ago. Let's see... SLIP eliminates the MAC layer checksumming and IPX eliminates the error checking on the next layer up... Yup... There was a receipe for random acts of terrorism. Now we have PPP (this was in the days BEFORE PPP) and you could do it. IPX depended on the lower layers for data integrity and and SLIP depended on the layer above it. Ooooppppssss....

Then we had the Novel 5.x NFS server that allow you to create
scripts that were SUID to root just by making them read only to the Novell workstations (ok - that's not performace related - I just think that security should be given a LITTLE thought). I worked at an outfit (Syntrex) that saw themselves as becoming the "K-Mart of Novell" and I was told that Novell was the be all and end all of networking and there was really no future in this antiquated TCP/IP stuff. I was laid off and given all sorts of nice neat little toys like an AT&T source license because they saw no future in Unix or TCP/IP. (Bitter - no... I have had my revenge in spades... They had no clue what they gave away and let slip through their fingers! :-) )

Now, Novell has been dragged kicking and screaming into the TCP/IP
world, and Novell has been forced to add memory protection (at a performace cost) to their servers, and the outfit that thought TCP/IP was history is now history (Syntrex went Chapter 13 about 10 years ago), and I've had the pleasure of slamming one particularly simple minded Novell rep (another ex Syntrex inDUHvidual) with more than one security hole (the perl module on the Novell web server was an absolute classic).

My point here is that packets per second don't mean jack shit if
you can't do it reliably and you can't do it securely. Novell failed on both of those counts and those are a contributing factor in their current troubles. They built their reputation on performance that was achieved at the expense of reliability and security. Now they have to play with the big boys and all the nasty kiddies out there who don't play nice.

Performance is important. Performance is desirable. Efforts to
improve performance are worthwhile. But performance should NEVER come at the cost of security or reliability or integrity. Comparisions with high performance systems which lacked security, reliability, and data integrity are suspect AT BEST. We should NEVER give up the quest for better performance but comparisons to an inferior operating system which can pump out packets faster than us is not the threat some people would like it to be.

</Minor Rant>

My regards and respects to Jeff. He says he was responsible for the
Novell 4.x and 5.x systems. I note that he omitted the 3.x OS. Acknowledged and respected! In my earlier days, I was the kernel jock responsible for a proprietary version of XENIX and worked on Microport UNIX (may someone forever drive a stake through that bastard's heart) and SCO Unix. I'm "contaminated goods" for certain projects so I can't contribute to certain applications like Taylor UUCP because I have legal source code to HDB UUCP (as if UUCP means jack in today world). Does the current taylor UUCP have the most possible efficient checksumming algorithm for the UUCP 'g' protocol? No... No way... I've seen one that stomps Taylor UUCP's ass and takes names from the AT&T SVR5 release 3.2 source tapes (took me a week to figure out just how it worked - damn those comment strippers). Does it matter? Not one bit.

I want to see Linux excel. Does that mean that it has to beat
every benchmark set by an operating system that cut every corner that would cause Linus to turn into a Quake Balrog? I think not.

Mike

 Michael H. Warfield    |  (770) 985-6132   |  [email protected]
  (The Mad Wizard)      |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Presumeably you'd immediately do a ret to some address, and there pop a base address off the stack to get some global memory. Is that right? Your context switches would be inline, and you'd have hardcoded which process to execute next in most cases.

I'll buy the concept that changing stacks amounts to changing contexts, so long as you follow certain rules. Obviously, rules are what define a context. What are the two instructions that precede and the two instructions that follow? I'd guess, something like this:

push bp
push $1
mov x, esp
mov esp, y
ret
$1 pop bp

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Is that your two instruction context switch? The problem is, it doesn't transfer control anywhere. Maybe it doesn't need to. I guess you could break your tasks up into lots of little chunks and compile each chunk inline and use actual calls to take you off the fast path. The stack changes are actually doing some useful work here: you might for instance be processing a network packet whose address is on the stack. But somehow I don't think this is your two-instruction context switch. The only halfway flexible two-instruction context switch I can think of is:

mov esp, y
ret

where you already know the stack depth where you are so you don't have to store it, and the task execution order is predetermined. This switches the *two* essential ingredients of a context: control+data. But there's a big fat AGI there and all the overhead of a jump so it doesn't get your superscalar performance.

Now my stupid question: why on earth do you need a billion context switches a second?

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Pavel, The optimization exploits the multiple piplines in Intel's processors,
and yes, it does execute more than one instruction per clock, it's optimized
to execute in the processors parallel pipelines. The EMON numbers are accurate,
and you can download the kernel and verify for yourself. These types of optimizations
are possible when people have acccess to Intel Red Cover documents, then you
get to know just how Intel's internal architectures are affected by different coding optimizations.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

optimizing away AGIs has been documented by public Intel PDFs for years:

[...] Since the Pentium processor has two integer pipelines, a register used as the base or index component of an effective address calculation (in either pipe) causes an additional clock cycle if that register is the destination of either instruction from the immediately preceding clock cycle. This effect is known as Address Generation Interlock (AGI).

(ditto for the p6 core CPUs), and GCC (IIRC) tries to avoid AGI conflicts as much as possible. (and this kind of stuff belongs into the compiler)

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Odd. When I profile Linux with EMON, I see tons of them. Anywhere code does

mov eax, addr
mov [addr], ebx

one will get generated. Even something as simple as:

mov    eax, addr
inc    eax
dec    eax
mov    [addr]. ebx

will avoid an AGI (since the other pipeline has now been allowed a few clocks to fetch the address and load it).

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

AGIs were a real problem on P5 class Intel CPUs. On P6 core CPUs, most forms of addresses (except memory writes) do not generate any AGIs. And the AGI on P6 cores does not keep up the pipeline, unless you reuse the same address. (which would be stupid in most cases) I bet Crusoe's have no AGIs at all. Do you see the trend in CPU design?

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Probably You mean this :

mov r/imm, %eax
mov (%eax), %ebx

- Davide
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

"Jeff V. Merkey" wrote:

In the example of an AGI generating code fragment, while I described the sequence of creating the AGI with immediate address usage correctly (i.e. if you load an address into a register then immediately attempt to use it, it will generate an AGI), I failed to put the register in the coding example.

A couple of folks are testing the gcc compiler for AGI problems as a result of this post, and I am posting the corrected code for their tests.

This code fragment will generate an AGI condition:

mov eax, addr
mov [eax].offset, ebx

You can do it with any register combination, BTW, eax and abx are provided as examples. For those who are monitoring the code produced by gcc, this is the example to use to test generate an AGI correctly.

:-)

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I had already posted the correction.
It was clear that You had forgot something coz Your old code fragment did not generate AGI.

- Davide
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

typing too fast late at night. I described it correctly, I just forgot to add the register indirection.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

There's also another optimization in this kernel that allows it to achieve greater than
100% scaling per processor by using a strong affinity algorithm (I hold the patent
on this algorithimn, and by posting code based on it under the GPL, I have released
it to the general public).

It relies on an anomoly in the design of Intel's cache controllers, and with memory based
applications, I can get 120% scaling per procesoor by jugling the working set of
executable code cached accros each processor. There's sample code with this kernel
you can use to verify....

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

FYI, this is a very old concept and a scalability FAQ item. It's called "sublinear scaling", and SGI folks have already published articles about it 10 years ago.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

^-- extralinear. whatever.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ingo,

You don't even know what it is enough to comment intelligently. You can write the patent office and obtain a copy. The patent is currently in dispute between Novell and several other companies over S&E ownership, and there's a court hearing scheduled to resolve it (lukily I don't have to deal with this one). Nice thing about being an inventor, though, is I have rights to it, no matter who ends up with the S&E assignment. (dogs fights over a bone ...).

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

When you say it reloads it's VM, you mean it reloads the CR3 register? This will cost you about 15 clocks up front, and about 150 clocks over time as each TLB is reloaded (plus is will do LOCK# assertions invisibly underneath for PTE fetches) One major difference is that in NetWare, CR3 does not ever get changed in any network I/O fast paths, and none of the TCP or IPX processes exist in user space, so there's never this background overhead with page tables being loaded in and out. CR3 only gets mucked with when someone allocates memory and pages in an address frame (when memory is alloced and freed).

The user space activity is what's causing this, plus the copying. Is there an easy way to completely disable multiple PTE/PDE address spaces and just map the entire address space linear (like NetWare) with a compile option so I could do a true apples to apples comparison?

Jeff

It's more. I am seeing tons of segment register reloads, which are very heavy, and atomic reads of PTE entries.

I'll look at this.

Correct. You're right -- it's in the tcp code, however, I remember going over this when I was reviewing Alan's code -- that's what I was think of.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Yes.

No. In 2.4 you could probably use the on demand lazy vm mechanism ingo described for the nfsd processes. In 2.2 it is a bit more tricky, if I remember right lazy mm needed quite a few changes.

But before doing too many changes i would first verify if that is really the problem.

PTEs are read for aging on memory pressure in vmscan.

segment register reload happen on interrupt entry, to load the kernel cs/ds (are they that costly?). If it was really costly you could probably check in interrupt entry if you're already running in kernel space and skip it.

2.2 does not use checksum-copy-to-user in TCP RX, csum is either done separate or in hardware. It does csum-copy-fragment for TX, as well as UDP.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

We will never beat NetWare on scaling if this is the case, even in 2.4. Andre and my first job will be to create an arch port with MANOS that disables this and restructures the VM.

They are. segment register reloads will trigger the following:

IDT table atomic fetch to verify  (LOCK#) (if triggered by task gate from INTR)
GDT table atomic fetch to verify  (LOCK#)
LDT table atomic fetch to verify  (LOCK#) (if present)
PDE table atomic fetch to verify  (LOCK#)

The process has to verify that the loaded segment descriptor is valid, and it will fetch from all these tables to do it, with up to 4 (LOCK#) assertions occurring invisibly in the hardware underneath (which will generate 4 non-cacheable memory references, in addition to wrecking havoc on the affected L1/L2 cache lines). Oink. It only does this when you load one, not when you save one, like pushing it on the stack. Since you look at MANOS code, you'll note that in CONTEXT.386, I do and add esp, 3 * 4 instead of poppping the segment registers off the stack if they are in the kernel address space.

Linux should do the same, if possible as an optimization.

Yes, I know, this is what I was referrring to. Received are always ugly, there's always at least one copy into cache for the write, sometimes more...

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

In the 2.4 case if you are just running NFS daemons then there are no tlb reloads going on at all. Whats murdering you is mostly memory copies. I would suspect if you rerun the profiles on a box with a much lower memory bandwidth that the effect will be even more visible
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I don't think that's true. As far as I can see the nfsd processes do not do the lazy VM magic (but it would be reasonably easy to add)

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I just guess the end result will be as crash prone as Netware when you install any third party software ;)

lazy mm is probably a better path, as long as you stay in kernel threads and a single user mm it'll never switch VMs.

Interesting. You could do easily the same in Linux by changing (in 2.2) the SAVE_ALL macro in arch/i386/kernel/irq.h and doing the same in ret_from_intr's RESTORE_ALL macro (after changing it to a RESTORE_ALL_INT or you'll break system calls)

Actually I don't even know why irq.h's SAVE_ALL even loads __KERNEL_CS, it should be already set by the interrupt gate. __KERNEL_DS probably needs to be still loaded, but only when you came from user space.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

When I'm following this thread, you guys seem to forget the _basics_:
The Linux networking stack sucks!

Everybody tries to work around the networking stack. We just recently developped a rpc protocol which makes 180MBytes/second (over a Quadrics Network) because the linux network layer was way too slow. At speeds above 100MBytes/s, copies start to hurt.

Why not solve the problem at the source and completely redesign the network stack? Get rid of the old sk_buff & co! Rip the whole network layer out! Redesign it and give the user a possibility of Zero-Copy networking!

Reto
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

For one because you don't need to do that to get zero copy networking for most real world cases. Tux already implements the neccessary infrastructure for these.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

And what if I'd like to use the network for something different than html?

I'm sorry but I don't know TUX. Does it implement its own TCP stack?

Reto
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Read the tux source. Then come back and ask sensible questions

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

--------- Received message begins Here ---------

Also pay attention to the security aspects of a true "zero copy" TCP stack. It means that SOMETIMES a user buffer will recieve data that is destined for a different process.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

The moment you try and do zero copy like that you end up playing so many MMU games the copy is faster. We do zero copy from the kernel side of the universe thats a lot lot saner

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Jesse Pollard <[email protected]> said:

[...]

Why? AFAIKS, given proper handling of the issues involved, this can't happen (sure can get tricky, but can be done in principle. Or am I off-base?)

Horst von Brand                             [email protected]
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

As I understand the current implementation, this can't. One of the optimizations I had read about (for a linux test) used zero copy to/from user buffer as well as zero copy in the kernel. I believe the DMA went directly to the users memory.

This causes a problem when/if there is a context switch before the data is actually transferred to the proper location. The buffer isn't ready for use, but could be examined by the user application (hence the security problem).

It was posed that this is not a problem IF the cluster (and it was a beowulf cluster under discussion) is operated in a single user, dedicated mode. In which case, to examine the buffer would either be a bug in the program, or a debugger looking at a buffer directly.

To my knowlege, zero copy is only done to/from device and kernel. Userspace has to go through a buffer copy (one into user space; one output from user space) for all IP handling. All checksums are either done by the device, or done without copying the data.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]

Any opinions expressed are solely my own.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Right. This means you have to ensure (somehow blocking the process(es) with access to the buffer(s) involved) that nobody can see half-filled buffers. Tricky, but not impossible, at least not in principle. Or play VM games and switch the areas underneath atomically. The VM games we have been told are costlier than the average copy on "typical" machines (PCs, presumably ;-), plus you'd have to either ensure aligned buffers (how?) or keep two copies of whatever surrounds them (where is the advantage then?).

Horst von Brand                             [email protected]
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

The code in the networking layer is fine, in fact, it's absolutely great. This is not
the problem. The problem are all the clocks wasted reloading TLBs and
the background
memory activitiy caused by Linux's heavy dependence in Intel's protection hardware model.

Step 1 is to load the entire OS at ring 0 with protection completely disabled system wide,
put in the kernel optimizations for AGI and context switch locking, and stub the top half of Linus's scheduler. The global memory structures in his kernel may or may not hurt performance, depending on how they are accessed by multiple processors. I will need to start profiling with a ring 0 port first and determine frequency of access that's hardware measured.

The next step would be to start peeling off different subsystems and re-parallelizing them
on the merged kernel. There's an awful lot of areas in Linus' that are going to be a
problem, but I'll work through them one at a time. When I first parallelized NetWare, I wrote
an independent SMP kernel then loaded the entire NetWare 4.1 image as a single process. The next step was to start peeling off layers from NetWare and plugging them in one by one and parallelizing them. I am using the same approach here. When I was finished, I had peeled NetWare like a banana and completely reintegrated it on a new kernel. This approach works because it allows you to stage each layer.

The next step will be to modify the loader to allow the existing protection scheme to exist in true user space. WRD's will hold off CR3 switching so long as I/O is coming in or out of the box. I anticipate this will take until March of next year to get right.

Jeff

packets via a WTD scheduler,
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

One other point. You guys can try as hard as you can to hack around this problem, but it will never scale like NetWare unless a fundamentally different approach is adopted. I've built this before, and what we are doing is building in all the Linux functionality with the identical code -- basically a Linux NetWare -- into a NetWare framework. All the protection can still be there, it just needs to be restructured with some LAN based design assumptions. I think the industry will be pleased with the result. I do not believe any of the proposed hacks will get there unless this different approach is investigated.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ummm, last I looked Linux held the Specweb99 record; by a wide margin...

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

...does that remove any memory copies???

To be best does not mean that there's no place for improvment.

Can anybody please help me and tell me where to start understanding what tux does?

www.tux.org does not seem to be the right place :-(

I don't want to make linux bad or stand on anybodys toes. I'd just like to improve linux like everybody else and I do not know every single peace of source code floating around the world by heart ;-)

Reto
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Good to know, your previous message might have fooled some people when it comes to these intentions ;)

--------------

--------------

cheers,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

It doesn't hold the file and print scaling record. NetWare does..

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Indeed, we haven't made a file serving plugin for
the TUX zero-copy stuff yet...

Oh, and I haven't found a bunch of printers yet that are fast enough to beat the printserving record ;))
*runs like hell*

cheers,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Rik

Is there any documentation about the Tux zero-copy implementation so that I don't have to read half of the 2.4 kernel sources before having a clue?

Are the kernel changes going to be in the mainstream kernel?

Does Tux implement a new interface so that a userspace app can do zero-copy stuff with the network?

Sorry for the newbie questions, but it's really hard to find information about Tux (other than the holy source code, of course ;-)

TIA

Reto

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Reading the 2.4 sources won't do you much good since the Tux layer isn't integrated ;)

Dunno, ask Ingo and/or Linus...

Maybe there's some documentation included with the Tux source code or maybe there's something on the Tux web page? Have you tried looking there?

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

You got me on this one. I would also agree that NPDS is the worst piece of crap ever written.

:-)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/