What's left over.

Hi Linus,

Here is the list of features which have are being actively
pushed, not NAK'ed, and are not in 2.5.45. There are 13 of them, as appropriate for Halloween.

Most were submitted repeatedly *well* before the freeze. It'd
be nice for you to give feedback, and decide which ones (if any) are still up for review.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

From: http://www.kernel.org/pub/linux/kernel/people/rusty/2.6-not-in-yet/

Rusty's Remarkably Unreliable List of Pending 2.6 Features [aka. Rusty's Snowball List]

A: Author
M: lkml posting describing patch
D: Download URL
S: Size of patch, number of files altered (source/config), number of new files.
X: Impact summary (only parts of patch which alter existing source files, not config/make files)
T: Diffstat of whole patch
N: Random notes

In rough order of invasiveness (number of altered source files):

In-kernel Module Loader and Unified parameter support
A: Rusty Russell
D: http://www.kernel.org/pub/linux/kernel/people/rusty/patches/Module/
S: 841 kbytes, 302/36 files altered, 22 new
T: Diffstat
X: Summary patch (598k)
N: Requires new modutils

Fbdev Rewrite
A: James Simmons
M: http://www.uwsg.iu.edu/hypermail/linux/kernel/0111.3/1267.html
D: http://phoenix.infradead.org/~jsimmons/fbdev.diff.gz
S: 4852 kbytes, 168/29 files altered, 124 new
T: Diffstat
X: Summary patch (182k)

Linux Trace Toolkit (LTT)
A: Karim Yaghmour
M: http://www.uwsg.iu.edu/hypermail/linux/kernel/0204.1/0832.html
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103491640202541&w=2
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103423004321305&w=2
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103247532007850&w=2
D: http://opersys.com/ftp/pub/LTT/ExtraPatches/patch-ltt-linux-2.5.44-vanilla-021026-2.2.bz2
S: 257 kbytes, 67/4 files altered, 9 new
T: Diffstat
X: Summary patch (90k)

statfs64
A: Peter Chubb
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103490436228016&w=2
D: http://marc.theaimsgroup.com/?l=linux-kernel&m=103490436228016&w=2
S: 42 kbytes, 53/0 files altered, 1 new
T: Diffstat
X: Summary patch (32k)

ext2/ext3 ACLs and Extended Attributes
A: Ted Ts'o
M: http://lists.insecure.org/lists/linux-kernel/2002/Oct/6787.html
B: bk://extfs.bkbits.net/extfs-2.5-update
D: http://thunk.org/tytso/linux/extfs-2.5/
S: 497 kbytes, 96/34 files altered, 34 new
T: Diffstat
X: Summary patch (167k)

ucLinux Patch (MMU-less support)
A: Greg Ungerer
M: http://lwn.net/Articles/11016/
D: http://www.uclinux.org/pub/uClinux/uClinux-2.5.x/linux-2.5.44uc3.patch.gz
S: 2218 kbytes, 25/34 files altered, 429 new
T: Diffstat
X: Summary patch (40k)

Crash Dumping (LKCD)
A: Matt Robinson, LKCD team
M: http://lists.insecure.org/lists/linux-kernel/2002/Oct/8552.html
D: http://lkcd.sourceforge.net/download/latest/
S: 18479 kbytes, 18/10 files altered, 10 new
T: Diffstat
X: Summary patch (18k)

POSIX Timer API
A: George Anzinger
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103553654329827&w=2
D: http://unc.dl.sourceforge.net/sourceforge/high-res-timers/hrtimers-posix-2.5.44-1.0.patch
S: 66 kbytes, 18/2 files altered, 4 new
T: Diffstat
X: Summary patch (21k)

Hotplug CPU Removal Support
A: Rusty Russell
D: http://www.kernel.org/pub/linux/kernel/people/rusty/patches/Hotcpu/hotcpu-cpudown.patch.gz
S: 32 kbytes, 16/0 files altered, 0 new
T: Diffstat
X: Summary patch (29k)

Hires Timers
A: George Anzinger
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103557676007653&w=2
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103557677207693&w=2
M: http://marc.theaimsgroup.com/?l=linux-kernel&m=103558349714128&w=2
D: http://unc.dl.sourceforge.net/sourceforge/high-res-timers/hrtimers-core-2.5.44-1.0.patch http://unc.dl.sourceforge.net/sourceforge/high-res-timers/hrtimers-i386-2.5.44-1.0.patch http://unc.dl.sourceforge.net/sourceforge/high-res-timers/hrtimers-hrposix-2.5.44-1.1.patch
S: 132 kbytes, 15/4 files altered, 10 new
T: Diffstat
X: Summary patch (44k)
N: Requires POSIX Timer API patch

EVMS
A: EVMS Team
M: http://www.uwsg.iu.edu/hypermail/linux/kernel/0208.0/0109.html
D: http://evms.sourceforge.net/patches/2.5.44/
S: 1101 kbytes, 7/10 files altered, 44 new
T: Diffstat
X: Summary patch (4k)

initramfs
A: Al Viro
M: http://www.cs.helsinki.fi/linux/linux-kernel/2001-30/0110.html
D: ftp://ftp.math.psu.edu/pub/viro/N0-initramfs-C21
S: 16 kbytes, 5/1 files altered, 2 new
T: Diffstat
X: Summary patch (5k)

Kernel Probes
A: Vamsi Krishna S
M: lists.insecure.org/linux-kernel/2002/Aug/1299.html
D: http://www.kernel.org/pub/linux/kernel/people/rusty/patches/Misc/kprobes.patch.gz
S: 18 kbytes, 4/2 files altered, 4 new
T: Diffstat
X: Summary patch (5k)

I'm unlikely to be able to merge everything by tomorrow, so I will consider tomorrow a submission deadline to me, rather than a merge deadline. That said, I merged everything I'm sure I want to merge today, and the rest I simply haven't had time to look at very much.

This apparently breaks things like DRI, which I'm fairly unhappy about, since I think 3D is important.

This one is just huge, and I have little personal judgement on it.

I don't know what this buys us.

I haven't even seen it.

I don't know why people still want ACL's. There were noises about them for samba, but I'v enot heard anything since. Are vendors using this?

I've seen this, it looks pretty ok.

This is definitely a vendor-driven thing. I don't believe it has any relevance unless vendors actively support it.

I think I'll do at least the API, but there were some questions about the config options here, I think.

No objections, but very little visibility into it either.

This one is likely another "vendor push" thing.

Not for the feature freeze, there are some noises that imply that SuSE may push it in their kernels.

I want this.

Probably.

Linus

Because People Are Stupid(tm). Because it's cheaper to put "ACL support: yes" in the feature list under "Security" than to make sure than userland can cope with anything more complex than "Me Og. Og see directory. Directory Og's. Nobody change it". C.f. snake oil, P.T.Barnum and esp. LSM users

It's nearly useless in a Unix-only context, true, however there's a rather serious impedance mismatch for serving files to Windows that this addresses. Emulating ACLs on the fly with groups to fit into the Windows model is mostly doable but ain't pretty.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

It's only nearly useless if you have some desire as an admin to constantly be creating groups and changing group lists for users. This is not a feature which is useful only when serving files to Windows machines, not even nearly. AFS, Solaris, Irix etc have support for ACLs and have a great deal of people who use them. The simple yet common situation of one user who wants to give even just read access to another specific user for a given file is a pain in the ass to deal with given the current structure.

Stephen

Note that as far as ACL's go, enough people have convinced me that we want them, with clear real-life issues. So don't worry about them, I'll merge it.

Linus

Ok, so now lets work on a Documentation/filesystems patch pointing out a few of the common pitfalls, as I definitely agree they invite some grave mistakes and are best avoided in most scenarios.

- /tmp-style symlink issues on shared directories
- vast majority of software (including security tools) ACL-unaware
- much harder to check for correctness

Al, I'm sure you have more..

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

- surprising inheritance of of the ACL of the directory

This is a known problem in NTFS land, and some people suggest that per-directory ACLs are enough for everyone for exactly this reason.

Hi!

Okay... Have ~/bin/phonebook and I'd like it to be rw- to me, r-- to jarka and mj, and --- to everyone else. How do I do that without ACLs? Adding a group is root-only operation.

This seems like a pretty common situation to me, and current solutions are not nice. [I guess ~/bin/ with --x and ~/bin/my-secret-password-only-jarka-and-mj-knows/phonebook would solve the problem, but...!]
Pavel

When do you have heart between your knees?

Can't even this be spied from /proc/*/fd ?

Hi!

Not sure... Its true that if users are not carefull (i.e. do

cat ~/bin/my-secret-password-only-jarka-and-mj-knows/phonebook

it can be seen on ps -aux ;-).
Pavel

When do you have heart between your knees?

In article <1036103335.25512.40.camel@bip>,

Or ptrace, /proc/pid/mem, etc. If you can execute a binary, it has to be loaded into memory in a process running as you, so you can read it.

Mike.

In message <[email protected]> you wri te:

Yes, the patch stubs out inter_module_*, in favor of get_symbol() & put_symbol().

This breaks the three users: one in drivers/mtd/ and two in drivers/char/drm/. I have a patch which fixes them (untested), or I can simply put the inter_module_* code back in.

It's been around for a while. Geert, Russell?

Haven't looked at it.

It's fairly old, but Peter Chubb said there was some vendor interest for v. large devices. Peter?

SAMBA needs them, which is why serious Samba boxes use XFS. Tridge, Ted?

The controls are in driverfs etc, and that's always been in flux. 8(

The rest is v. small, basically extending ksoftirqd, workqueues and migration threads to disable them. Then it's all arch-specific.

They have, IIRC. Interestingly, it was less invasive (existing source touched) than the LVM2/DM patch you merged.

Good. The big payoff is moving stuff out of the kernel, which can't really be done in a stable series.

Sent.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

oh yes, all the Linux based storage appliances use ACLs. Posix ACLs aren't ideal for Samba, but they are *much* better than having no ACLs at all. The Posix ACL code has been in Samba for a long time (getting close to 3 years now?).

Eventually I'd like to see a combination of LSM with a new ACL system give the ability to support full NT ACLs on Linux (which is also needed for full nfsv4 support), but that is way too much to do for the 2.6 kernel.

For the majority of windows users the mapping Samba does internally between Posix ACLs and NT ACLs is sufficient for now.

I think that it would be a very good thing for Posix ACLs to be included in the 2.6 kernel, especially in ext3.

Extended attributes are also important as they give a place to store all the extra DOS info that has no other logical place in a posix filesystem. For example, we can put the 'read only', 'archive', 'hidden' and 'system' attributes there. If we don't have extended attributes then we need to use a nasty kludge where these map to various unix permission bits, but the mapping is terrible and doesn't give the correct semantics (especially for things like read only on directories).

My main concern with using extended attributes in this way is performance. My experience with XFS is that as soon as you start adding extended attributes then the performance drops a lot, but I haven't tested performance with the ext3 extended attributes so maybe they don't have the same problem.

Cheers, Tridge

--
http://samba.org/~tridge/

Add bloat to make windows clients happy?

More bloat that does really solve Linux problems... sounds like nasty hacks to make winduhs hacks work better.

Don't get me wrong, I'm not against sane ACLs (POSIX ACLs are not) os EAs, but justification of "it makes windows clients easier" is pretty horrendous IMO.

I'd would at some point like to see decent ACLs, but I don't want to see 'windows ACLs' and all the SID nonsense.

--cw

POSIX ACLs are more complicated than what would be inherently necessary, if we were in a situation where we could design from scratch. Unfortunately we are not in that situation. I've heard dozens of people complain about POSIX ACLs (and other kinds as well); nobody was able to come up with something truly better so far.

--Andreas.

XFS doesn't have ACLs either in plain 2.5.

But that only because dm added stuff to the generic code where we told it. It's a lot more code than dm and it adds new discovery code at the same time we start moving stuff _out_ of the kernel to initramfs.

If you can SuSE has merged it any IBM patch posted here should get in, coming from big blue seems to be a basic merge criteria in Nuernberg :)

The existing NAS boxes that use Linux and XFS tend to base their kernels on the 2.4-xfs tree from cvs on sgi.com. It works well and the SGI guys have been very good about fixing problems when they crop up.

I think that the biggest beneficiary of adding extended attributes and ACLs into ext3 for 2.6 would be more casual users (home, small office etc) as they will then be able to use ACLs in Samba without the pain of switching to a different kernel.

Cheers, Tridge

--
http://samba.org/~tridge/

FUD. I added to three areas of existing code:

i) Every man and his dog uses mempools in conjuction with slabs, so rather than having everyone redefining their own alloc/free functions I added the following huge functions to mempool.c. In no way were they mandatory.

    /*
     * A commonly used alloc and free fn.
     */
    void *mempool_alloc_slab(int gfp_mask, void *pool_data)
    {
            kmem_cache_t *mem = (kmem_cache_t *) pool_data;
            return kmem_cache_alloc(mem, gfp_mask);
    }

    void mempool_free_slab(void *element, void *pool_data)
    {
            kmem_cache_t *mem = (kmem_cache_t *) pool_data;
            kmem_cache_free(mem, element);
    }

ii) vcalloc, this *didn't* get merged, and will probably end up getting
moved into dm.h.

iii) ioctl32 support: people have argued against an ioctl interface,
and I'm inclined to agree with them, which is why I'm going to publish an fs interface shortly. However, given that we are currently using an ioctl interface how do we avoid adding support for 32bit userland/64 kernel space ? If EVMS isn't touching these files does that mean they're not supporting these architectures ?

arch/mips64/kernel/ioctl32.c
arch/ppc64/kernel/ioctl32.c
arch/s390x/kernel/ioctl32.c
arch/sparc64/kernel/ioctl32.c

So given that (ii) didn't get merged, which of (i) and (iii) were you objecting to ?

- Joe

Yeah, historically we have avoided things like this.

kcalloc gets proposed every year or so too.

Well, I'll note that ALSA compartmentalizes their ioctl32 handling within their own subsystem, which seems like a decent solution.

That said, [maybe I'm biased <g>], using an fs interface allows one to completely eliminate an ioctl32 interface. That would be the direction I would greatly prefer by the time 2.5.x hits the code freeze.

Best regards, and congrats for getting it merged,

Jeff

I would like to see both of these in because tons of kernel fixing that has been done through audits has been about

	get_user(a, ...)
	kmalloc(a * sizeof(b), ..)

We end up with loads of ugly > MAXINT/sizeof(foo) if checks in the code that ought to be in one place

In message <[email protected]> you write:

[ 40-line detailed explanation snipped ]

Woah! War's over dude! We won!

I used Rusty's Unreliable Intrusiveness-o-meter (number of existing non-config files touched), as I said.

I didn't read code or anything so unscientific or accurate. But both DM and EVMS were way down on the "intrusiveness" list.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

:)

Sorry, it wasn't meant to be an agressive email. However comments like this do get picked up out of context and passed around until they become the accepted truth. I'm still trying to work out where 'dm can't handle mirroring or raid' rumour came from.

- Joe

It's huge because it moves a lot of files around:
1. drivers/char/agp/ -> drivers/video/agp/
2. drivers/char/drm/ -> drivers/video/drm/
3. console related files in drivers/video/ -> drivers/video/console/

(1) and (2) should be reverted, but apparently they aren't reverted in the patch at http://phoenix.infradead.org/~jsimmons/fbdev.diff.gz yet. The patch also seems to remove some drivers. Haven't checked the bk repo yet.

James, can you please fix that (and the .Config files)?

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Done. I have a new version of that patch at the same place. It is against 2.5.45.

http://phoenix.infradead.org/~jsimmons/fbdev.diff.gz

Its still pretty big. We can save the moving of the agp code for post halloween.

Yes, people use it. Not quite sure why though, I guess ACLs buy some flexibility over the user/group/other model but if the "unlimited groups" patch goes in (is in?) I'm happy ;)

Personally I do think either the unlimited groups patch or ACLs are needed in order to sanely run a large anoncvs setup.

regards,

Rik

Bravely reimplemented by the knights who say "NIH".
http://www.surriel.com/		http://distro.conectiva.com/
Current spamtrap:  <a href=mailto:"[email protected]">[email protected]</a>

The feeling I got on this was the ability to let users define their own groups. Perhaps I'm not following it closely enough but that was the impression I got in terms of "what this does for us"; I'm probably missing other things. Just that ability would be nice in my view though. Isn't it something that's been in AFS for a long time too? I've got a few friends who've played with AFS before (at CMU and the like) and really enjoyed the ACLs there.

Just my thoughts,

Stephen

Yea, I haven't looked at the submitted implementation, but at CMU ACLs were critical to be able to selectively share data between a very large set of users w/o bugging an administrator. Given multiple classes per semester which had multiple group projects, where you may have different groups for each project, I have no clue how anyone would be able to handle the (unix)group management required. ACLs let the users themselves manage what people got what access to their data.

How else can I fix my partner's bugs (or vice-versa), give the clumsy TA read only access, and let the cheat across the hall figure it out for himself? (There may very well be a good solution to this w/o ACLs but I've not seen it in use.)

So yea, I'd love to see a common ACLs API.
-john

[ Cc: trimmed ]

Note that POSIX ACLs don't seem to solve this either: they only let you control access in terms of existing users or groups.

IMHO, this is one of the standard pitfalls of ACLs: if they don't let you aggregate information, you quickly end up with huge ACLs hanging off every file, and each of those ACLs wants to be carefully maintained. I've seen a lot of this in my VMS days. (Unix is a bit better, because you can control access at a directory level, while VMS needs the ACL on each file, because you can open files directly by VMS' equivalent to an inode number, without traversing the directory hierarchy. Of course, many users didn't know that :-)

To make ACLs truly scalable, it would be nice to be able to express permissions in terms of access to other filesystem objects. E.g. "everybody who can read file ~me/acls/my_friends can write the directory on which this ACE hangs". This should work like a symlink, i.e. if I add new friends to my_friends, I don't have to update all my ACLs.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

I've never looked at the POSIX ACL spec, so forgive my ignorance.

While it would be nice to have user-definable ACL groups ("my friends" or "History 255 TAs") in addition to existing users and groups, I still don't find this to be critical. Sure, it adds (possibly quite a bit of) extra data to every file, but it gives you the granularity you need for the situation I described. It seems like user-definable ACL groups would be a nice extra feature on top of existing users or groups, but not a necessity.

Ugh, that seems dangerous. Too many forgotten ACL links and then I could accidentally give a vague acquaintance access to all my data meant for close friends.

Regardless, while ACLs do result in extra data per file being used, it is my understanding that ACLs allow you to solve problems that currently aren't solvable w/o administrator intervention. In my experience using them w/ AFS, they have been extremely useful.

-john

The idea is that you'd typically have (a) (small number of) specific location(s) where you keep your files representing groups, e.g. $HOME/acls/ for your personal lists, maybe ~project/acls/ for projects, etc.

If you think already this is dangerous, then you should be terrified by regular, non-aggregateable ACLs ;-)

I'm not saying that ACLs aren't useful, only that the lack of aggregateability makes them hard to maintain, so that people frequently fall back to setup scripts that simple re-create their ACL configuration. Once you're at this point, ACLs have lost much of their usefulness, and you might as well use some suid program that creates groups for you.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Oh! Ok, that's exactly like the user-definable ACL groups I was describing. My mistake, I thought you were suggesting some crazy ACL symlink like: "Make file foo's ACL be the same as file blah's ACL" and if I then go and add some untrusted user to blah's ACL it would then automatically change foo's ACL. That just seemed a bit out there, but it was just my mis-interpretation. Sorry :)

Eh, as long as the ACLs are per-file, I can't ever accidentally give access to a file I didn't mean to. The corner cases of "remove my ex-friend from all my files" could be annoying, but could be done w/ the equiv of chgrp -r

Hmmm. I'm way out of my realm of competency here. I just know ACLs were *really* useful w/ AFS.

I probably should just go read the specs. Anyone have a pointer, or care to explain what the differences are between AFS's ACLs and POSIX ACLs?

thanks
-john

Well, with "foo" getting the ACL from "bar", changing the ACL of "bar" would change "foo", but not vice versa. Of course, the idea is that you're careful when changing "bar", just like you'd be careful with your SSH keys.

chgrp -r gets nasty if you have files which are stored off-line. On the other hand, using the concept that ACEs add rights, but never take them away, even an off-line "ACL link target" would fail on the safe side, by not adding more rights.

I've forgotten most things I knew about AFS ACLs (I used them at IBM about eight years ago), but http://acl.bestbits.at/ and in particular http://acl.bestbits.at/cgi-man/acl.5 seem to have everything about POSIX ACLs. They're not very complicated.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

POSIX 1003.1e draft 17 (withdrawn) is available at
<http://wt.xpilot.org/publications/posix.1e/>.

--Andreas.

Processes need to be a member of 20+ groups to make anoncvs work? Sounds like anoncvs is broken then.

--cw

Technically speaking you can achieve ACL like permissions/behavior using the historical UNIX security model by creating a group EACH time you run into a unique case permission scenario.

Without ACLs, if Sally, Joe and Bill need rw access to a file/dir, just create another group with just those three people in. Over time, of course, this leads to massive group proliferation. Without Tim Hockin's patch, 32 groups is maximum number of groups a user can be a member of.

Dax

I'm not arguing against this... I'm claiming POSIX ACLs are mostly brain-dead and almost worthless (broken by committee pressure and too many people making stupid concessions).

If we must have ACLs, why not do it right?

How many people actually need this level of complexity?

Why are we adding all this shit and bloat because of perceived problems most people don't have? What next, some kind of misdesigned in-kernel CryptoAPI?

--cw

Ok, I'll allow myself to be trolled.

What's wrong with our current 2.5.45 crypto api?

It's synchronous and assume everything is synchronous. Lots of hardware (most) doesn't work that way.

--cw

Think of it another way: many users will likely _require_ atomic encryption / decryption (done in softirq contexts etc), and thus a synchronous interface. Also, it simplifies the code and makes it more efficient.

Any hardware that needs to go off and think about how to encrypt something sounds like it's so slow as to be unusable. I suspect that anything that is over the PCI bus is already so slow (even if it adds no extra cycles of its own) that you're better off using the CPU for the encryption rather than some external hardware.

In short, from what I can tell, there is no huge actual reason to ever allow a asynchronous interface. Such interfaces are likely fine for things like network cards that can do encryption on their own on outgoing or incoming packets, but that is not a general-purpose encryption engine, and would not merit being part of an encryption library anyway.

[ Such a card is just a way to _avoid_ using the encryption library - the
same way we can avoid using the checksumming stuff for network cards that can do their own checksums ]

We'll see. I'd rather have a simpler interface that works for all relevant cases today, and then if external crypto chips end up being common and sufficiently efficient, we can always re-consider. Are the DMA-over-PCI roundtrip (and resulting cache invalidations) overheads really worth the extra hardware?

Linus

Except almost all hardware out there that does this stuff is async to some extent...

I'm just speaking as someone who has (sadly) done this a couple of times already for commercial real-world products. I'm no expert, I don't claim to be and admit there is still plenty to learn...

... that said, having access to lots of hardware, both our own and other peoples, almost all of it needs to be driven asynchronously to get good performance (or by a large number of threads).

In article <[email protected]>,

That's not my argument. I realize that external hardware on a PCI bus _has_ to be asynchronous, simply because it is so slow.

The question I have is whether such external hardware is even worth it any more for any standard crypto work. With a regular PCI bus fundamentally limiting throughput to something like a maximum of 66MB/s (copy-in and copy-out, and that's so theoretical that it's not even funny - I'd be surprised if RL throughput copying back and forth over a PCI bus is more than 25-30MB/s), I suspect that you can do most crypto faster on the CPU directly these days.

Maybe not. The only numbers I have is the slowness of PCI.

Linus

I'd be amazed of current CPUs would be able to do asymmetric encryption at anywhere within an order of magnitude of those rates.

Symmetric encryption is something else. This is the reason many encryption products (ie, pgp) only use asymmetric encryption for encrypting a symmetric session key, and not encrypting the entire message.

Regards,

bert hubert

http://www.PowerDNS.com          Versatile DNS Software & Services
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

This may be true of a typical workstation or large server, but your router may not have such a modern CPU in it. Crypto accelerators are likely a much bigger win on embedded routers or other small appliances with CPUs such as the AMD Elan or other 486 to Pentium class processors.

-- Gerald

Yes, and as a tangent, the same class of embedded devices also benefit from TCP/IP offload facilities. The same argument against a crypto-api supporting crypto hardware has been used in the past to argue against a Linux kernel TCP/IP hardware offload layer. The argument is completely invalid once one considers the typically lower speed of an embedded processor going into a crypto or network-edge device.

Even better, synthesizable SoC designs like IBM PPC4xx and reconfigurable processors architectures have opened further the concept of an on-chip crypto or tcp/ip offload macro cell which virtually eliminates PCI speed/latency concerns for these assist engines. It should be no surprise that embedded Linux is highly desired in these application specific processors.

Regards,

Matt Porter
[email protected]
This is Linux Country. On a quiet night, you can hear Windows reboot.

A 1GHz PIII will do about 8MBytes/sec of 3DES

Plug in a 2.4Gbs broadcom crypto chip into a 64bit PCI-X slot with the same CPU and you should be capable of doing at least 10 times that.

Stuff like RSA is much slower (and benefits more from hardware)

BTW - there are some outdated cryptolib patches with an async interface around somewhere (along with patches for freeswan to use the async api).

I guess the crypto guys like Chris will add the async API if they need it (which they do i think ;).

~mc

It may be faster on your beefy 8 CPU boxes. But many people are creating, for example, little wireless access points with 200 Mhz StrongArm CPUs and similar little devices that lack the major CPU horsepower of big-iron system. Such boxes would be far better off offloading crypto to a little crypto chip, right?

-Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Linus Torvalds <[email protected]> :
[...]

Issue 'openssl speed' and wait for more numbers.

Short lived hybrid sessions kill (not that this or any of the current reasons for asynchronous crypto really matters imho).

Instant benchmark:

                  sign    verify    sign/s verify/s
rsa 1024 bits   0.0148s   0.0008s     67.7   1198.6 (PIV 2GHz)
                  sign    verify    sign/s verify/s
rsa 1024 bits   0.0478s   0.0026s     20.9    381.6 (PII 350MHz)

The 'numbers' are in 1000s of bytes per second processed.

type              8 bytes  64 bytes  256 bytes  1024 bytes  8192 bytes
des ede3         3930.00k  4027.43k   4032.30k    4002.19k    3973.12k (PIV)
type              8 bytes  64 bytes  256 bytes  1024 bytes  8192 bytes
des ede3         1058.51k  1061.25k   1090.70k    1097.44k    1091.36k (PII)

blowfish is ~10x faster btw.

--
Ueimor

Get over it! If you haven't noticed, CryptoAPI is merged already. The only bloat ACLs cause is the size of the source tarball. If your connection is slow or you are out of diskspace, too bad! I'm sure I'm not the only one who is tired of hearing people whine about "bloat" wrt the sources and demanding that features they don't use be ignored. No one (non-core) feature will be useful to everyone, that is a given fact. The point is that while you see no use for it, there are many others out there who do. ACLs are something which have existed in the Solaris/BSD world for a long time now, and people who have admin these boxen find ACLs to be quite useful.

Cheers,
Nicholas

One look at the From:
understanding has blossomed
.procmailrc grows

Your point is?

Chris is write that crypto api is misdesigned if we want to use hardware cryptocards

Alan,

Thanks for setting me straight, your assertion is correct, of course. I was under the impression that the CryptoAPI code was merged initially for IPSEC support and would be revamped and expanded at a later date to support a wide variety of interfaces?

Cheers,
Nicholas

I'll reserve judgement until we actually get access to some decent [made in the past few years] hardware crypto cards, and take a hard look at their PCI bus utilization... until then it is mostly vague handwaving...

[vendors - any takers?]

Hardware support was not an initial goal, as the requirements are not yet fully known.

An asynchronous scheduling interface is in planning but not yet implemented, as we need to further analyze the requirements of all of the possible hardware scenarios (e.g. IPsec NIC offload).

Hardware accelerators are generally a known issue, with already proven solutions (e.g. the OpenBSD crypto queue). We don't know much about IPSec NIC offload yet, however.

- James

James Morris
<[email protected]>

If Sally, Joe and Bill need rw access to a directory, and Joe and Bill are using existing userland (any OS I'd seen), then Sally can easily fuck them into the next month and not in a good way.

_That_ is the real problem. Until that is solved (i.e. until all userland is written up to the standards allegedly followed in writing suid-root programs wrt hostile filesystem modifications) NO mechanism will help you. ACLs, huge groups, whatever - setups with that sort of access allowed are NOT SUSTAINABLE with the current userland(s).

I think the normal intent is to let Sally, Joe, and Bill have their own private directory protected from THE REST OF THE USERS.

If a member of your trusted circle goes rogue, then, yup you are screwed for the moment. It shouldn't last a whole month though.

That is what backups, and employment termination is for.

Dax

Then give them all the same account and be done with that. Effect will be the same.

Unless I'm missing something, that only works if all the users need *exactly* the same permissions to all files, which isn't a good assumption.

Example: Sally is an accountant, Joe and Bill are engineers.

Bill and Joe are working on a project, and Sally is cost control for that project - they all need access to the project files. Bill and Joe do not need access to officer salary data, but Sally does. Bill and Joe need access to other projects (not necessarily the same ones), but Sally doesn't. Oops.

- Steve

That's the point. In practice shared writable access to a directory can be easily elevated to full control of each others' accounts, since most of userland code is written in implicit assumption that nothing bad happens with directory structure under it. And there is nothing kernel can do about that - attacker does action you had explicitly allowed and your program goes bonkers since it can't cope with that. Mechanism used to allow that action doesn't enter the picture - be it ACLs, groups or something else.

So you're not really arguing against ACLs, you're complaining that userspace is broken when there's shared write access. That's fine, userspace should be fixed, inclusion of ACLs into the kernel shouldn't be denied because of this. ACLs should be optional, of course, and if you want them some really noisy warnings about the problems of shared writeable area with current userspace tools. Of course, that same warning should probably be included in 'groupadd'.

Stephen

No. I'm saying that ACLs do not have a point until at least basic
userland gets ready for setups people want ACLs for. Adding features that can't be used until $BIG_WORK is done is idiocy in the best case and danger in the worst. Especially since $BIG_WORK does not depend on these features.

I think samba alone counts as enough user-land usage.

And if it turns out nobody else ever wants to use them, that's fine too.

Linus

         ^^^^^^

While that may be true in theory, in practice it's not necessarily the case. Many people don't have the expertise to make use of such exploits. And before you say that they can download a pre-cooked exploit kit, let me tell you that there are plenty of people who don't have the time or inclination to do that.

I've seen you talk about these kinds of things before, and you always seem to be talking about the typical nightmarish undergrad CS lab where the kids spend all their time trying to crack each other and the system. And I'm not saying that these don't exist: I've seen it.

But there are other environments (say a research lab with grad students, post-docs and faculty) where the inhabitants either don't have the skills or don't have the interest in cracking accounts. Everyone is too busy doing their own research. Cracking the mysteries of the universe seems to be more interesting.

So group write access and ACL's *can* lead to wanton cracking, but for many environments it's not an issue. For many, the dangers lie outside the firewall, not inside.

Note that I'm not specifically advocating ACL's, I'm just letting you know that the problem you're concerned about is, for good reason, not a problem for everyone.

I will note that one appealing aspect of ACL's is that they do not require administrator intervention. That's good for a user who just wants to set something up without having to wait for the sysadmin. It's also good for the sysadmin (excepting control freaks) who doesn't want to do things that the users can (or should) actually be doing by themselves.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

Then why do we need 'non-repudiation' w/r/t certificates? Isn't
the
idea to provide a way to isolate "bugs" in the "security" system. If something
is written to a file by the group signon, who wrote it?

we dont

--cw

Hi!

Do you mean symlink attack?

So userland needs to be improved. It already needs that modifications because of /tmp. Is there any new issue there?
Pavel

When do you have heart between your knees?

Correct me if I'm wrong but I believe a process has to be restarted to have its group membership list changed?

That's a huge difference from ACL behavior which allow for changes to file access rights without the need to restart the accessing process.

	Leszek.

-- [email protected] 2:480/33.7          -- REAL programmers use INTEGERS --
-- speaking just for myself...

How about being able to:
- Debug synchronization problems among processes (there is no other tool to do this, not gdb, not strace, not printf, ...)
- Measure exact time spent wainting for kernel and which other processes a process had to wait for.
- Measure exact time it takes for an interrupt's effects to propagate throughout the entire system.
- Understand the exact behavior the system has to input. (what is the exact sequence of processes that run when I press a key).
- Identify sporadic problems in very saturated systems. (thousands of servers and one of them is doing weird stuff).
- etc.

Providing system tracing is a necessity for any sort of complex application development and system monitoring. Some people simply can't use Linux without this sort of tool and I am at pains to explain to them why they actually have to patch their kernel to be able to debug their inter-process synchronization problems.

Users don't have to patch their kernel to use gdb and I don't see why they should need to patch their kernel to understand how their various processes interact with the kernel and vice-versa.

Karim

===================================================
Karim Yaghmour
[email protected]
Embedded and Real-Time Linux Expert
===================================================

Linus,

LTT is one step in allowing Linux to continue to move towards being a
viable alternative for more than just hackers. It is part of a larger effort to provide reliability and serviceability. Concretely it allows application/subsystem programmers to understand the performance of their applications and the system. I should note, it also allows people to improve kernel behavior as well. As we have communicated in the past, the ability to gather and analyze this data is vital. From my correspondences with Ingo

"If you care about performance you will want to trace. On two previous kernels I have worked on I've heard this comment ["we don't need tracing"]. Once the infrastructure was in it was used and appreciated." There were world-class programmers involved in these projects that did not see the value of such infrastructure until they were able to use it.

I think Karim provided a list of possible uses, there are countless applications of this - I'll list some more:
seeing where unexplained idle tie is occurring understanding where interrupt processing time is going understanding interactions between applications - which is running when etc etc etc

If you look around the kernel, subsystems, and applications, you will find growing numbers of one-off-ways of gathering this information. Providing a unified way for different developers to communicate about performance will significantly improve the ability to performance debug different applications, drivers, system/application interaction, etc.

LTT has existed for a long time now and recent additions have been well
motivated: For a while now I have been working with the RAS team at IBM and with Karim Yaghmour to streamline LTT and make it perform well on MPs. We have addressed all the concerns raised by yourself, Ingo, and others from previous postings. If there remains concern, it is also possible for one to disable tracing. Some of the features we put into LTT came from ideas we prototyped in K42 (www.research.ibm.com/K42) which in turn was developed based on my experience writing a tracing infrastructure for IRIX while working for SGI, and other's experiences with AIX's tracing facilities.

LTT is a valuable aspect in allowing developers using Linux to understand their application's and the system's behavior. It serves to strengthen Linux's RAS capabilities and would be great to get included into 2.5. Thanks.

Thank you.

Robert Wisniewski
The K42 MP OS Project
Advanced Operating Systems
Scalable Parallel Systems
IBM T.J. Watson Research Center
914-945-3181
http://www.research.ibm.com/K42/
[email protected]

I don't mean to pick on LTT, I haven't used it, it may be the best thing since sliced bread.

I can tell you how to present this and any other feature similar to this in a way which would make me a lot more willing to accept it, which presupposes I'm doing Linus' job which of course I am not. However, it's likely that Linus has similar views but he gets to chime in and speak for himself.

All of these tools/features/whatever add some cost. The cost can be measured in lots of different ways:

- lines of code
- lines of code which can't be configed out
- call depth increases
- stack size increases
- cache foot print increases
- parallelism (think preempt)
- interface changes

I suspect there are other metrics and it would be very cool if others would chime in with their pet peeves.

What would be cool is if there was some way to quantify as much as possible of the accepted set of costs so that that could be balanced against the value of the change, right?

The one that always gets me is

"I've added feature XYZ, I benchmarked it with <whatever, usually LMbench> and it didn't make a difference"

That is almost certainly misleading. The real thing you want to do is quantify the actual costs because there can be non-zero costs that do not show up in benchmarks. For example, suppose that the benchmark neatly fits in the onchip caches and it only uses 1/2 of those caches. Your change could increase the cache foot print to just fill the caches, the benchmark says no difference, you declare success and move on. The problem is that almost all changes are good enough that they match this description. Measuring them in isolation doesn't tell us enough. If I combine two changes, both of which use up 1/2 the cache, there is no longer any room for anything else in the cache.

I'd love to see a trend where patch requests for any non-trivial patch included before/after data for the above metrics (and any others that people see as useful). I'd love to see some people taking just one of the above and making a tool which measures that metric. Then we combine the tools into a "patch measurement suite" and start prefixing patches with

    Code changes:
	+1234 -5678 = -4444	(all code)
	+123 -567 = -444	(all code subject to CONFIG_XYZ)

    Call depth:
	+2 for read()
	+2 for write()
	no change for all other system calls

    Stack size:
	+2099 bytes for read()/write() path

    Cache misses:
	No change for benchmark1, 2, 3
	12,000 data read misses for lat_ctx ....
    
    Etc.

What does the list think of this?

---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

An excellent engineering practice but extremely difficult to do. This is the holy-grail of software design and I don't think it would work for an extremely loosely connected set of developers.

There is no central control of the system (or chain of accountability) and that knocks down the practicality of this plan. It would work extremely well in another project, though.

} What does the list think of this?

...

Larry,
You're right - whoever wrote that useless LMbench anyway :-)

I agree it would be great to have have a tool that allows us to gather information on some of what you suggest below - but it's hard - people in software engineering have been working on such things for a long time. Further, what you mention below does not make sense in isolation. For example a package could add 1000 lines of code and have almost no impact, while another 10 lines of code could make a huge difference. So while the below metrics are fine, without arguing about the expected impact they're not necessarily helpful.

That's why benchmarks are still helpful as they are indicative of what expected performance might be. If you're trying to get at maintainability then I might (being a K42 convert) argue for a different strategy altogether.

So what about LTT then. Well sure enough we did run LMbench as some other tests. We ran a kernel compile, a tar, and LMbench - and posted results to lkml. While this hardly represents all possibilities, showing little performance impact on these is a positive statement about impact on other applications.

To address some of the list below:
lines of code: a lot - almost all can be configed out,
call depth increase: we can analyze - complicated since while it is a
couple levels - other calls in the code may be to
cache footprint: how? - simulate? this is tough - qualitatively I think for
ltt is small because the same code is used across all trace events. And less frequent trace events won't interfere
parallelism: not quite sure what you mean here - we not have a non-blocking
lockless scheme to address what I think the concern is here
interface changes: I argue very very positive - as in my letter to Linus
getting various developers to talk about performance with a common mechanism would be a big win

I'm sure this doesn't fully address your concerns - but if others feel some of the below numbers are really important we can certainly go about getting more accurate results then my above off-the-cuff info.

Robert Wisniewski
The K42 MP OS Project
Advanced Operating Systems
Scalable Parallel Systems
IBM T.J. Watson Research Center
914-945-3181
http://www.research.ibm.com/K42/
[email protected]

----

Hello Larry,

First, thanks for your feedback.

I understand and share you concern about the use of micro-benchmarks to qualify/quantify the impact additional code on the kernel. This is precisely the reason why I chose not to use micro-benchmarks in the Usenix article I presented about LTT at the 2000 annual technical conference. I was suprised to see some of the selection commitee members actually come up to me and say: "I'm so glad to see a paper that doesn't use micro-benchmarks."

That's why we elected to create 2 separate sets of benchmarks, one using real-life applications (kernel build, bzip2, etc.) and one using LMbench. Personnally, I would have been satisfied with just the real-life applications, but I know that many folks on the LKML want to see LMbench numbers, so we included those too. That said, I find it very positive that you keep a healthy dose of self-criticism towards your own tool, this is exactly the kind of stuff that makes LMbench so good. So too is it with LTT. I've always been on the lookout for reducing costs here and there while acheiving maximal functionality.

Fortunately, repeated testing and analysis on LTT by many parties using many tools have confirmed that the current LTT has very low impact on many fronts, including static code modifications.

So, for example, we had one example run of LMbench where we ran kernel compiles in the background (i.e. a script restarted the kernel compile every time it ended). To make it as simple as possible, here's the elapsed time taken to run LMbench on 4x SMP system in the various configurations:
---------------------------------------------------------------------

vanilla                         14:27
vanilla+ltt+ltt off             14:26
vanilla+ltt+ltt on              14:31
vanilla+ltt+ltt on+daemon on    14:32

vanilla+ltt+ltt on+kernel compile 15:03
vanilla+ltt+ltt on+kernel compiles+daemon on 15:13
---------------------------------------------------------------------

As you can see, the differences in percentages are all within the 2% range we mentioned earlier.

To address the specific metrics you mentioned:

We've posted diffstats with every patch we published on the LKML.

We're talking 3 for syscalls and 2 for all other events in order to reach the core tracing function proper (this could easily be reduced by 1 if it's really a problem). Add 1 for locking scheme and 3 for the non-locking scheme. I'm not counting the calls we make to kernel services, which somewhat goes to show that this is a flawed measure because I've never seen any thorough analysis of call depths for kernel services. Can't say that it wouldn't be an interesting research project to see someone do that for the entire kernel, we may find some interesting results.

This really depends on the quantity of data being passed to the tracer, which varies greatly from one event to the other. I can say this, however:
in all the testing I've seen done on LTT in the past, there has never been a stack problem. This isn't an invitation for being reckless. I am aware of stack issues and have been on the lookout for the any related problem.

Bob has said it best. I think the best that we can do about this is to follow the known-to-be-good guidelines about cache interference. The discussion Ingo and Bob had on this issue in relation to LTT, for example, shows that we've thought this through.

Beyond everything I've said above, I'd invite you to download LTT and try it out. I'm sure you'll see why this is important for Linux users.

BTW, while I'm on the subject of LMbench, I've been trying to find a way to run it on an embedded system. The problem is that this thing needs a compiler and that would mean having to cross-compile gcc itself and so on, which creates storage problems etc. Are there any plans to make a mini-LMbench?

Thanks again,

Karim

===================================================
Karim Yaghmour
[email protected]
Embedded and Real-Time Linux Expert
===================================================

There are a fair number of NAS vendors who do linux boxes with Samba and XFS because of the ACL support, Quantum being the one Tridge now works for by the way. The reason they want it is so they can support the features NT folks are used to having in their file servers. Now, we could just let the NT folks use NT servers instead....

Even getting XFS ACLs running in 2.5 requires part of this patch set.

Steve

I don't really care about ACLs so much one way or the other, but we DEFINITELY use EAs with Lustre, so at the minimum if we could have that part of the changes I'd be happy.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

I'm kind of new here, but I'll present my case in hope that someone listens to me.

This is something that we're just starting to use in my department in Purdue - we work with clustering, and LKCD will let us determine why our nodes decide to kernel panic since it's generally not worthwhile to connect a head to each machine.

I see LKCD as having a big impact by allowing kernels to be debugged after they have panic'd (and thus don't send out a message to syslog). It can especially be usful in compute farms, or other scenerios where it's difficut or cost prohibitive to connect a console (or console server) to each individual machine.

I think that the integration between RAID and LVM is a good thing, and EVMS's 'plug-in module' architecture will help tremendously to bring interoperation with other systems' volume management subsystems. Specifically, the interoperation with IBM's JFS LVM and MS's LVM will be helpful for people trying to migrate their servers over from those OS's to GNU/Linux.

-- Pat

Purdue University ITAP/RCS
Information Technology at Purdue
Research Computing and Storage
http://www-rcd.cc.purdue.edu

JFS has no lvm, it just sits on any blockdevice. The support for Windows dynamic disks actually layers ontop of the MD driver..

To be more specific, I'm talking about AIX's JFS, not linux's JFS...

--
Purdue Universtiy ITAP/RCS
Information Technology at Purdue
Research Computing and Storage
http://www-rcd.cc.purdue.edu

http://dilbert.com/comics/dilbert/archive/images/dilbert2040637020924.gif

I teach Linux classes to corporate IT guys (~300 or so this year) and many of them are migrating from Solaris or deploying Linux along side Solaris.

Solaris has had ACLs since 2.5.1 (1996), and EAs since 2.9 (May 2002).

Having ACL in Linux is a VERY COMMON REQUEST that I hear from the students.

FWIW.

Dax Kelson
Guru Labs

Once again here is my kexec patch once again, updated to work with 2.5.45.

sys_kexec is a system call that allows linux to act as a bootloader for another arbitrary kernel.

What the code does:
It copies data from user space, into buffers in kernel space.
The buffers in kernel space are rearranged so that later I can use
a simply memcpy, to put the data in the page at it's final destination.
The device_shutdown, and the reboot notifier are called.
- This ensures the hardware devices are in a quiescent state so I do not have to worry about them messing up the transfer of control.
The final copy routine is copied to a buffer that won't get stomped. The machine is placed into 32bit protected mode with paging disabled. The final copy routine copies the buffers to their final destination
(which is normally, very similar to where the kernel is running).
The final copy routine jumps to the new loaded kernel image.

At this point the interface is fixed. Anything additional that needs to happen, can be done in user space by adding a stub routine that gets called before the loaded kernel is called. In particular I can directly execute a bzImage which has a 16bit real mode interface.

There is kernel work left to get the device drivers to tell their devices to shut up. (device_shutdown). But device_shutdown already exists, I just have a good test case for it.

Except for the final copy which is very machine specific the rest of the code is generic and has actually been tested on alpha. Eventually I am hoping for ports to other platforms but I am concentrating on x86 so I can do a quality job.

There has been testing and review on the Linux kernel mailing list. Starting with a review of the syscall interface about six months ago. And people testing to be certain they can use the code. While not all of the bugs are worked out in the user space code. The system call is solid.

Everything is configurable so there should be not footprint increase for people who do not want this functionality.

Eric

 MAINTAINERS                        |    7 
 arch/i386/Kconfig                  |   17 +
 arch/i386/kernel/Makefile          |    1 
 arch/i386/kernel/entry.S           |    1 
 arch/i386/kernel/machine_kexec.c   |  142 +++++++++
 arch/i386/kernel/relocate_kernel.S |   99 ++++++
 include/asm-i386/kexec.h           |   25 +
 include/asm-i386/unistd.h          |    1 
 include/linux/kexec.h              |   48 +++
 kernel/Makefile                    |    1 
 kernel/kexec.c                     |  577 +++++++++++++++++++++++++++++++++++++
 kernel/sys.c                       |   61 +++
 12 files changed, 980 insertions

diff -uNr linux-2.5.45/MAINTAINERS linux-2.5.45.x86kexec/MAINTAINERS
@@ -934,6 +934,13 @@
 W:	http://www.cse.unsw.edu.au/~neilb/patches/linux-devel/
 S:	Maintained
 
+KEXEC
+P:	Eric Biederman
+M:	[email protected]
+M:	[email protected]
+L:	[email protected]
+S:	Maintained
+
 LANMEDIA WAN CARD DRIVER
 P:	Andrew Stanley-Jones
 M:	[email protected]

diff -uNr linux-2.5.45/arch/i386/Kconfig linux-2.5.45.x86kexec/arch/i386/Kconfig
@@ -784,6 +784,23 @@
 	depends on (SMP || PREEMPT) && X86_CMPXCHG
 	default y
 
+config KEXEC
+	bool "kexec system call (EXPERIMENTAL)"
+	depends on EXPERIMENTAL
+	help
+	  kexec is a system call that implements the ability to  shutdown your
+	  current kernel, and to start another kernel.  It is like a reboot
+	  but it is indepedent of the system firmware.   And like a reboot the
+	  you can start any kernel with it not just Linux.  
+	
+	  The name comes from the similiarity to the exec system call. 
+	
+	  It is on an going process to be certain the hardware in a machine
+	  is properly shutdown, so do not be surprised if this code does not
+	  initially work for you.  It may help to enable device hotplugging
+	  support.  As of this writing the exact hardware interface is
+	  strongly in flux, so no good recommendation can be made.
+
 endmenu
 
 

diff -uNr linux-2.5.45/arch/i386/kernel/Makefile linux-2.5.45.x86kexec/arch/i386/kernel/Makefile
@@ -25,6 +25,7 @@
 obj-$(CONFIG_X86_MPPARSE)	+= mpparse.o
 obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o nmi.o
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
+obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o
 obj-$(CONFIG_SOFTWARE_SUSPEND)	+= suspend.o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq.o
 obj-$(CONFIG_PROFILING)		+= profile.o

diff -uNr linux-2.5.45/arch/i386/kernel/entry.S linux-2.5.45.x86kexec/arch/i386/kernel/entry.S
@@ -740,6 +740,7 @@
 	.long sys_epoll_create
 	.long sys_epoll_ctl	/* 255 */
 	.long sys_epoll_wait
+	.long sys_kexec
 
 
 	.rept NR_syscalls-(.-sys_call_table)/4

diff -uNr linux-2.5.45/include/asm-i386/kexec.h linux-2.5.45.x86kexec/include/asm-i386/kexec.h
@@ -0,0 +1,25 @@
+#ifndef _I386_KEXEC_H
+#define _I386_KEXEC_H
+
+#include <asm/fixmap.h>
+
+/*
+ * KEXEC_SOURCE_MEMORY_LIMIT maximum page get_free_page can return.
+ * I.e. Maximum page that is mapped directly into kernel memory,
+ * and kmap is not required.
+ *
+ * Someone correct me if FIXADDR_START - PAGEOFFSET is not the correct
+ * calculation for the amount of memory directly mappable into the
+ * kernel memory space.
+ */
+
+/* Maximum physical address we can use pages from */
+#define KEXEC_SOURCE_MEMORY_LIMIT (FIXADDR_START - PAGE_OFFSET) 
+/* Maximum address we can reach in physical address mode */
+#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)
+
+#define KEXEC_REBOOT_CODE_SIZE	4096
+#define KEXEC_REBOOT_CODE_ALIGN 0
+
+
+#endif /* _I386_KEXEC_H */

diff -uNr linux-2.5.45/include/asm-i386/unistd.h linux-2.5.45.x86kexec/include/asm-i386/unistd.h
@@ -261,6 +261,7 @@
 #define __NR_sys_epoll_create	254
 #define __NR_sys_epoll_ctl	255
 #define __NR_sys_epoll_wait	256
+#define __NR_sys_kexec		257
   
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */

diff -uNr linux-2.5.45/kernel/Makefile linux-2.5.45.x86kexec/kernel/Makefile
@@ -21,6 +21,7 @@
 obj-$(CONFIG_CPU_FREQ) += cpufreq.o
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o
+obj-$(CONFIG_KEXEC) += kexec.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

There are people within IBM in Germany, India and England, as well as a number of companies (Intel, NEC, Hitachi, Fujitsu), as well as SGI that are PAID to support this. In addition, Global Services at IBM uses this as a front-line method for resolving customer problems. If you're looking for names of people to sign up to support it (both vendors and non-vendors), I can make that list up for you.

There are a number of us (developers, support staff, and other interested parties) who bend over backwards, day in and day out to make sure this stuff works and helps people, even if it isn't kernel developers (directly -- indirectly, you get bug reports that are sane and useful).

It's not sexy kernel stuff, but it is very important, and if you'd like, I can have representatives from at least 10 major corporations (Fortune 500 companies) contact you to request that this go in.

We're generating 2.5.45 patches now, and we ask that you include the patches when they are posted.

I don't know what else to say except that people really want this stuff and all of us in the LKCD community work really hard together to make this project useful for everyone.

Please include this in your next snapshot.

--Matt

P.S. Copying some of the users and developers.

That's fine. And since they are paid to support it, they can apply the patches.

What I'm saying by "vendor driven" is that it has no relevance for the standard kernel, and since it has no relevance to that, then I have no incentives to merge it. The crash dump is only useful with people who actively look at the dumps, and I don't know _anybody_ outside of the specialized vendors you mention who actually do that.

I will merge it when there are real users who want it - usually as a result of having gotten used to it through a vendor who supports it. (And by "support" I do not mean "maintain the patches", but "actively uses it" to work out the users problems or whatever).

Horse before the cart and all that thing.

People have to realize that my kernel is not for random new features. The stuff I consider important are things that people use on their own, or stuff that is the base for other work. Quite often I want vendors to merge patches _they_ care about long long before I will merge them (examples of this are quite common, things like reiserfs and ext3 etc).

THAT is what I mean by vendor-driven. If vendors decide they really want the patches, and I actually start seeing noises on linux-kernel or getting requests for it being merged from _users_ rather than developers, then that means that the vendor is on to something.

Linus

To add to that list, here at Purdue University, we actively look at crash dumps on other architectures, such as IBM AIX, and are starting to do the same on Linux machines, after discovery of LKCD.

This has much relevance for the standard kernel, as much relevance as gdb has for people using applications. While a majority of non-techno-geek end-users probably don't care about the patch, I'm certain that there are plenty of organizations out there like Purdue that WANT lkcd to become a
standard part of the Linux kernel. Until then, we're forced to do our
own kernel patching every time we push out a new kernel.

We actively use it.

LKCD isn't a 'random new feature'. It's something that is present in nearly ever other "Unix" on the market. (Yes I know Unix != Linux). It's a feature that should have been integrated by now IMHO.

Again, we're the end-user, not the vendor, and we're trying to drive to have it included. I've talked with outher sys admins in my department here at Purdue, and have gotten a unanimous response that "It would be a good and useful feature to have."

Pat
--
Purdue Universtiy ITAP/RCS
Information Technology at Purdue
Research Computing and Storage
http://www-rcd.cc.purdue.edu

http://dilbert.com/comics/dilbert/archive/images/dilbert2040637020924.gif

On Thu, 31 Oct 2002, Linus Torvalds wrote:

|>On Wed, 30 Oct 2002, Matt D. Robinson wrote:
|>That's fine. And since they are paid to support it, they can apply the 
|>patches.  

Sure, but why should they have to? What technical reason is there for not including it, Linus?

I completely don't understand your reasoning here. I use it for my home, not for work, and that's important for me. And not everyone can spend their evenings rolling up the next set of patches for a distribution. Yes, vendors want it, they need it, but there are plenty of people like me that want this in too!

We want to see this in the kernel, frankly, because it's a pain in the butt keeping up with your kernel revisions and everything else that goes in that changes. And I'm sure SuSE, UnitedLinux and (hopefully) Red Hat don't want to spend their time having to roll this stuff in each and every time you roll a new kernel.

I mean, PLEASE, Linus, what do we have to do? There are so many interests in this stuff, and I really, truly don't get what's wrong with putting this in the kernel?

Have you looked at it? Have you looked at how it is now structure to be non-invasive? How it will allow other kernel developers to generate their own dumping methods? I mean, we sent you E-mails weeks ago, and you didn't respond to any of them with even a word of acknowledgement of receipt.

|>What I'm saying by "vendor driven" is that it has no relevance for the 
|>standard kernel, and since it has no relevance to that, then I have no 
|>incentives to merge it. The crash dump is only useful with people who 
|>actively look at the dumps, and I don't know _anybody_ outside of the 
|>specialized vendors you mention who actually do that.

I do. Others like myself do. And not just for development purposes. I don't like to see my system crash after installing one of your new kernels and not be able to figure out what's wrong. The nice thing is that LKCD there, it works, and I can just look at the crash report instead of wishing that my console buffer didn't just scroll off. Oh, I know, I'll just wait for it to happen again ... yeah, like that's real intelligent.

|>I will merge it when there are real users who want it - usually as a
|>result of having gotten used to it through a vendor who supports it. (And
|>by "support" I do not mean "maintain the patches", but "actively uses it"
|>to work out the users problems or whatever).
|>
|>Horse before the cart and all that thing.
|>
|>People have to realize that my kernel is not for random new features. The
|>stuff I consider important are things that people use on their own, or
|>stuff that is the base for other work. Quite often I want vendors to merge
|>patches _they_ care about long long before I will merge them (examples of
|>this are quite common, things like reiserfs and ext3 etc).

Other vendors have merged LKCD a long time ago and use it, and expect it to be there. And users like myself find it valuable on their desktops, their servers, etc. I mean, there's someone using this at Purdue that's responded to you, just another kernel user that likes to have this stuff there automatically.

|>THAT is what I mean by vendor-driven. If vendors decide they really want
|>the patches, and I actually start seeing noises on linux-kernel or getting
|>requests for it being merged from _users_ rather than developers, then
|>that means that the vendor is on to something.

TurboLinux, MonteVista, Veritas, SuSE, and UnitedLinux have LKCD. With the most recent changes, I think Red Hat can put LKCD in now such that it isn't invasive to their distribution.

I think SuSE has already expressed a desire to have this in. If you want to hear from others, I'll asked them to respond to you.

|> Linus

--Matt

[ Ok, this is a really serious email. If you don't get it, don't bother
emailing me. Instead, think about it for an hour, and if you still don't get it, ask somebody you know to explain it to you. ]

There are many:

- bloat kills:

My job is saying "NO!"

In other words: the question is never EVER "Why shouldn't it be accepted?", but it is always "Why do we really not want to live without this?"

- included features kill off (potentially better) projects.

There's a big "inertia" to features. It's often better to keep features _off_ the standard kernel if they may end up being further developed in totally new directions.

In particular when it comes to this project, I'm told about "netdump", which doesn't try to dump to a disk, but over the net. And quite frankly, my immediate reaction is to say "Hell, I _never_ want the dump touching my disk, but over the network sounds like a great idea".

To me this says "LKCD is stupid". Which means that I'm not going to apply it, and I'm going to need some real reason to do so - ie being proven wrong in the field.

(And don't get me wrong - I don't mind getting proven wrong. I change my opinions the way some people change underwear. And I think that's ok).

Tough. That's YOUR problem.

Linus

On Thu, 31 Oct 2002, Linus Torvalds wrote:

|>[ Ok, this is a really serious email. If you don't get it, don't bother 
|>  emailing me. Instead, think about it for an hour, and if you still don't 
|>  get it, ask somebody you know to explain it to you. ]

Thanks for the response. I don't think I need an hour. This is pretty simple.

|>On Thu, 31 Oct 2002, Matt D. Robinson wrote:
|>> 
|>> Sure, but why should they have to?  What technical reason is there
|>> for not including it, Linus?
|>
|>There are many:
|>
|> - bloat kills:
|>
|>	My job is saying "NO!"
|>
|>	In other words: the question is never EVER "Why shouldn't it be
|>	accepted?", but it is always "Why do we really not want to live 
|>	without this?"

This isn't bloat. If you want, it can be built as a module, and not as part of your kernel. How can that be bloat? People who build kernels can optionally build it in, but we're not asking that it be turned on by default, rather, built as a module so people can load it if they want to. We made it into a module because 18 months ago you complained about it being bloat. We addressed your concerns.

Some people, particularly large SSI configurations, can't live without this. You shouldn't crash once. Crashing twice, or more often, is inexcusable.

|> - included features kill off (potentially better) projects.
|>
|>	There's a big "inertia" to features. It's often better to keep 
|>	features _off_ the standard kernel if they may end up being
|>	further developed in totally new directions.

I can't argue against this ... to do so would mean that you don't accept any new features for 2.5, and there are a lot of projects like mine that need to go in, although I do understand your concerns.

|>	In particular when it comes to this project, I'm told about
|>	"netdump", which doesn't try to dump to a disk, but over the net.
|>	And quite frankly, my immediate reaction is to say "Hell, I
|>	_never_ want the dump touching my disk, but over the network
|>	sounds like a great idea".

We've integrated the "netdump" capabilities as a dump method for LKCD. It's an option for dumping, just like all the other dump methods available to people? Want to dump to disk? Use LKCD. Want to dump on the network? USE LKCD. What's wrong with that?

We've created a net dump method that allows you to dump across the network from Mohammed Abbas (modified from Ingo's netconsole dump). It integrates into LKCD beautifully. If you want that patch with the rest of our LKCD patches, we can include it, no problem.

|>To me this says "LKCD is stupid". Which means that I'm not going to apply 
|>it, and I'm going to need some real reason to do so - ie being proven 
|>wrong in the field.

Hopefully some of this changes your mind.

|>(And don't get me wrong - I don't mind getting proven wrong. I change my 
|>opinions the way some people change underwear. And I think that's ok).
|>
|>> I completely don't understand your reasoning here.
|>
|>Tough. That's YOUR problem.

It is. I lose sleep because this is my problem. I lose time on the weekends because this is my problem.

If you've _reviewed_ the LKCD patches and still have the opinions you've mentioned above, then I'll consider this your position and be done with it. Otherwise, please accept the code.

We'll keep doing our best to keep up with your kernels in the meantime.

|> Linus

--Matt

I don't care one _whit_ about the size of the binary. I don't maintain binaries, adn the binary can be gigabytes for all I care.

The only thing I care about is source code. So the "build it as a module and it is not bloat" argument is a total nonsense thing as far as I'm concerned.

Anyway, new code is always bloat to me, unless I see people using them.

Guys, why do you even bother trying to convince me? If you are right, you will be able to convince other people, and that's the whole point of open source.

Being "vendor-driven" is _not_ a bad thing. It only means that _I_ am not personally convinced. I'm only one person.

Linus

So, you don't like bloat, such as having 22 different file systems (only including the ones that can be placed on disk, not things like devfs or smbfs...). That's more filesystems than I have dollars in my wallet at
the moment. For the amount of utility that this code provides, it's
definately not 'bloat'.

HEY!!! WE'RE USING IT!!!

Now this sounds more like something I'd hear from Sun trying to get a fix for a version of Solaris without having to buy a new one. I thought the whole point of Free Software was sharing with the community, and doing what's best for the community.

That's the same as claiming that George W. Bush is just one person....

So I'll plea yet again, please add LKCD!

Pat
--
Purdue Universtiy ITAP/RCS
Information Technology at Purdue
Research Computing and Storage
http://www-rcd.cc.purdue.edu

http://dilbert.com/comics/dilbert/archive/images/dilbert2040637020924.gif

It sounds to me like there needs to be L-K traffic when problems are solved using LKCD.

Personally I love crash dumps... in 33 years of computing I have spent a total of 1-2 years doing nothing but enhancing and developing post-processing facilities. The true benefit is not just the "crashed here, add a null check nonsense". It is the ability to examine the whole system state. With an inboard trace table, you can even go back in time. You can look at call stacks, locks held, state of allocated memory, etc etc. If you save callstacks and time with allocated memory, you can track down storage growth problems. I have spent weeks winkling problems out of crash dumps, solving problems the developers didn't even know existed.

With the right facility you can take crash dump snapshots and keep on running. It is a great tool for understanding a system.

But until there is a flow of results - good quality fixes - resulting from such analysis, I can see exactly why LT is doubtful.

john alvord

"Matt D. Robinson" <[email protected]> dijo:

[...]

Bloat is not just RAM/CPU/... usage when in use, it is much more about developers who have to understand, work with, and consider how to use or interface with the new code. Even more so when it is not builtin, as this creates _two_ scenarios to consider.

This is the sense of "bloat" that Linus is most worried about (and very rightly so, IMVHO). At lesat that is my observation over the years.

Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

I'm not sure I understand your point, Horst. There are four primary mechanisms which would invoke a dump:

	die() (or die_if_kernel())
	panic()
	interrupt-driven dumps
	sysrq()

Assuming you call these functions, there is a single dump() call that will perform dumping, the the dump_function_ptr (which is assigned when the dump module is loaded) is set. dump() is a simple function that basically says:

static inline void dump(char * str, struct pt_regs * regs)

{
	if (dump_function_ptr) {
		dump_function_ptr((char *)str, regs);
	}
}

str is for the panic() string, and regs are so you can create a proper stack trace for the failing task on the correct CPU.

I don't see how that can can attributed to bloating the kernel. If you don't panic(), the code is never invoked. If you don't load the dump module, dump_function_ptr isn't assigned. It's meant to be non-invasive, off to the side and called when required (or requested).

There is some additional code put in the kernel to disable interrupts, quiesce the system, and I think there are a few projects that can probably use the same code base (such as the suspend-to-ram project, which I was just informed about). All of that is called within the dump driver itself, otherwise it sits quietly off to the side, never getting called.

Using the dump driver infrastructure is like writing any plain-jane driver. You set up the _open(), _close(), etc., functions, assigning the ops table based on the dump method you want to use (disk, network, mini-oopser, etc.) This isn't that difficult, and it should only be loaded for those customer systems that want a specific dump style.

--Matt

Standard disclaimer: I'm not trying anymore to get this into the kernel at this time (via Linus). This is purely for educating those that aren't familiar with crash dumping for Linux.

On Sat, 2 Nov 2002, Horst von Brand wrote:

|>"Matt D. Robinson" <[email protected]> dijo:
|>
|>[...]
|>
|>> This isn't bloat.  If you want, it can be built as a module, and
|>> not as part of your kernel.  How can that be bloat?  People who
|>> build kernels can optionally build it in, but we're not asking
|>> that it be turned on by default, rather, built as a module so
|>> people can load it if they want to.  We made it into a module
|>> because 18 months ago you complained about it being bloat.  We
|>> addressed your concerns.
|>
|>Bloat is not just RAM/CPU/... usage when in use, it is much more about
|>developers who have to understand, work with, and consider how to use or
|>interface with the new code. Even more so when it is not builtin, as this
|>creates _two_ scenarios to consider.
|>
|>This is the sense of "bloat" that Linus is most worried about (and very
|>rightly so, IMVHO). At lesat that is my observation over the years.
|>

How do you deal with netdump when your network driver is what caused the crash?

Ideally I would like to see a dump framework that can have a number of possible dump targets. We should be able to dump to any combination of network, serial, disk, flash, unused ram that isn't wiped over restarts, etc...

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Actually, from a driver perspective, _the_ most likely driver to crash is the disk driver.

That's from years of experience. The network drivers are a lot simpler, the hardware is simpler and more standardized, and doesn't do as many things. It's just plain _easier_ to write a network driver than a disk driver.

Ask anybody who has done both.

But that's not the real issue. The real issue is that I have no personal incentives to try to merge the thing, and as a result I think I'm the wrong person to do so. I've told people over and over again that I think this is a "vendor merge", and I'm fed up with people not _getting_ it.

Don't bother to ask me to merge the thing, that only makes me get even more fed up with the whole discussion. This is open source, guys. Anybody can merge it. Because I don't particularly believe in it doesn't mean that it cannot be used. It only means that I want to see users flock to it and show my beliefs wrong.

Linus

It would be most unfortunate if the existance of netdump is used as a reason to deny LKCD's inclusion, or to simply dismiss LKCD as stupid.

While Red Hat advocates Ingo's netdump option, we have customer requests that are requiring us to look at LKCD disk-based dumps as an alternative, co-existing dump mechanism. Since the two methods are not mutually exclusive, LKCD will never kill off netdump -- nor certainly vice-versa. We're all just looking for a better means to be able to
provide support to our customers, not to mention its value as a development aid.

Dave Anderson
Red Hat, Inc.

What he really wants is for Andrew or Alan or someone else he trusts to merge it, get actual field results, and declare it useful. If people start visibly passing around crash dump results on l-k and solving problems with them, that'll help too. Until then all he has is his gut feel to go on.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

|>On Thu, 31 Oct 2002, Matt D. Robinson wrote:
|>> We want to see this in the kernel, frankly, because it's a pain
|>> in the butt keeping up with your kernel revisions and everything
|>> else that goes in that changes.  And I'm sure SuSE, UnitedLinux and
|>> (hopefully) Red Hat don't want to spend their time having to roll
|>> this stuff in each and every time you roll a new kernel.
|>
|>While Red Hat advocates Ingo's netdump option, we have customer
|>requests that are requiring us to look at LKCD disk-based dumps as an
|>alternative, co-existing dump mechanism.  Since the two methods are
|>not mutually exclusive, LKCD will never kill off netdump -- nor
|>certainly vice-versa.  We're all just looking for a better means
|>to be able to provide support to our customers, not to mention
|>its value as a development aid.

I think you and I are in agreement (as always has been in the past), Dave. LKCD is meant to create a base for disk, network, or any dump method. If Red Hat wants netdump to be the primary dumping method, that's Red Hat's decision, and more power to them. If SuSE wants disk dumps, that's SuSE's decision. But for both of them to have to roll their own every single release or kernel upgrade is unproductive.

What's most concerning about this entire discussion is that I bet < 20% of the people discussing this have actually LOOKED at the LKCD patches to see whether or not this is as invasive, difficult, bloated, or anything negative. We've spent over a month now posting them, getting comments, responding to all of the comments, making sure feedback is accounted for and responded to, only to get an "LKCD is stupid" type response.

--Matt

From the standpoint of just the driver that's true. However, the remote
machine and all the network bits between them are a string of single points of failure. Isn't it good that both disk and network can be supported.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

My concerns are solely with things like the correctness of the disk dumper. Its obviously a good way to do a lot more damage if it isnt done carefully. Quite clearly your dump system wants to support multiple dump targets so you can dump to pci battery backed ram, down the parallel port to an analysing box etc

I always liked the AIX dumper choices. You could either dump to
the swap area (and startup detects the dump and moves it to the filesystem before swapon) or provide a dedicated dump partition. The latter was prefered.
Either of these methods merely require the dumper to correctly
write to one disk partition. This is about as simple as you are going to get in disk dumping.

Joel

"You must remember this:
 A kiss is just a kiss,
 A sigh is just a sigh.
 The fundamental rules apply
 As time goes by."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127

Ehh.. That was on closed hardware that was largely designed with and for the OS.

Alan isn't worried about the "which sector do I write" kind of thing. That's the trivial part. Alan is worried about the fact that once you know which sector to write, actually _doing_ so is a really hard thing. You have bounce buffers, you have exceedingly complex drivers that work differently in PIO and DMA modes and are more likely than not the _cause_ of a number of problems etc.

And you have a situation where interrupts are not likely to work well (because you crashed with various locks held), so the regular driver simply isn't likely to work all that well.

And you have a situation where there are hundreds of different kinds of device drivers for the disk.

In other words, the AIX situation isn't even _remotely_ comparable. A large portion of the complexity in the PC stability space is in device drivers. It's the thing I worry most about for 2.6.x stabilization, by _far_.

And if you get these things wrong, you're quite likely to stomp on your disk. Hard. You may be tryign to write the swap partition, but if the driver gets confused, you just overwrote all your important data. At which point it doesn't matter if your filesystem is journaling or not, since you just potentially overwrote it.

In other words: it's a huge risk to play with the disk when the system is already known to be unstable. The disk drivers tend to be one of the main issues even when everything else is _stable_, for chrissake!

To add insult to injury, you will not be able to actually _test_ any of the real error paths in real life. Sure, you will be able to test forced dumps on _your_ hardware, but while that is fine in the AIX model ("we control the hardware, and charge the user five times what it is worth"), again that doesn't mean _squat_ in the PC hardware space.

See?

Linus

On the other hand, ISC's system 5 r3 ran on commodity x86 hardware and the
crash dumper worked on the various disk hardware I had occasion to use it on (mfm, scsi, ide), although one did need to make sure swap was larger than ram or bad things would happen. 8-{.

The thing is that Solaris, AIX, and ISC are written by commercial
companies, they realize that customers need to be able to debug systems which don't have a screen, a serial printer, etc. They do have disk.

I was hoping Alan would push Redhat to put this in their Linux so we
could resolve some of the ongoing problems which don't write an oops to a log, but I guess none of the developers has to actually support production servers and find out why they crash.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Perhaps i'm being grossly naive here, but none of these presumably x86 productions servers don't have a serial port? Not even PCI/ISA slots to add one? Serial would catch most of your oopsen anyway, and if you were borked enough that serial couldn't get the entire output, i somehow doubt dumping to disk could manage. And no i don't see anything wrong nor consider it studly to use oopses only for debugging...

Zwane

function.linuxpower.ca

I have distributed servers in 15 locations, six states, four timezones. In secure unattended locations like wiring closets. What do I do with the serial port? Do I double my colocation costs and have another system there to listen? Is the code on a sick system going to dial the modem on the serial line amd establish a connection?

I have a mix of Linux, Solaris, and AIX systems deployed, and only the Linux systems don't have this capability. Actually for the most part only the Linux systems NEED it, that's another problem, but reliability would go up if I could see the problem.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

I think several Red Hat people would disagree very strongly. Red Hat shipped with the kernel symbol decoding oops reporter for a good reason, and also acquired netdump for a good reason.

On 2 Nov 2002, Alan Cox wrote:

|>On Sat, 2002-11-02 at 05:17, Bill Davidsen wrote:
|>>   I was hoping Alan would push Redhat to put this in their Linux so we
|>> could resolve some of the ongoing problems which don't write an oops to a
|>> log, but I guess none of the developers has to actually support production
|>> servers and find out why they crash.
|>
|>I think several Red Hat people would disagree very strongly. Red Hat
|>shipped with the kernel symbol decoding oops reporter for a good reason,
|>and also acquired netdump for a good reason. 

It would be great if crash dumping were an option, at the very least to unify the netdump, oops reporter and disk dumping (for those that want it) into a single infrastructure. Long term, that's probably where this is going anyway. It takes away the religious "who is right" argument, which is fundamentally silly.

Maybe one day. I think quite a few Red Hat customers would appreciate it.

--Matt

P.S. IBM shouldn't have signed a contact with Red Hat without
requiring certain features in Red Hat's OS(es). Pushing for LKCD, kprobes, LTT, etc., wouldn't be on this list for a whole variety of cases if that had been done in the first place.

P.S. As an aside, too many engineers try and make product marketing
decisions at Red Hat. I personally think that's really bad for their business model as a whole (and I'm not referring to LKCD).

I would hope IBM have more intelligence than to attempt to destroy the product by trying to force all sorts of junk into it. The Linux world has a process for filterng crap, it isnt IBM applying force. That path leads to Star Office 5.2, Netscape 4 and other similar scales of horror code that become unmaintainably bad.

You think things like EVMS are a product marketing decision. I'm very glad you don't run a Linux distro. It would turn into something like the old 3com rapops rather rapidly by your models (3com rapops btw ceased to exist and for good reasons)

Alan

On 3 Nov 2002, Alan Cox wrote:

|>On Sun, 2002-11-03 at 01:24, Matt D. Robinson wrote:
|>> P.S.  IBM shouldn't have signed a contact with Red Hat without
|>>       requiring certain features in Red Hat's OS(es).  Pushing for
|>>       LKCD, kprobes, LTT, etc., wouldn't be on this list for a whole
|>>       variety of cases if that had been done in the first place.
|>
|>I would hope IBM have more intelligence than to attempt to destroy the
|>product by trying to force all sorts of junk into it. The Linux world
|>has a process for filterng crap, it isnt IBM applying force. That path
|>leads to Star Office 5.2, Netscape 4 and other similar scales of horror
|>code that become unmaintainably bad.

I think you misunderstand me. If IBM considers a feature to be useful, they should require distributions to put into a release from a contractual standpoint. That doesn't mean Red Hat has to put it into all their distributions -- it just means they have to produce something that IBM wants. If nobody else uses it, that's fine. IBM gets what they want, and Red Hat gets what they want. End of story.

You're looking at this from an engineering perspective and open source philosophy rather than a business unit at a company like IBM might look at it. That's not a bad thing to do, but the two concepts are very different from each other. The Linux world may filter "crap", which is great, but some of that "crap" is important to companies like IBM, and if they were smart they'd use their leverage ($$$) to make sure the "crap" ends up in the products they care to use/support. The rest of Linux can do whatever it wants, doing things the "Linux world" way.

|>> P.S.  As an aside, too many engineers try and make product marketing
|>>       decisions at Red Hat.  I personally think that's really bad for
|>>       their business model as a whole (and I'm not referring to LKCD).
|>
|>You think things like EVMS are a product marketing decision. I'm very
|>glad you don't run a Linux distro. It would turn into something like the
|>old 3com rapops rather rapidly by your models (3com rapops btw ceased to
|>exist and for good reasons)

Again, I wasn't mentioning any product in particular. Making decisions like GPL-only as an engineering philosophy rather than as a product marketing decision are more problematic than looking at EVMS vs. anything else as a question of which is technically better.

But again, that's a complete aside and would probably open up a plethora of opinions from people who care about both sides of that argument, and would inevitably head down an rathole infinitely deep.

--Matt

If you define "unmaintainably bad" as "having features you don't need" then I agree. But since dump to disk is in almost every other commercial UNIX, maybe someone would question why it's good for others but not for Linux.

I can agree on stuff the non-hacker wouldn't use, but that is exactly who uses the crash dump in AIX, the person who wants to send a compressed dump and money to IBM and get back a fix. Netdump assumes external resources and a functional secure network (is the dump encrypted and I missed it?) which home users surely don't have, and remote servers oftem lack as well.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

It is even in FreeBSD or Windows > ME

Greetings
Bernd

It isnt about features, its about clean maintainable code. netdump to me doesnt mean no dump to disk option. In fact I'd rather like to be able to insmod dump-foo.o. The correctness issues are hard but if the dump-foo is standalone, resets the hardware and has an SHA integrity check then it can be done (think of it as a post crash variant of the trusted computing TCB verification problem)

Lots of interesting legal issues but yes you can do it sometimes (DMCA, privacy, financial duties sometimes make it horribly complex). Even in the case where you only dump the oops its still valuable.

Encrypting the dump with the new crypto lib in the kernel would be easy, right now it doesnt.

My disk dump concerns are purely those of correctness. That means

1. After loading the module getting the block list for the dump target

2. Resetting and scratch initializing the dump device

3. Not relying on any code outside of the dump TCB that may have
been corrupted

4. At dump time turning off all bus masters, doing the dump TCB
verification and then dumping

Most of the pieces already exist.

On 3 Nov 2002, Alan Cox wrote:
|>Encrypting the dump with the new crypto lib in the kernel would be easy, |>right now it doesnt.

Piece of cake. It's like adding a dump compression module. You can load dump_gzip.o or dump_rle.o to specify the kind of compression you want to use. dump_crypto.o would be the same kind of thing. Just add another flag and away you go.

|>My disk dump concerns are purely those of correctness. That means
|>
|> [ ... ]
|>
|>Most of the pieces already exist.

It's just a matter of time, then.

--Matt

I certainly don't disagree, but the one critical problem is writing the dump to the right place, or at least not writing to the wrong place. I'd love to have disk, net, NVram, whatever choices, but disk is the one which would help the most. AIX and ISC have dump to swap, and the swapon copies the data back or clears it, with a fresh O/S load to ensure writing the right place.

Agreed, I would think about doing that with a mail server. But even an oops like ksymoops would be helpful. I started on systems with dumps, ksymoops is wonderful by comparison.

That could all be built as part of init, clearly you can't depend on demand loading the module.

If the modules are to be really self-sufficient it would have to include the driver. I'll let someone tell me that's not always the case if the driver can have its own data area.

Yes, although with separate code, stack and data that's less likely. In the bad old days self-modifying code was common.

The first part of that looks medium hard, particularly if the code has to be part of the dump module.

Clearly it can be done even better than the current implementation, and given an interface standard a replacement in the whole could be done.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Bah, it's enough that IBMs money totally fucked up the tree of one popular distribution..

On Fri, 1 Nov 2002, Linus Torvalds wrote:

|>Alan isn't worried about the "which sector do I write" kind of thing.  
|>That's the trivial part. Alan is worried about the fact that once you know
|>which sector to write, actually _doing_ so is a really hard thing. You
|>have bounce buffers, you have exceedingly complex drivers that work
|>differently in PIO and DMA modes and are more likely than not the _cause_
|>of a number of problems etc.

[ preamble - this is only a technical discussion, I'm interested
in feedback on what we can improve upon ]

I agree with you. We'd prefer to have a better low-level driver primitive sitting on top of two low-level disk drivers (IDE and SCSI). Fundamentally, though, this is difficult to do:

0) There's a lot of early stuff you take risks with, such as the partition size (assuming you can probe it), knowing that it hasn't changed since boot, and pre-allocating buffers for disk I/O operations. You always take the partition risk no matter what.

1) You have to establish that the IDE or SCSI device can be reset into an appropriate mode for seek/write mode -- if a DMA operation fails to the drive, and you can't reset the drive, you may be stuck.

2) Once the hardware reports back success, it is a matter of how you write the blocks. I once wrote the low-level IDE driver below request structures, writing sequentially to the drive, and ran into occasional drive lock-ups while writing during interrupt crashes. This was more likely due to my inexperience with the IDE driver than anything else.

|>And you have a situation where interrupts are not likely to work well
|>(because you crashed with various locks held), so the regular driver
|>simply isn't likely to work all that well.

This is simply an avoidance of certain code paths. We saw this problem earlier in 2.2 using kiobufs and got around it for the most part by doing our best to avoid the io_request_lock. That's why we haven't seen the lock contention problems for 2.5.

|>And you have a situation where there are hundreds of different kinds of |>device drivers for the disk.

This is the biggest problem, absolutely. Our idea moving forward was to create a _dump() primitive with drivers that allows you to determine, upon configuration of a disk dump device, whether or not the low-level driver supported dumping or not. I suggested this to Al Viro a long time ago on this list, but it didn't go anywhere.

That way the driver itself knows that it can support a low-level page-write method. If it doesn't, you can't use disk dumping to that device.

I'm willing to re-open this effort.

|>And if you get these things wrong, you're quite likely to stomp on your
|>disk. Hard. You may be tryign to write the swap partition, but if the
|>driver gets confused, you just overwrote all your important data. At which
|>point it doesn't matter if your filesystem is journaling or not, since you
|>just potentially overwrote it.

We haven't seen this before, but it is always a possibility for any dump scenario. That's why you some choose netdump instead. :)

|>In other words: it's a huge risk to play with the disk when the system is
|>already known to be unstable. The disk drivers tend to be one of the main
|>issues even when everything else is _stable_, for chrissake!
|>
|>To add insult to injury, you will not be able to actually _test_ any of 
|>the real error paths in real life. Sure, you will be able to test forced 
|>dumps on _your_ hardware, but while that is fine in the AIX model ("we 
|>control the hardware, and charge the user five times what it is worth"), 
|>again that doesn't mean _squat_ in the PC hardware space.

We have actually done a lot of testing with injection of failures into the middle of VM, network drivers, etc., in conjunction with disk dumping. Certainly it doesn't cover all the cases, but nothing ever will.

|> Linus

--Matt

[email protected] (Matt D. Robinson) wrote on 01.11.02 in <[email protected]>:

*If* you want safe dumping to a partition, it seems wrong to me to try to figure that out after the crash.

Instead,

* configure the crash space with a user-mode app or possibly a kernel command line arg
* Whenever repartitioning, check if the crash dump partition is affected, and if so, clear it until it is explicitely reconfigured
* Save a good checksum (say, md5 or sha1) of the crash partition config, and only dump if that checksum checks out

You might want to checksum even more than that, of course :-)

But there's certainly a reason Netware liked to crash dump to a series of floppies - too bad those are much too small for today's machines. When floppy sizes stopped to be slightly larger than standard RAM sizes[*], the computing public lost big time, and we haven't recovered from that.

[*] Apple ][+: 48 KB RAM, 140 KB floppy. IBM PC: 640 KB RAM, 1.2 MB floppy. (Yes, I know there were other combinations as well.) Where's my approximately-1-GB floppy that everyone and their aunt have installed today? No, CD writers are *not* universal. And burn-once CDs aren't much like floppies.

Of course, the same problem exists with general backup technology - tape the size of modern disks is not really affordable anymore.

MfG Kai

One question I have is how much of the driver problem you refer to is becouse of optimizations that the various drivers have, could you fall back to the simplest, works-with-everything, all-timeouts-longer-then-the-slowest-disk slug of a driver that could be used to do this dump?

David Lang

[ Cc: trimmed ]

Welcome to the wonderful world of code duplication. And don't forget the "simplified" TCP/IP stack for network dumps. Uh, USB-attached storage, anyone ? :-)

Special-case dump drivers make perfect sense in isolated cases (e.g. narrowly specified boxes) or as a band-aid solution.

But for a general solution, it seems more appropriate to me to solve the problem of moving the kernel data from the damaged system to an intact system only once, e.g. using the MCORE approach, than over and over again for all possible types of hardware and attachment methods.

The only inherent weakness I see in MCORE is the need to reliably reset a device, either to the point where it is operational (if used in the process of dumping), or at least to the point where it doesn't get in the way (if not used for the dump, e.g. video, HID, etc.).

But this should still be significantly easier than introducing "dumb" versions for all drivers. Besides, having a way for cleanly shutting down or resetting devices is desirable in other contexts, too (e.g. kexec).

- Werner (disclaimer: not affiliated with Mission Critical Linux, any vendor, or any other form of gainful employment)

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

This is just a random tangential thought here, but FWIW:

Why not just have a simple backup stripped-down "hardened" copy of Linux lying around in a physical RAM region not used by the copy of Linux actually running. Granted the running Linux doesn't do random physical accesses when dying, the crash handler could then just boot that secondary Linux which would then have a RAM disk containing the appropriate scripts and binaries to handle the actual crash. Given the cost of RAM these days, reserving a MB or two for this purpose should probably not be that bad.

Karim

===================================================
Karim Yaghmour
[email protected]
Embedded and Real-Time Linux Expert
===================================================

Congratulations, you've just re-invented MCORE :-) That's exactly what they do on systems where rebooting through the firmware doesn't preserve RAM.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Oh well, can't have a freshmeat db in my head I guess ;) That said, I like this approach since you don't need to care about new drivers and so on ... but since it's already out there I guess it's advantages have been covered elsewhere ...

Karim

===================================================
Karim Yaghmour
[email protected]
Embedded and Real-Time Linux Expert
===================================================

I dealt with crash dumps quite a lot over 10 years with SCO UNIX, OpenServer and UnixWare: which were addressing the PC market, not own hardware.

It's a real worry that writing a crash dump to disk might stomp in the wrong place, but I don't recall it ever happening in practice. But occasionally, yes, a dump was not generated at all, or not completed.

Of course, you could argue that SCO's disk drivers were more stable :-) which might or might not be a compliment to them.

Hugh

IIRC, some years ago wuarchive.wustl.edu went down for a few days because the machine paniced and dumped to the wrong partition...

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Hugh Dickins <[email protected]> said:

[...]

What I remember about hardware compatibility for SCO Unix and Solaris on ia32 is _not_ funny. Lightyears from what Linux handles today without breaking a sweat.

How do you test that? Not in some contrieved situation, under real crashes. Don't just consider crashes in the official $DISTRIBUTION kernel, but in Linus' BK tree, or some of the random, two-or-three-letter-trees of the day (_that_ is where crashes happen, _that_ is where the info would be most valuable). It gets _real_ hairy _real_ fast to make sure you don't scribble over /home or /etc on the user's disk...

If you only handle a few, thoroughly tested, high-end controllers and disks, that is not too hard to do.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513

Sorry for being unclear: by "in practice" I meant "under real crashes" i.e. I was referring more to what we heard back from users than my own testing.

Hugh

Quite clearly SCO, Sun, and IBM have been doing this for years without offering dozens of options. I don't need it to sing and dance, I just need a way to put the dump where I can find it. I'm not going to put another box in at the end of a serial or parallel port, I don't have NVram, I do have lopts of disk, and so does almost everyone else. I have remote systems in wiring closets all over the country (all four time zones). They are at the end of open net connections, unreliable and untrusted. I don't want to bet that I have a working VPN, or that I can safely send all that data without it being read by someone other than me.

The AIX support has a group just to beat on dumps customers send. What more evidence is needed that people can and do use the capability.

I had hoped that someone would do this for Linux, I never dreamed that it would be kept out of the kernel by people who clearly don't understand the problems if distributed and clustered headless systems.

I guess the development folks are working on more important things like xiafs and morse code dumps to the keyboard LEDs.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

You paid someone for this for AIX. So the solution is obvious for Linux.

No, it's included in AIX, SCO and Solaris. And analysis is included in support contracts. With all the stuff added to Linux to keep up with both M$ and commercial UNIX, I can't imagine why anyone would be against this. At least anyone who wanted Linux to compete in the commercial server market.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

None of those are free.

So buy your Linux from a vendor who supports it.

AIX has 4 people doing dumps in Austin (otherwise known as ZTRANS). There are others in other countries.
The folks from other countries were brought to Austin for training (usually for 3 months).
There is usually one person in L3 doing dumps in Austin for service, although every subsystem has someone that specializes in reading dumps for that subsystem.

The first 4 people only do a scan of the dump to see if it's a known problem. If it's not
a known problem AND it's in AIX code it goes to whoever it is that owns that subsystem.

Dumps are only the beginning with AIX. Trace hooks along with dumps are VERY
useful.
The trace hooks are also what the performance people use.

The Linux kernel is _extremely_ painful to debug compared to AIX.

--
Jennie Haywood
[email protected]
Everyone is crazy. It's just a matter of degree. [email protected]
-
The oak tree in your backyard is just a nut that held its ground.

Good! This means, people debugging the code have actually to think and don't produce "turn on debugger, step here, there, patch a band aid, done" solutions you see with various other "commercial products" (can anyone really say "Internet Explorer" on this list and live? ;-) )

Regards
Henning

Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     [email protected]

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   [email protected]
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Some of us debug hardware. Regardless of the nice theories about reviewing your code they don't actually work on hardware because no amount of code review will let you discover things like undocumented 2uS deskew delays, or errors in DMA engines

A debugger won't help you here either. A pci bus probe, a 'scope and a logic analyzer do.

(And experience, elbow grease, experience and a nice amount of ESP :-) I do hate hardware. Had to debug too much of it (and just on m68k/MCS-51 where the clock rates are low and the parts easy to solder...).

Regards
Henning

Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     [email protected]

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   [email protected]
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

I find that hard to believe. You're saying it's impossible to use a software debugger to debug the interface between the software and the hardware? Eg. errors in the hardware that cause periodic anomalies in the output read by the software would be one thing they could catch, along with diagnosing that a problem is caused by flaky hardware rather than the latest not-well-tested VM code. In that last case, since bad hardware can usually cause a panic, I see crash dumps as an invaluable resource ;-). (No Linus, I'm not pushing them, just stating my opinion.)

Pat
--
Purdue Universtiy ITAP/RCS
Information Technology at Purdue
Research Computing and Storage
http://www-rcd.cc.purdue.edu

http://dilbert.com/comics/dilbert/archive/images/dilbert2040637020924.gif

No. IMHO it is impossible to use a software debugger to catch 2uS deskew delays or errors in DMA engines. That's what logic analyzers are for. If you attach or fire up the debugger, the timing changes and you're no longer testing the failure case but something different.

I am, BTW completely your opinion. Personally I find it horrid that "the XIAFS resurrection" is winked through with "will be probably accepted for the hack value" and LKCD is rejected with "bloat" arguments.

But hey, it _is_ Linus' kernel and he may choose as he likes. I e.g. run vendor kernels (for 2.4).

Regards
Henning

Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     [email protected]

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   [email protected]
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Unfortunately, just making it hard doesn't guarantee that they won't try anyway. If you're lucky, at least their band aid will be so disgusting that you won't be fooled into thinking they might be right.

But ultimately, it's an attitude problem. Even people who learn about their bugs by source code reading may then produce a shabby fix.

Hmm, I wonder if Linus has ever done any protocol design, followed by validation. I always find the havoc a protocol validator (e.g. Spin) wreaks a very instructive demonstration of how much source code level "correctness" really buys you :-) (Or what chances you'd stand of realizing what happened just from an Oops.)

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Netdump drives the system itself. Any dump driver has to as it cant assume the system is in a remotely sane state

In message <[email protected]> you write:

Both the lkcd and ide mini-oopser have that (although the mini-oopser has only x86-ide for now).

The mini-oopser has different aims than LCKD: they want to debug one system, I want to make sure we're reaping OOPS reports from those 99% of desktop users who run X and simply reboot when their machine crashes once a month.

I did *not* put the mini-oopser on the Snowball list, because I don't have time to polish it.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

On Fri, 1 Nov 2002, Rusty Russell wrote:

|>The mini-oopser has different aims than LCKD: they want to debug one
|>system, I want to make sure we're reaping OOPS reports from those 99%
|>of desktop users who run X and simply reboot when their machine
|>crashes once a month.

I'd like to incorporate the mini-oopser as an LKCD dump method. I'll chat with you off-line about this. Shouldn't be that difficult to do.

|>I did *not* put the mini-oopser on the Snowball list, because I don't
|>have time to polish it.
|>
|>Rusty.

Thanks,

--Matt

In message <[email protected]> you write:

That would defeat the "mini" part 8)

Cheers,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

We'll be spending the next six months stabilising and hardening the used-to-be-2.5 kernel. If grunts like me can get hold a copy of the other person's kernel image from time-of-crash, that has a ton of value.

(Disclaimer: I've never used lkcd. I'm assuming that it's possible to gdb around in a dump)

It could help. But like serial console, the random person whose kernel just died often can't be bothered setting it up, or simply doesn't have the gear, or the crash is not repeatable.

So. _If_ lkcd gives me gdb-able images from time-of-crash, I'd like it please. And I'm the grunt who spent nearly two years doing not much else apart from working 2.3/2.4 oops reports.

Oh, and as Rusty has pointed out, we lose a *lot* of oops reports because users are in X and the backtrace doesn't make it to the logs. Rusty has a little app which dumps just the oops report to
disk somewhere. Want that too.

Exactly, sometimes you don't even need the dump itself, The user who has the dump just types lcrash and report -w file.txt and lcrash writes a consolidated report with the most interesting information from the dump to the file.txt and he can sent it to you and you've much information you often miss in problem reports.

The report consists of: time when the dump was created, time when the report was created, the architecture, the hostname, kernel version and compile time, the kernel (dmesg) buffer with all the oopses logged into it, a short task list with process adress, id's, state, flags, cpu and process name, and finally a full CPU dump of every CPU with all registers, current process and function and a symbolic stack backtrace of the CPU.

Sometimes this is all you need to know and if you need to know e.g. the stack backtrace of a not running process at the time of the dump, you can ask the user to simply run trace <process address> and lcrash gives you the backtrace of this process:

lcrash> t[race] 0x1408000
================================================================
STACK TRACE FOR TASK: 0x1408000 (kjournald)

STACK:
0 schedule+894 [0x3164e]
1 interruptible_sleep_on+174 [0x31eae]
2 journal_revoke+<ERROR> [0x10889c0c]
3 kernel_thread+70 [0x18c1e]

showing the full task scruct, a sub-struct or a field is also simple:

p[rint] ((struct task_struct *)0x1408000)->pending

struct sigpending {
        head = (nil)
        tail = 0x1598700
        signal = sigset_t {
                sig = {
                        [0] 0
                }
        }
}

"feels" a bit like gdb

I don't know if there is an lkcd->ELF core converter yet, but it should be doable. However, lcrash is quite powerful, it comes with sial, an integrated C interpreter that permits easy access to the symbol and type information, obviosly, it allows to write code like this:

        void
        showprocs()
        {
        	struct proc* p;
                for(p=*(struct proc**)procs; p; p=p->p_next)
                        do something...
                }
        }

Looks nice... :-)

I also don't know if (k)gdb knows about tasks, lcrash at least knows about them and this may when you look into a specific task(I'm not an expert)

Of cource lcrash can do dissembing, find symbols,

Maybe the lkcd people can do so, but I think they can also give a hands-on workshop to lcrash.

You can use lcrash also on running system to browse around, learn and save dumps from it without interrupting it, you just need lcrash, the System.map and the Kerntypes file from kernel for using it.

Yep, I think it would be good even if Linus just accepts the infrastructure patch of lkcd which needs to be in the kernel, the vafourite dump method module can then be downloaded, compiled installed and configured without much pain, I think that people can start using it in a broader range without the hassle of needing to patching and booting a special kernel.

Bernd

PS: lcrash is only one of the many frontends, as I've read in this thread, there is also Dave Anderson's "crash" tool which works with LKCD dumps, netdump dumps, etc. There is also qlcrash, an qt frontend for lcrash for people who like to click!

I updated Dave Anderson's (Mission Critical) crash code to work with LKCD core dumps when I updated LKCD to support the ia64. Dave's crash code uses gdb as a command interpreter. It's not quite as flexible as using gdb macros on core dumps but it's very close and has lots of support for various kernel structures. For example, you can't just have ddd walk through data structures by simply clicking on pointers in data structures like you normally can.

Yes, ideally I'd like to have an integration between live gdb stub debugging and crash debugging. I'd like to even be able to use ddd/gdb on a core file and simulate execution. When using gdb on the kernel I've found it nice to move the cursor over the PC and move it to the end of panic(). Then single step back out of panic and re-execute the code that returned the error code that caused us to decide to panic. Doing this in asm language with a asm debugger is too difficult for most folks.

I really liked HP's kwdb approach. kwdb has a tiny TCP/IP stack and has direct hooks into the trap vectors like a normal kgdb stub. The nice thing is you can attach to a crash system over the internet from anywhere in the world to debug the panic. I wasn't able to get HP to release the kwdb gdb stub into the public domain. The gdb hacks are available at:

http://h21007.www2.hp.com/dspp/tech/tech_TechSoftwareDetailPage_IDX/1,1703,257,00.html

but are based on a very old version of gdb and ia64 libraries.

You can snarf a copy from:

ftp://people.redhat.com/anderson

One area that I'm not sure of is if the lkcd kernel changes are a problem with the kgdb patch (http://kgdb.sourceforge.net/). Perhaps I can check into that in the near future.

I'd prefer to have both kgdb (http://kgdb.sourceforge.net/)
remote debugging and kgdb crash support available in stock kernels like the BSD kernels (NetBSD, FreeBSD). I don't know why the kgdb stub wasn't integrated into the kernel for the ia32 and ia64 platforms. I suppose for reasons like we are hearing now on the LKCD kernel hooks. The current LKCD code is at least a step in that direction.

[email protected]

As in 'sometimes not even when hundreds of people start haranguing me about it in public forums'?

Perhaps not the best analogy.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

Heh. I like my analogies bad. The best analogies should make you go "huh!" - kind of like a pink poodle in a tutu.

Linus

And guess what ? Netdump is one of various LKCD dump methods ...

regards,

Rik

A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

[yes, I realize the LKCD merge debate is over, bear with me :)]

I'm sort of in the middle on this issue: The existence of netdump does not imply that disk dumps are a bad thing.

netdumps require a net dump server, and it is simply not realistic at all to assume that users seeing crashes will always have a netdump server set up in advance, or even have multiple machines to make that possible. Disk dumps are valuable because their requirements are very low, and because of all the user-support reasons that Andrew Morton mentioned in this thread.

That said, I used to be an LKCD cheerleader until a couple people made some good points to me: it is not nearly low-level enough to truly be of use in crash situations. netdump can work if your interrupts are hosed/screaming, and various mid-layers are dying. For LKCD to be of any use, it needs to _skip_ the block layer and talk directly to low-level drivers.

So, I think the stock kernel does need some form of disk dumping, regardless of any presence/absence of netdump. But LKCD isn't there yet...

Jeff

I'm not so convinced about this. I like the Mission Critical
approach: save the dump to memory, then either boot through the firmware or through bootimg (nowadays, that would be kexec), then retrieve the dump from memory, and do whatever you like with it.

The huge advantage here is that you don't need a ton of specialized dump drivers and/or have much of the original kernel infrastructure to be in a usable state. The rebooted system will typically be stable enough to offer the full range of utilities, including up to date drivers for all possible devices, so you can safely write to disk, scp all the mess to your support critter, or post an automatic flame to linux-kernel :-)

The weak points of the Mission Critical design are that early memory allocation in the kernel needs to be tightly controlled, that architectures that wipe CPU caches on reboot need to commit them to memory before the firmware restart, and that drivers need to be able to recover from an "unclean" hardware state. (I think we'll see much of the latter happen as kexec advances. The other two issues aren't really special.)

Actually, at the RAS BOF I thought that IBM were developing LKCD in this direction, and had also eliminated a few not so elegant choices of Mission Critical's original design. I haven't looked at the LKCD code, but the descriptions sound as if all the special-case cruft seems to be back again, which I would find a little disappointing.

There might be a case for specialized low-overhead dump handlers for small embedded systems and such, but they're probably better maintained outside of the mainstream kernel. (They're more like firmware anyway.)

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Yes, we are putting that in as one of the alternative dump targets available. I have done quite a bit of work on that implementing the ideas we talked about at OLS, and that's what I've been referring to as the memory dump target. Its not quite ready yet and we need something like kexec to be available which we can use on Intel systems to achieve the softboot (the acceptance status of that still doesn't seem to be clear), so I was looking at this as a follow-on thing once the core infrastructure is there. More so because we probably need to give it some time to stabilize and try it on different environments and look at the issues you mention.

Why do we even consider the other options when we are doing this already ? Well, as we discussed earlier there's non-disruptive dumps for one, where this wouldn't work. The other is that before overwriting memory we need to be able to stop all activity in the system for certain (system may appear hung/locked up) and I'm not fully certain about how to do this for all environments. Maybe an answer lies in rethinking some parts of the algorithm a bit.

Also having the interface allows people to develop more specific/ reliable solutions for their environment. So we do not necessiate code duplication, but if something exists, then the infrastructure can use it.

The general feeling here is that a one solution fits all thing may not work best right now ... and hence the focus on an interface based approach that gives us the needed flexibility.

Hope that helps a bit.

Regards
Suparna

Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India

Great !

Yes, I've just checked with Eric, and he hasn't received any indication from Linus so far. I posted a reminder to linux-kernel. I'd really hate to see kexec miss 2.6.

But they're very different anyway, aren't they ? I mean, you could even implement them (well, almost) from user space, with today's kernels.

This is certainly the hairiest part, yes. I think we have about four types of devices/elements to worry about:

- those that just sit there, and never talk unless spoken to
- those that may generate interrupts
- those that DMA if you ask them nicely
- those that DMA when they feel like it (e.g. copy an incoming network packet to the next buffer in the free list)

The latter are the real problem. I see the following possibilities for dealing with them:

- faith-based computing: pray that nothing bad will befall your system :-)
- de-activate them individually. There should be a lot of work that can be shared with power management. And that's one of the reasons why I think the memory target should be available early, or convergence will take forever.
- try to reset them, without necessarily knowing what they are or what they do. I don't know is there is a useful way for resetting the PCI bus by software, but if there is one, this looks like the most promising strategy to me, even if it may be somethat lacking in elegance.
- if all else fails, maybe introduce an "unsafe" memory type for potential DMA targets of unpredictable devices, that will not be re-used. I hope we won't need this, though. (But in case such a memory type gets introduced by the memory-scrubbers, at least you could blame _them_ :-)

Yes, this is certainly important. I just think that the "memory target" concept is closer to a general solution than disk dumps. But there are always those 5% with special needs, and it's good if they can use the same framework.

Thanks,
- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Let me ask the same dumb question - what does kexec need that a dumper doesn't. In other words given reboot/trap hooks can kexec happily live as a standalone module ?

kexec needs:
- a system call to set it up
- a way to silence devices (difference to dumper: kexec normally operates under an intact system, so it's more similar to, say, swsusp. But I expect that cleaning up device power management would also clear the path for more reliable dumpers.)
- a bit of glue, e.g. to switch to "real mode", etc. AFAIK, none of this touches other code, but there are of course some assumptions here on what other codes provides or does.
- device drivers that can bring silent devices back to life (normally, device drivers do this already, but kexec may uncover dormant bugs in this area)

Since recent kernels already preserve memory areas with BIOS data, kexec is actually quite a bit less intrusive than bootimg was.

"Module", as in "software package": yes, the main problem spot would be the system call allocation, which is also inconvenient for other developers. By the way, kexec does not tap into the kernel's reboot process, so no such hooks are needed. If LKCD wants to use kexec, it can do whatever it does to be invoked at the time of a crash, and then call kexec directly.

"Module", as in "loadable kernel module": I think so, although it's currently "bool", not "tristate". Also, you'd have the system call issue again.

So not merging it is mainly inconvenient to use, adds the system call allocation as a continuous annoyance, and makes it a little harder to work on the related infrastructure. But then, despite being somewhat obscure, bootimg and kexec have been in use for years, the design is about as lean as it can get, and it's cool. What more could you ask for ? :-)

What kexec needs now is more exposure, so that the BIOS compatibility issues get noticed and fixed, it is ported to other architectures, and that more people can start figuring out how to use it, and how to build a boot environment. Then, maybe in a year or two, we can send "Methuselah" LILO and "nice little OS" GRUB off to their well-deserved retirement.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

Device, file, insmod...

So you need to register with the power management as the last thing to be suspended and do a suspend before kexec.

I'm mostly worried about how to make these things fit the least intrusively into the kernel.

I don't know what Eric thinks about using something else than a system call, but I think he made a quite reasonable choice.

The data structure isn't entirely trivial, so a misc device plus ioctl would be a bit on the ugly side. I vaguely remember having proposed something like this a while ago (may have been for pivot_root), and everybody went "noooo!!" ;-)

insmod would be possible, although with a rather unusual parameter passing scheme. Also, when using kexec from inside the kernel (e.g. MCORE), the insmod solution would have to split kexec into the interface and the kexec core.

But yes, there's always a means to avoid adding a new system call. /dev/syscall with an ioctl

struct syscall_ioctl {
	const char *symbol_name;
	va_list ap;
};

anyone ? :-) (Implementing it might be a bit of a challenge :)

Well, kexec just calls device_shutdown. The problem isn't the interface, it's that device_shutdown apparently doesn't work too well (devices not supporting it, some semantics mixup, etc.). But this is general infrastructure work, that should be done with or without kexec.

Just look at Eric's kexec patch. It isn't particularly intrusive:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103604471723358&w=2

(For 2.5.45. The patch fails for 2.5.46, because new system calls were added ...)

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

<snip>

<snip>

You could probably skip the system call to set it up. Example: I could imagine a bizarre set of pseudo-devices:

	# insmod kexec
	# cat bzImage > /proc/kexec/next-image
	# echo "root=805" > /proc/kexec/next-cmndline
	# echo 1 > /proc/kexec/reboot

and hide away that dirty little sequence with a nice kexec(3) library routine.

The Two Kernel Monte trick (that rewrote when insmod'ed the kernel's function pointers for sys_reboot) was also effective, but that apparently isn't an option any longer.

I'll 2nd that sentiment, and add another big one: fixing (apparent) problems with drivers and chipset-munging code, so that devices can be reliably re-probed/re-inited/etc. after the reboot.

Long term, I think it would be advantageous to be able to avoid SCSI and other time consuming device probes for the common and simple reboot case of 1) the currently running kernel is being rebooted, and 2) no changes to the device configuration have occured. Shouldn't we be able to "save away" what is in sysfs, and then re-inject that state after a fast reboot?

Andy

Yes, yes, there are many ways to do this. This isn't the issue. The questions regarding this are:

- it kexec allowed to use a system call ?
- if yes, is a system call the technically right solution ?
- if yes, is it a practical solution ?

So far, it hasn't been considered inherently wrong to use system calls, even for highly Linux-specific functions, and even if they aren't performance-critical (just think of pivot_root). (*)

If this perception has changed, such a change of policy would also affect kexec, but then we don't need to discuss kexec but the policy change. (I don't know - is such a change in the air ?)

(*) By the way, I remember now where I brought up some hack for
avoiding to use a system call - it was for bootimg :-)

Now, if we assume that it's okay for kexec to use a system call, the next question is whether kexec should indeed use it, i.e. whether a system call makes sense for what it is trying to do. Since there are no device files or network elements naturally involved here (i.e. other major kernel function interfaces), the answer seems to be "yes".

Last but not least, we need to decide whether using a system call would be painful for Eric or for kexec users. This would be the case if kexec isn't merged, and the kexec patch would need frequent updates because system calls have changed.

I understand Alan's question as the "what if ... ?" type. If kexec is indeed rejected for merging, it may make sense to change the interface to something which may be technically less elegant, but which makes patch maintenance easier to handle.

Yes, kexec is likely to turn up a few problems in this area, too. Right now, we only hear about such issues if some BIOS lets something slip through. With kexec, such problems should show up sooner.

Definitely. May I refer you to my booting paper, which discusses all this in section 5 ? :-)

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

That's not obvious. By the same logics, we would need syscalls for turning off overcommit, etc., etc.

FWIW, I suspect that

	open("/proc/image", O_EXCL|O_WRONLY);
	bunch of lseek()/write()
	close()

would be more natural - definitely easier to understand than arguments of your sys_kexec(). It's easy to switch from your code to that - you put initialization into ->open(), pulling segments from userland into ->write(), use default ->lseek() and do actual work on ->close() if no errors had happened. file->private_data will point to intermediate state you need.

After all, that's what happens - you form an image, writing to it user-supplied data from given buffers at given offsets and when you are done with that you commit the changes. IMO special syscall is less natural match for that than sequence above - commit-on-close is not something unusual, so it matches the semantics of all syscalls involved...

Okay okay, add file system specific ioctls and sysctl to my list of alternative mechanisms :-)

Hmm, interesting. Yes, that should work. One would of course have to retain the current interface for in-kernel use (e.g. MCORE), but that's probably okay. Let's see what Eric thinks about it - it's his code after all.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

For the record my opinion is there is extra code bloat but it is ok if it is built as kexecfs. Any other way of getting a magic file to work with seems currently insane.

Eric

Eric W. Biederman wrote:
[ Al's FS-based kexec interface ]

Yes, such an interface change would only make sense if you couldn't get the system call, or if there would actually be a useful way for setting up kexec using "third party" programs. But it seems unlikely to me that somebody could get all the magic right just by using dd.

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

First take a look at a ELF header. There is a one to one mapping between the arguments to kexec and the segments found there.

Second lseek()/write() pairs do not have the capacity to specify holes/bss segments kexec does, so it would not be a 1 to 1 transform. But I can live without holes.

Third I am not fully certain it makes sense to implement a function that will boot into a user specified image remotely. If the export process has too many capabilities we could be in trouble.

Are you arguing for more /proc files? Where does the magic file come
from? I cannot request the allocation of a device number because the
allocation was frozen before 2.4 started. Though char 1 minor 11
seems the obvious choice. Or should it be a magic file in sysfs
instead of procfs? All of the require the code to live someplace where I need to allocate a place in the namespace. So there is no inherent advantage over a system call. And unless someone exports the hooks to properly shutdown the system to modules it is useless.

Given that this is a seldom used system function I agree that it does not need to be optimized.

I do not have any problem with changing the interface to something more palatable to other kernel developers. But I will only do it for one of two reasons. My patch will never get accepted in any reasonable time frame and it makes maintenance easier for me. Or makes the interface palatable for acceptance, into the kernel. Neither position currently appears apparent.

----------
Now to dig into the heart of the issue.

I could write the new kernel image into /dev/mem and just jump to it. Because that is really all I want an interface to do. There are several practical problems, with something quite that simple.

No kernel shutdown code is run, so I am left with processors flying all over the place, devices doing all manner of things, after their device drivers have walked away. Something needs to put the system in a quiescent state. The fix I call the reboot notifiers, and device_shutdown. (And then implement a bunch of ->shutdown() methods)

As we all know writing to /dev/mem is not safe because the memory is being use for other things. So I need a way to safely use memory during the transition, from one kernel to another.

Personally I would love to be able to allocate one big contiguous buffer that the kernel is not using and neither is the image I will eventually load. Then I could just memcpy from that buffer and I would be done.

Alas memory management in the kernel is done in pages, and can be fragmented after running for many moons. So I need to allocate all of my memory in pages, and I need to let the kernel know where it will all eventually live so I can correctly order the memcpy operations.

Once all the memory copying is sorted out I need to jump to the new kernel (a kernel being anything that runs without an OS). Logically all you should have to do is do a single jump instruction but in practice there is much more that has to be done. The kernel when it loads up looks around and enables all sorts of cpu optimizations so the kernel runs as well as possible on the processor. The new kernel image needs to be given a least common denominator interface so it can
enable what it is prepared to take advantage of. In addition to what
the normal shutdown path can accomplish on x86 this involves disabling page, changing the gdt, and changing the idt, and possibly disabling SMP. It should be possible to enhance device_shutdown so it can properly disable SMP though if that will happen still remains in the air.

-----------------------------------------

So kexec needs:

- An allocated slot in some namespace.
- The ability to request the kernel devices shut themselves down.
- Buffers that are safe to use.
- The ability to transition the cpu into a state that is suitable for jumping to another kernel.
- Awareness of it's existence.

To some extent every piece of this is intimately tied to the kernel implementation, from the ability to modify page tables, when jumping to a new kernel, to the best algorithm for finding a safe memory buffer, to the proper way to shutdown devices this week, and being intimately tied to the kernel the code needs to find a home in the kernel.

Eric

And the question I was building up to, but forgot to ask.

Given that the kexec code is tied intimately to the kernel implementation.

Given that there is no real advantage in an incremental write model for kexec users (except not needing to allocate a syscall number).

Do you see a better way to structure the kexec interface?

Another file in proc, not carefully placed is just a hair better than an ioctl. Using /proc is not desirable because there are uses of kexec that need a very small kernel, and /proc is a pig, is otherwise useless size bloat.

For some uses including the one that drove me to write it CONFIG_KEXEC and CONFIG_TINY will both be defined.

Eric

Reverse Mappings are cool, and one reason tehy're cool is, in theory, you can grab a page of physical memory away from something else. In theory code could be written to ask the kernel "could you please swap this the heck out, pin the page in memory, and give it to me instead now?" And it can refuse ("it's already pinned by something else, maybe it's a kernel page, go away"), it can block a bit ("gotta flush it to disk, wait until DMA is done"), or it could immediatley comply ("it was a clean buffer, have it, keep it, stuff it and mount it on the wall for all I care...").

This means you can retroactively get contiguous areas of memory by shoving stuff aside. If it's in use, it'll swap back in immediately. (An obvious optimization occurs, but that's not necessary for minimal functionality.)

So the the whole problem of needing contiguous areas of memory could, in theory, be addressed using RMAP.

http://penguicon.sf.net - Terry Pratchett, Eric Raymond, Pete Abrams, Illiad, 
CmdrTaco, liquid nitrogen ice cream, and caffienated jello.  Well why not?

Hi!

Actually, sys_reboot has void * parameter. Reusing it as "next-image" char * seems okay to me.
Pavel

Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?

Kexec primarily needs the reboot/trap hooks in working order, and exported, for it to live externally to the kernel.

Currently the reboot_notifier call chain is private to sys.c, and is not exported even to other parts of the kernel.

Even together device_shutdown, and the reboot_notifier do not properly shutdown the cpus on an SMP system.

Plus we are missing quite a ->shutdown methods at random in the kernel, and if kexec is not easily available someone might not get around to writing and debugging them.

Plus a system call seems the natural interface for something that appears to be a reboot.

Eric

In replying to another post by Al Viro I managed to think this through. kexec needs:

- An allocated slot in some namespace.
- The ability to request the kernel devices shut themselves down.
- Buffers that are safe to use.
- The ability to transition the cpu into a state that is suitable for jumping to another kernel.
- Awareness of it's existence.

Most of this code is intimate with how the kernel currently behaves and needs at least minor adjustments for things like living in PAE mode.

The safe buffers a kernel can probably avoid.

I cannot see the core functionality of kexec every living happily as a standalone module. The kexec code accomplishes nothing. If there is something useful it does it can probably be moved elsewhere and the line count reduced. The entire code base is basically obsessed with getting safe temporary buffers for the new kernel to live in, and given improvements to how the kernel allocates memory that are theoretically possible with rmap I could remove that code as well.

The only thing that keeps kexec at all maintainable outside the kernel is that big fundamental changes do not happen often. But the kernel must be tracked, closely. I don't see that as a recipe for a standalone module. I can barely even see making the code a module, or what the point would be.

The reason kmonte fails in so many cases where kexec succeeds is precisely because kmonte is a module.

If we include machine_kexec or something very similar to but more generalized to the list of exported functions, perhaps kexec could just have the buffer allocation code and live happily outside of the kernel. But as it is, if we want to factor kexec into pieces so one piece can live happily as a standalone module it will take some serious design work, and a total rethink of the implementation. And we will still have to add code to the kernel.

Eric

Note that kexec doesn't bother me at all, and I might find myself using it myself.

though: clearly "sys_reboot()" is the place to add a case for "reboot into this image". No? That's where we shut down devices anyway, and it's the sane place to say "reboot into the kexec image"

Which still leaves you with a real sys_kexec() to actually _load_ the image, or course. I think loading of the image should be a totally separate event from the actual booting of the image, since we may want to load the image early, then do various user-level shutdown (unmounting etc), and then reboot.

Right now the kexec() stuff seems to mix up the loading and rebooting, but I didn't take a very deep look, maybe I'm wrong.

Anyway, I don't really get why the kexec() system call would not just be

	void *kexec_image = NULL;
	unsigned long kexec_size;

	int sys_kexec(void *uaddr, size_t len)
	{
		void *new;

		if (!capable(CAP_ADMIN))
			return -EPERM;

		/* Get rid of old image if any.. */
		if (kexec_image) {
			vfree(kexec_image);
			kexec_image = NULL;
		}

		/* Zero length just meant "get rid of it" */
		if (!len)
			return 0;

		if (!access_ok(VERIFY_READ, uaddr, len))
			return -EFAULT;

		new = vmalloc(len);
		if (!new)
			return -ENOMEM;

		if (memcpy_from_user(new, uaddr, len)) {
			vfree(new);
			return -EFAULT;
		}

		kexec_image = new;
		kexec_size = len;
		return 0;
	}

and be done with it that way? Then the actual "reboot" (and that would be in the existing "sys_reboot()") basically just does something like

	memcpy(kernelbase, kexec_image, kexec_size);

at the very end (while obviously having to be careful about itself being out of the way. It can avoid the page table issue by using the "page *" array that vmalloc uses internally anyway: see "area->pages[]" in vmalloc).

Note that the two-phase boot means that you can load the new kernel early, which allows you to later on use it for oops handling (it's a bit late to try to set up the kernel to be loaded at that time ;)

Linus

Yes, that's exactly what we need to support a soft-boot based dump mechanism, much like the Mission Critical folks split up the bootimg syscall to do the early load on a sane system, and the actual soft-boot at crash time. And it fits in naturally as you point out ..

Regards
Suparna

Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India

Good. Just before I saw this message I sent you my patch ported to 2.5.46, and from the feed back on this one it looks like people would appreciate a tweak or two.

That sounds reasonable to me. Especially as that lines up a little more with what the mcore people want as well. Until today I hadn't realized they were using a spare current to process oopses. For just booting another kernel all of the staging can currently be done by reading the new kernel into your process before calling the user-level shutdown code.

It currently happens all in one step because I had never gotten feedback that people wanted it in two steps.

Given that it is definitely a good idea to split the patch up into two pieces. And a kernel for oops handling should work once all of other problems are resolved.

Using area->pages[] is an interesting idea.

1) The address or addresses to load the new kernel at. (Think kernel + ramdisk)
2) The address to jump to start the new kernel.
3) My interesting buffer handling.

The question is how much of that do we need.

Thinking out loud, and hopefully answering your question.
- We need a working stack when the new kernel is jumped to so PIC code can exist at the entry point.

- An oops processing kernel needs to load at an address other than 1MB, or at the very least it's boot sequence needs to squirrel away the old contents of the kernel text and data segments, which reside at 1MB, before it moves down to 1MB.

- When we transfer control to the trampoline in machine_kexec we need to be able to refer to everything with physical addresses.

- I do not see a way out of running my buffer verifier algorithm. The problem is that I do not want to put complex logic in the assembly machine_kexec trampoline. So I want to be able to pass it something it can just memcpy to it's final resting place. Which means the buffer pages either need to be the final resting place of the new kernel (ideal) or are not a page that of the final resting place.

- I can dig up area->pages[] but I don't see vmalloc buying me anything. Doing the copies and allocations a page at a time is not
hard. I have to sort the contents of the pages, and where they
are located so I need to undo the virtual mapping.
area ->pages is all by struct pages *, which is most inconvenient when you are tearing down page tables, I would need to put the pages into another data structure that just had the page frame number or physical page address anyway.

- Once I am using my own data structure to track the pages, and I am already vetting the pages for safe locations. Going the rest of the way to my current interface is not a big step, and I have already tested that code.

So either I have blinders on, or there is very little percentage in changing how I load an image. But to make the oops processing easier I will split it up into two parts.

Then I guess the reasonable thing to do is to modify sys_reboot to call machine_kexec instead of machine_restart when a kexec_image is present. Or should I add another magic number, and another case to sys_reboot?

	case LINUX_REBOOT_CMD_RESTART:
		notifier_call_chain(&reboot_notifier_list, SYS_RESTART, NULL);
		system_running = 0;
		device_shutdown();
		printk(KERN_EMERG "Restarting system.\n");
+		if (kexec_image)
+			machine_kexec(kexec_image);
		machine_restart(NULL);
		break;

O.k. In the next couple of days I will split the loading, and executing phase of my kexec code into parts, and resubmit the code. The we can dig in on what it takes to make kexec run stably.

Eric

I'd mentioned it a few times in the context of mcore, but probably didn't explain myself clearly enough then.

Yes, this fits the model we need.

Yes, that bit of memory save logic exists in the mcore mechanism. These pages are saved away in compressed form in memory and written out later after dump.

Now to avoid these pages from being used by the new kernel until the dump is safetly written out to disk, mcore patches some of the initialization code to mark these pages (containing saved dump) as reserved.

Great !

Regards
Suparna

Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India

Given that "bird-eye" description why not to make a "normal" restart a particular case of kexec where you just have one kernel loaded from an external storage? It does not seem to be that much different although some issues are skipped or taken for granted. Or I am talking nonsense?

Michal

When kexec is separated into two pieces I agree. As I had it
initially in one step it does not look at all like reboot. Now I
just need to think up a new magic number for sys_reboot.

[snip wonderful vision of the theoretical simplicity of sys_kexec].

In case I was not sufficiently clear last night. It could be as simple as your example code if I replaced vmalloc by __get_free_pages/alloc_pages, and allocated a large contiguous area of ram. But MAX_ORDER limits me to 8MB images, and allocating an 8MB chunk is unreliable, and even a 2MB chunk is dangerous.

So I must use some form of scatter/gather list of pages, like area ->pages[] to make it work. Things get tricky because I gather (via memcpy) the pages at a location that potentially overlaps the source pages. So I must walk through the list of pages making certain I when I gather (memcpy) the buffer pages into their final location I will not stomp on a buffer page I have not come to yet. Correctly doing that untangling is where the complexity in kernel/kexec.c comes from.

Eric

I am now officially grumpy. From a code perspective splitting kexec into two phases load, and execute is a simple change to make. From a semantics standpoint things get ugly, and messy. And that means I can't just dash off another patch.

There are currently 2 cases that it would be nice to have work.
1) Load a new kernel and immediately execute it.
2) Load a new kernel and execute it on panic.

At first glance splitting the code into a load and execute phases allows us to use one mechanism to accomplish both goals. In practice that does not work. There are 2 problems.

panic does not call sys_reboot it rolls that functionality by hand. And to a certain extent it makes sense for panic to take a different path because we know something is badly wrong so we need to be extra careful.

In staging the image we allocate a whole pile of pages, and keep them locked in place. Waiting for years potentially until the machine
reboots or panics. This memory is not accounted for anywhere so no
one can see that we have it allocated, which makes debugging hard. Additionally in locking up megabytes for a long period of time we create unsolvable fragmentation issues for the mm layer to deal with.

In a unified design I can buffer the image in the anonymous pages of a user space process just as well as I can in locked down kernel memory. So factoring sys_kexec in to load and execute pieces only helps for executing the new image on a kernel panic, and that case does not actually work.

So currently factoring kexec looks like a pointless exercise, that will just lead to more pain.

I am left with the following questions.
- How should the pages allocated to an early loaded image be accounted for?
- How do we avoid making life hard for the mm system with an early loaded image?
- Is it safe to call sys_reboot from panic?
- Can we simply factor out the sequence:

		notifier_call_chain(&reboot_notifier_list, SYS_RESTART, NULL);
		system_running = 0;
		device_shutdown();

And place it into it's own subroutine?
- What does the current mcore implementation do? And is that a good model to follow to resolve some of these issues?

Eric

I really don't think (1) is _ever_ a valid thing to do.

The fact is, loading a new kernel wants filesystems and a fully working system. While executing it wants the filesystems quiescent.

Forget about panic for now. It's a design issue - it should be possible to work, but somebody else can do it if the infrastructure is done right.

And in a unified design, I won't apply the patches. It's that simple.

Linus

There are two cases I am seeing users wanting.
1) Load a new kernel on panic.
- Extra care must be taken so what broke the first kernel does not break this one, and so that the shards of the old kernel do not break it.
- Care must be taken so that loading the second kernel does not erase valuable data that is desirable to place in a crash dump.
- This kernel cannot live at the same address as the old one, (at least not initially).

2) Load a new kernel under normal operating conditions. And when you have a normal user space that boils down to:
- Acquire the kernel you are going to boot.
- Run the user space shutdown scripts, so the system is in a clean state.
- Execute the new kernel.
- The normal case is that the newly loaded kernel will live at the same physical location where the current kernel lives.

Currently my code handles starting a new kernel under normal operating conditions. There are currently two ways I can implement a clean user space shutdown with out needing locked buffers in the kernel until the very last moment.

Method 1 (This works today with my sample user space):
- copy the kernel to /newkernel
- init 6
- if [ -r /newkernel ]; then
/sbin/kexec /newkernel
else
/sbin/reboot
fi
- /sbin/kexec reads in /newkernel
- /newkernel is parsed to figure out how it should be loaded
- sys_kexec is called to copy the kernel from user space anonymous memory to temporary kernel buffers.

Method 2 (For people with read only roots):
- /sbin/delayed_kexec /path/to/new/kernel
- Read in the /path/to/new/kernel into anonymous pages
- Parse it and figure out how it should be loaded
- Run the shutdown scripts from /etc/rc6.d/
- Call sys_kexec, which will copy the data from user space anonymous pages, to kernel space.

This is to just make it clear that I am not working from a FUNDAMENTALLY BROKEN interface, nor from a broken model of machine maintenance. I am quite willing to make changes assuming I understand what is gained with the change.

What I currently support is a moderately nice interface, that has the kernel doing as much as it can without being bogged down in the specific details in any one file format, or needing something besides a 32bit entry point to jump to.

I model an image as a set of segments of physical memory. And I copy the image loaded with sys_kexec to it's final location, before jumping to the new image. There are two reasons for this. It takes 3 segments to load a bzImage (setup.S, vmlinux, and an initrd). And an arbitrary number of segments maps cleanly to a static ELF binary.

There is only one difficult case. What happens when the buffers the kernel allocates are physically in one of the segments of memory of the new kernel image. Possible especially for the initrd which is loaded at the end of memory.

I then use the following algorithm to sort the potential mess out before I jump to the new code. And since this code depends on swapping the contents of pages, knowing the physical location of the pages, and is not limited to 128MB I am reluctant to look a vmalloc variant. I can more get my pages from a slab if I need to report I have them.

static int kimage_get_off_destination_pages(struct kimage *image)

{
	kimage_entry_t *ptr, *cptr, entry;
	unsigned long buffer, page;
	unsigned long destination = 0;

	/* Here we implement safe guards to insure that
	 * a source page is not copied to it's destination
	 * page before the data on the destination page is
	 * no longer useful.
	 *
	 * To make it work we actually wind up with a 
	 * stronger condition.  For every page considered
	 * it is either it's own destination page or it is
	 * not a destination page of any page considered.
	 *
	 * Invariants 
	 * 1. buffer is not a destination of a previous page.
	 * 2. page is not a destination of a previous page.
	 * 3. destination is not a previous source page.
	 *
	 * Result: Either a source page and a destination page 
	 * are the same or the page is not a destination page.
	 *
	 * These checks could be done when we allocate the pages,
	 * but doing it as a final pass allows us more freedom
	 * on how we allocate pages.
	 * 
	 * Also while the checks are necessary, in practice nothing
	 * happens.  The destination kernel wants to sit in the
	 * same physical addresses as the current kernel so we never
	 * actually allocate a destination page.
	 *
	 * BUGS: This is a O(N^2) algorithm.
	 */

	
	buffer = __get_free_page(GFP_KERNEL);
	if (!buffer) {
		return -ENOMEM;
	}
	buffer = virt_to_phys((void *)buffer);
	for_each_kimage_entry(image, ptr, entry) {
		/* Here we check to see if an allocated page */
		kimage_entry_t *limit;
		if (entry & IND_DESTINATION) {
			destination = entry & PAGE_MASK;
		}
		else if (entry & IND_INDIRECTION) {
			/* Indirection pages must include all of their
			 * contents in limit checking.
			 */
			limit = phys_to_virt(page + PAGE_SIZE - sizeof(*limit));
		}
		if (!((entry & IND_SOURCE) | (entry & IND_INDIRECTION))) {
			continue;
		}
		page = entry & PAGE_MASK;
		limit = ptr;

		/* See if a previous page has the current page as it's 
		 * destination.
		 * i.e. invariant 2
		 */
		cptr = kimage_dst_conflict(image, page, limit);
		if (cptr) {
			unsigned long cpage;
 			kimage_entry_t centry;
			centry = *cptr;
			cpage = centry & PAGE_MASK;
			memcpy(phys_to_virt(buffer), phys_to_virt(page), PAGE_SIZE);
			memcpy(phys_to_virt(page), phys_to_virt(cpage), PAGE_SIZE);
			*cptr = page | (centry & ~PAGE_MASK);
			*ptr = buffer | (entry & ~PAGE_MASK);
			buffer = cpage;
		}
		if (!(entry & IND_SOURCE)) {
			continue;
		}

		/* See if a previous page is our destination page.
		 * If so claim it now.
		 * i.e. invariant 3
		 */
		cptr = kimage_src_conflict(image, destination, limit);
		if (cptr) {
			unsigned long cpage;
 			kimage_entry_t centry;
			centry = *cptr;
			cpage = centry & PAGE_MASK;
			memcpy(phys_to_virt(buffer), phys_to_virt(cpage), PAGE_SIZE);
			memcpy(phys_to_virt(cpage), phys_to_virt(page), PAGE_SIZE);
			*cptr = buffer | (centry & ~PAGE_MASK);
			*ptr = cpage | ( entry & ~PAGE_MASK);
			buffer = page;
		}
		/* If the buffer is my destination page do the copy now 
		 * i.e. invariant 3 & 1
		 */
		if (buffer == destination) {
			memcpy(phys_to_virt(buffer), phys_to_virt(page), PAGE_SIZE);
			*ptr = buffer | (entry & ~PAGE_MASK);
			buffer = page;
		}
	}
	free_page((unsigned long)phys_to_virt(buffer));
	return 0;
}

static kimage_entry_t *kimage_dst_conflict(
struct kimage *image, unsigned long page, kimage_entry_t *limit)

{
	kimage_entry_t *ptr, entry;
	unsigned long destination = 0;
	for_each_kimage_entry(image, ptr, entry) {
		if (ptr == limit) {
			return 0;
		}
		else if (entry & IND_DESTINATION) {
			destination = entry & PAGE_MASK;
		}
		else if (entry & IND_SOURCE) {
			if (page == destination) {
				return ptr;
			}
			destination += PAGE_SIZE;
		}
	}
	return 0;
}

static kimage_entry_t *kimage_src_conflict(
struct kimage *image, unsigned long destination, kimage_entry_t *limit)

{
	kimage_entry_t *ptr, entry;
	for_each_kimage_entry(image, ptr, entry) {
		unsigned long page;
		if (ptr == limit) {
			return 0;
		}
		else if (entry & IND_DESTINATION) {
			/* nop */
		}
		else if (entry & IND_DONE) {
			/* nop */
		}
		else {
			/* SOURCE & INDIRECTION */
			page = entry & PAGE_MASK;
			if (page == destination) {
				return ptr;
			}
		}
	}
	return 0;
}

Having had time to digest the idea of starting a new kernel on panic I can now make some observations and what I believe it would take to make it as robust as possible.

- On panic because random pieces of the kernel may be broken we want to use as little of the kernel as possible.

- Therefore machine_kexec should not allocate any memory, and as quickly as possible it should transition to the new kernel

- So a new page table should be sitting around with the new kernel already mapped, and likewise other important tables like the gdt, and the idt, should be pre-allocated.

- Then machine_kexec can just switch stacks, page tables, and other machine dependent tables and jump to the new kernel.

- The load stage needs to load everything at the physical location it will initially run at. This would likely need support from rmap.

- The load stage needs to preallocate page tables and buffers.

- The load stage would likely work easiest by either requiring a mem=xxx line, reserving some of physical memory for the new kernel. Or alternatively using some rmap support to clear out a swath of physical memory the new kernel can be loaded into.

- The new kernel loaded on panic must know about the previous kernel, and have various restrictions because of that.

Supporting a kernel loaded from a normal environment is a rather different problem.

- It cannot be loaded at it's run location (The current kernel is sitting there).

- It should not need to know about the previously executing kernel.

- Work can be done in machine_kexec to allocate memory so everything does not need to be pre allocated.

- I can safely use multiple calls to the page allocator instead of needing a special mechanism to allocate memory.

And now I go back to the silly exercise of factoring my code so the new kernel can be kept in locked kernel memory, instead of in a file while the shutdown scripts are run.

Unless the linux kernel is modified to copy itself to the top of physical memory when it loads I have trouble seeing how any of this will help make the panic case easier to implement.

Eric

PLEASE tell me why you don't just use the 20-line "vmalloc()" function I already wrote for you?

It works for all cases - and since you do need to load the kernel into memory anyway, it's not using any more memory than your existing code. And it's infinitely more flexible to have a clearly separated load-process, than having to have some load happen at reboot time (whether by init or by anything else).

And since the kernel is fully working at the load time, you can even do things like swap out pages in order to make room for the kernel at the right place. So you can even do something like this:

	int alloc_kernel_pages(unsigned long *array, int nr_pages,
		unsigned long min_address)
	{
		void *bad_page_list = NULL;
		int i = 0, retval;

		while (i < nr_pages) {
			unsigned long page = __get_free_page(GFP_USER);

			if (!page)
				goto oom;

			if (page < min_address) {
				*(void **)page = bad_page_list;
				bad_page_list = (void *)page;
				continue;
			}
			array[i] = page;
			i++;
		}
		retval = 0;
	out:
		while (bad_page_list) {
			unsigned long page = (unsigned long) bad_page_list;
			bad_page_list = *(void **)bad_page_list;
			free_page(page);
		}
		return retval;
	oom:
		while (i > 0)
			free_page(array[--i]);
		retval = -ENOMEM;
		goto out;
	}

and now you are guaranteed that all the allocated pages are above a certain mark (change the "min_address" to be a "validity callback" or whatever if you want to be fancy and allow arbitrary rules, which is good if you want to allow pages in the low 1M on x86, for example), which means that your final reboot stage is _much_much_ simpler and you don't ever have to worry about overlap.

Use one of the pages to allocate the memcpy() trampoline and the actual data structures used for the copy, for example. Use the rest for the actual kernel data.

Keep it simple.

Linus

The reasons I don't jump on board:
- It does not handle multiple segments.
Without multiple segments I think I simply out essential complexity of the problem. A bzImage has at least 2.

- vmalloc is artificially limited to 128MB.

- I still have to run code to prevent imperfect overlaps. A perfect overlap being a source buffer living in it's destination address.

- I still have to run code to find the physical addresses of the pages, and locate those in non-destination pages, and form a linked list of pages for that.

I am trying to process it but I don't see why having the load happen as a seperate syscall is clearer. Having it happen as a seperate architecture independent function I understand.

asmlinkage long sys_kexec(unsigned long entry, long nr_segments,
struct kexec_segment *segments)

{
	/* Am I using to much stack space here? */
	struct kimage image;
	int result;

	/* We only trust the superuser with rebooting the system. */
	if (!capable(CAP_SYS_BOOT))
		return -EPERM;

	lock_kernel();

//// This chunk does the load and there is no kernel shutdown code //// run yet.

	kimage_init(&image);
	result = do_kexec(entry, nr_segments, segments, &image);
	if (result) {
		kimage_free(&image);
		unlock_kernel();
		return result;
	}

//// ----------- I can snip here for your two syscall version -----------

//// This part is the kernel shutdown

	/* The point of no return is here... */
	notifier_call_chain(&reboot_notifier_list, SYS_RESTART, NULL);
	system_running = 0;
	device_shutdown();
	printk(KERN_EMERG "Starting new kernel\n");

//// And here is where I start the new kernel.

	machine_kexec(&image);
}

I have clearly separated load code, that runs before any of the kernel starts to shutdown. Until it completes successfully I do not start to shutdown the kernel. My user space is shut down but that is a different story.

Swapping out pages is nice, but when user space is shutdown there shouldn't be any extra pages in the kernel to swap out, and if you are that tight on memory that you need to swap it won't work, anyway.

Which is a good algorithm but it has the potential to allocate a lot of extra pages, and I have implemented this it in the past. It's worst case is just nasty.

My current code allocates at most 1 extra page and works gracefully if it happens to allocates the pages it really wanted to use. It is just a hair more complex, and it makes everything else very simple.

Exactly and that is why I do it where I do it. In the C load code. In the kernel so it has to be written only once.

Yep.

After loading everything I have a total of 243 lines of code. 100 lines of assembly doing the copies in the trampoline. 143 lines of C modifying the page tables, the gdt, and the idt, copying the trampoline to the correct place, and going for it.

And despite my utter puzzlement on why you want the syscall cut in two. I will now go cut along the dotted line. If that is all it takes to have piece I can do that. A sore head from all of the scratching trying to figure out why it needs to be cut in two, but I can cut sys_kexec in two.

Eric

Thats a matter for user space and the unpacker

Just grabbing a load of pages and using kmap/scatter gather by hand is not

To use kmapped memory I need to setup a page table to do the final copy. And to setup a page table I need to know where the memory is going to be copied to.

So my gut impression at least says an interface that ignores where the image wants to live just adds complexity in other places, and makes for an interface that is hard to maintain long term, because you depend on a lot of kernel implementation details, which are likely to change in arbitrary ways.

Eric

Linus' alloc_kernel_pages function would actually be able to handle this, provided that the "validity callback" checks if the allocated page happens to be in one of the destination areas.

I'm not so sure if this implementation is really that much more compact than your current conflict resolution, though. Also, it may be hairy in scenarios where you actually expect to fill more than 50% of system memory. (But your concerns about a 128MB limit scare me, too. I realize that people have taken initrds to extremes I never quite imagined, but that still looks a little excessive :-)

- Werner

  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/

I have not heard of more than about 90MB. One of the things I would not be surprised to see in the next couple of years as memory gets cheaper is diskless systems that don't even bother doing NFS root and just put everything in an initrd. But that is not the main concern.

Since there are more polite ways of allocating memory already implemented. Sucking up a 16MB hunk of some ones vmalloc space is quite rude. Currently the limit is pretty much 50% of system memory or 1GB whichever is less because the code must be loaded into user space first, and I don't touch high memory. Although I guess if it was mmaped read only the limit may be higher.

I don't expect to come to close to using all of system memory except on limited memory systems. But it is always nice to be polite.

Eric

And ?

I find it hard to believe you can't drive an MMU if you can write code that boots one Linux from another

One of the simplifying things I do between OS's is turn of the MMU, or at least give it a 1-1 trivial mapping with physical memory.

If all of that memory is hanging out there forever. It probably makes sense to be high memory capable. But for the first rev of this I won't be. Addresses > 4GB are a major pain to work with on x86.

But I do have a test machine that can reproduce that so I can test for strange bugs. I added a BIOS option to put all but 512M out of 4GB above the 4GB line.

Eric

I'm amazed about your puzzlement, since everybody else seem to get my arguments, but as long as you play along I don't much care.

I will explain once more why it needs to be cut into two, even if you're apparently willing to do it even without understanding:

When you reboot, you often cannot load the image.

This is _trivially_ true for panics or things like

- I don't understand why you do not want to accept this. Even if your code doesn't even _handle_ panics, it's so obvious that this is true that I don't understand why you want a limitation in your particular current implementation to be a fundamental flaw of the whole idea.

But it is _also_ true for any standard setup where you don't have a special "init" that knows about loading the kernel, and where to load it from.

- Do you want to rewrite every "init" setup out there, adding some way to tell init where to load the kernel from?

Or do you want to just split the thing in two, so that you can load the kernel _before_ you ask init to shut down, and just happily use bog-standard tools that everybody is already familiar with..

The two-part loader can clearly handle both cases. And if _you_ don't want a two-part loader, you can do exactly what you do now by just doing two system calls.

As to vmalloc - I don't actually much care how the first and second parts are implemented. I suggested a vmalloc()-like approach just because your patch looks unnecessarily complicated to me. But while I am convinced that the two-phase loading/exec is absolutely the way to do it, the actual low-level implementation is just a detail.

Linus

That the load needs to be separate for handling panics is trivially true. I simply have a very hard time believing that the load you want for the normal case will be the load you want for a panic. I think I want to be much more paranoid in preparing for the kernel to blow up. And I want to move data around much more carefully. And that care adds restrictions I want for the normal case.

So splitting it up to prepare for panic handling just looks like over design.

When you can change the init setup with just a couple of lines of shell script seeing if file exists in magic location (say a special ramfs or tmpfs), I guess it does not look hard to me.

Right which is why I don't much care, so long as I don't have to test reboot on panic any time soon...

I doubt we will see eye to eye on this one. So I will now finish up the patch to split this all up.

I'd love to make it simpler as well if I saw a clear opportunity that wasn't just moving the complexity somewhere else. But when I really look at it I think that I am just wordy.

Eric

O.k. Here is the splitup version of my kexec
Added are
sys_reboot(LINUX_REBOOT_CMD_KEXEC)
sys_kexec_load(unsigned long entry, unsigned long nr_segments, struct kexec *segments, unsigned long flags);

The flags field is currently enforced to be zero, but it leaves the window open to tweak what the load does for the panic case.

Currently (because of missing hardware shutdown code) the code only approaches stable in UP without APICs.

Generating a patch to cleanly shutdown the apics, and releasing a sample user space is the next step.

Eric

 MAINTAINERS                        |    7 
 arch/i386/Kconfig                  |   17 
 arch/i386/kernel/Makefile          |    1 
 arch/i386/kernel/entry.S           |    1 
 arch/i386/kernel/machine_kexec.c   |  142 ++++++++
 arch/i386/kernel/relocate_kernel.S |   99 +++++
 include/asm-i386/kexec.h           |   25 +
 include/asm-i386/unistd.h          |    1 
 include/linux/kexec.h              |   46 ++
 include/linux/reboot.h             |    2 
 kernel/Makefile                    |    1 
 kernel/kexec.c                     |  643 +++++++++++++++++++++++++++++++++++++
 kernel/sys.c                       |   23 +
 13 files changed, 1008 insertions

diff -uNr linux-2.5.46-bk6/MAINTAINERS linux-2.5.46-bk6.x86kexec/MAINTAINERS
@@ -968,6 +968,13 @@
 W:	http://www.cse.unsw.edu.au/~neilb/patches/linux-devel/
 S:	Maintained
 
+KEXEC
+P:	Eric Biederman
+M:	[email protected]
+M:	[email protected]
+L:	[email protected]
+S:	Maintained
+
 LANMEDIA WAN CARD DRIVER
 P:	Andrew Stanley-Jones
 M:	[email protected]

diff -uNr linux-2.5.46-bk6/arch/i386/Kconfig linux-2.5.46-bk6.x86kexec/arch/i386/Kconfig
@@ -784,6 +784,23 @@
 	depends on (SMP || PREEMPT) && X86_CMPXCHG
 	default y
 
+config KEXEC
+	bool "kexec system call (EXPERIMENTAL)"
+	depends on EXPERIMENTAL
+	help
+	  kexec is a system call that implements the ability to  shutdown your
+	  current kernel, and to start another kernel.  It is like a reboot
+	  but it is indepedent of the system firmware.   And like a reboot the
+	  you can start any kernel with it not just Linux.  
+	
+	  The name comes from the similiarity to the exec system call. 
+	
+	  It is on an going process to be certain the hardware in a machine
+	  is properly shutdown, so do not be surprised if this code does not
+	  initially work for you.  It may help to enable device hotplugging
+	  support.  As of this writing the exact hardware interface is
+	  strongly in flux, so no good recommendation can be made.
+
 endmenu
 
 

diff -uNr linux-2.5.46-bk6/arch/i386/kernel/Makefile linux-2.5.46-bk6.x86kexec/arch/i386/kernel/Makefile
@@ -24,6 +24,7 @@
 obj-$(CONFIG_X86_MPPARSE)	+= mpparse.o
 obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o nmi.o
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
+obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o
 obj-$(CONFIG_SOFTWARE_SUSPEND)	+= suspend.o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq.o
 obj-$(CONFIG_PROFILING)		+= profile.o

diff -uNr linux-2.5.46-bk6/arch/i386/kernel/entry.S linux-2.5.46-bk6.x86kexec/arch/i386/kernel/entry.S
@@ -743,6 +743,7 @@
 	.long sys_epoll_ctl	/* 255 */
 	.long sys_epoll_wait
  	.long sys_remap_file_pages
+	.long sys_kexec_load
 
 
 	.rept NR_syscalls-(.-sys_call_table)/4

diff -uNr linux-2.5.46-bk6/include/asm-i386/kexec.h linux-2.5.46-bk6.x86kexec/include/asm-i386/kexec.h
@@ -0,0 +1,25 @@
+#ifndef _I386_KEXEC_H
+#define _I386_KEXEC_H
+
+#include <asm/fixmap.h>
+
+/*
+ * KEXEC_SOURCE_MEMORY_LIMIT maximum page get_free_page can return.
+ * I.e. Maximum page that is mapped directly into kernel memory,
+ * and kmap is not required.
+ *
+ * Someone correct me if FIXADDR_START - PAGEOFFSET is not the correct
+ * calculation for the amount of memory directly mappable into the
+ * kernel memory space.
+ */
+
+/* Maximum physical address we can use pages from */
+#define KEXEC_SOURCE_MEMORY_LIMIT (FIXADDR_START - PAGE_OFFSET) 
+/* Maximum address we can reach in physical address mode */
+#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)
+
+#define KEXEC_REBOOT_CODE_SIZE	4096
+#define KEXEC_REBOOT_CODE_ALIGN 0
+
+
+#endif /* _I386_KEXEC_H */

diff -uNr linux-2.5.46-bk6/include/asm-i386/unistd.h linux-2.5.46-bk6.x86kexec/include/asm-i386/unistd.h
@@ -262,6 +262,7 @@
 #define __NR_sys_epoll_ctl	255
 #define __NR_sys_epoll_wait	256
 #define __NR_remap_file_pages	257
+#define __NR_sys_kexec_load	258
 
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */

diff -uNr linux-2.5.46-bk6/include/linux/reboot.h linux-2.5.46-bk6.x86kexec/include/linux/reboot.h
@@ -21,6 +21,7 @@
  * POWER_OFF   Stop OS and remove all power from system, if possible.
  * RESTART2    Restart system using given command string.
  * SW_SUSPEND  Suspend system using Software Suspend if compiled in
+ * KEXEC       Restart the system using a different kernel.
  */
 
 #define	LINUX_REBOOT_CMD_RESTART	0x01234567

@@ -30,6 +31,7 @@
 #define	LINUX_REBOOT_CMD_POWER_OFF	0x4321FEDC
 #define	LINUX_REBOOT_CMD_RESTART2	0xA1B2C3D4
 #define	LINUX_REBOOT_CMD_SW_SUSPEND	0xD000FCE2
+#define LINUX_REBOOT_CMD_KEXEC		0x45584543
 
 
 #ifdef __KERNEL__

diff -uNr linux-2.5.46-bk6/kernel/Makefile linux-2.5.46-bk6.x86kexec/kernel/Makefile
@@ -21,6 +21,7 @@
 obj-$(CONFIG_CPU_FREQ) += cpufreq.o
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o
+obj-$(CONFIG_KEXEC) += kexec.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

diff -uNr linux-2.5.46-bk6/kernel/sys.c linux-2.5.46-bk6.x86kexec/kernel/sys.c
@@ -16,6 +16,7 @@
 #include <linux/init.h>
 #include <linux/highuid.h>
 #include <linux/fs.h>
+#include <linux/kexec.h>
 #include <linux/workqueue.h>
 #include <linux/device.h>
 #include <linux/times.h>

@@ -206,6 +207,7 @@
 cond_syscall(sys_lookup_dcookie)
 cond_syscall(sys_swapon)
 cond_syscall(sys_swapoff)
+cond_syscall(sys_kexec_load)
 
 static int set_one_prio(struct task_struct *p, int niceval, int error)
 {

@@ -414,6 +416,27 @@
 		machine_restart(buffer);
 		break;
 
+#ifdef CONFIG_KEXEC
+	case LINUX_REBOOT_CMD_KEXEC:
+	{
+		struct kimage *image;
+		spin_lock(&kexec_image_lock);
+		image = kexec_image;
+		if (!image || arg) {
+			spin_unlock(&kexec_image_lock);
+			unlock_kernel();
+			return -EINVAL;
+		}
+		notifier_call_chain(&reboot_notifier_list, SYS_RESTART, NULL);
+		system_running = 0;
+		device_shutdown();
+		printk(KERN_EMERG "Starting new kernel\n");
+		machine_kexec(image);
+		/* We never get here... */
+		spin_unlock(&kexec_image_lock);
+		break;
+	}
+#endif
 #ifdef CONFIG_SOFTWARE_SUSPEND
 	case LINUX_REBOOT_CMD_SW_SUSPEND:
 		if (!software_suspend_enabled) {

I think this comes from being the guy down in the trenches implementing
the code. And it is sometimes hard to look up, far enough to have design
discussions.

I totally agree that having a load/exec split is the right approach now that I can imagine an implementation where the code will actually work for the panic case. Before it felt like lying. Doing the split-up, promising that kexec on panic will work eventually, when I could not even see it as a possibility was at the core of my objections.

What brought me around is that I can add a flag field to kexec_load. With that flag field I can tell the kernel please step extra carefully this code will be used to handle kexec on panic. Without that I may be up a creek without a paddle for figuring out how to debug that code.

To be able to support this at all I have had to be very creative in inventing debugging code. Which is why I have the serial console program kexec_test. It provides visibility into what is happening when nothing else will. That and memtest86 which will occasionally catch DMA's that have not been stopped, (memory errors on good ram) I at least have a place to start rather than a blank screen when guessing why the new kernel did not start up.

Eric

kexec is a set of system calls that allows you to load another kernel from the currently executing Linux kernel. The current implementation has only been tested, and had the kinks worked out on x86, but the generic code (kexec_load) should work on any architecture.

Some machines have BIOSes that are either extremely slow to reboot, or that cannot reliably perform a reboot. In which case kexec may be the only alternative to reboot in a reliable and timely manner.

The patch is archived at:
http://www.xmission.com/~ebiederm/files/kexec/

And is currently kept in two pieces.
The pure system call.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.47.x86kexec.diff

And the set of hardware fixes known to help kexec.
http://www.xmission.com/~ebiederm/files/kexec/linux-2.5.47.x86kexec-hwfixes.diff

A compatible user space is at:
http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.5.tar.gz
This code boots either a static ELF executable or a bzImage.

A kernel reformater that bypasses setup.S in favor of a version that uses fewer BIOS calls, (increasing the reliability) is at:
ftp://ftp.lnxi.com/pub/mkelfImage/mkelfImage-1.18.tar.gz

In bug reports please include the serial console output of kexec kexec_test. kexec_test exercises most of the interesting code paths that are needed to load a kernel (mainly BIOS calls) with lots of debugging print statements, so hangs can easily be detected.

To be polite to your user space there are now options:
--load (which just loads the new kernel)
--exec (which starts a previously loaded kernel).
I expect to integrate more gracefully with init as time goes on, but this is what I can do in a timely manner.

Without applying the hardware fixes you must build a kernel that is uniprocessor and does not use an APIC, to have a chance at this code working. Cleaning up various hardware fixes and getting them integrated into the kernel is the next step.

Hopefully this has an interface Linus likes now.

Eric

 MAINTAINERS                        |    7 
 arch/i386/Kconfig                  |   17 
 arch/i386/kernel/Makefile          |    1 
 arch/i386/kernel/entry.S           |    1 
 arch/i386/kernel/machine_kexec.c   |  142 ++++++++
 arch/i386/kernel/relocate_kernel.S |   99 +++++
 include/asm-i386/kexec.h           |   25 +
 include/asm-i386/unistd.h          |    1 
 include/linux/kexec.h              |   46 ++
 include/linux/reboot.h             |    2 
 kernel/Makefile                    |    1 
 kernel/kexec.c                     |  643 +++++++++++++++++++++++++++++++++++++
 kernel/sys.c                       |   23 +
 13 files changed, 1008 insertions

diff -uNr linux-2.5.47/MAINTAINERS linux-2.5.47.x86kexec/MAINTAINERS
@@ -968,6 +968,13 @@
 W:	http://www.cse.unsw.edu.au/~neilb/patches/linux-devel/
 S:	Maintained
 
+KEXEC
+P:	Eric Biederman
+M:	[email protected]
+M:	[email protected]
+L:	[email protected]
+S:	Maintained
+
 LANMEDIA WAN CARD DRIVER
 P:	Andrew Stanley-Jones
 M:	[email protected]

diff -uNr linux-2.5.47/arch/i386/Kconfig linux-2.5.47.x86kexec/arch/i386/Kconfig
@@ -784,6 +784,23 @@
 	depends on (SMP || PREEMPT) && X86_CMPXCHG
 	default y
 
+config KEXEC
+	bool "kexec system call (EXPERIMENTAL)"
+	depends on EXPERIMENTAL
+	help
+	  kexec is a system call that implements the ability to  shutdown your
+	  current kernel, and to start another kernel.  It is like a reboot
+	  but it is indepedent of the system firmware.   And like a reboot
+	  you can start any kernel with it not just Linux.  
+	
+	  The name comes from the similiarity to the exec system call. 
+	
+	  It is on an going process to be certain the hardware in a machine
+	  is properly shutdown, so do not be surprised if this code does not
+	  initially work for you.  It may help to enable device hotplugging
+	  support.  As of this writing the exact hardware interface is
+	  strongly in flux, so no good recommendation can be made.
+
 endmenu
 
 

diff -uNr linux-2.5.47/arch/i386/kernel/Makefile linux-2.5.47.x86kexec/arch/i386/kernel/Makefile
@@ -24,6 +24,7 @@
 obj-$(CONFIG_X86_MPPARSE)	+= mpparse.o
 obj-$(CONFIG_X86_LOCAL_APIC)	+= apic.o nmi.o
 obj-$(CONFIG_X86_IO_APIC)	+= io_apic.o
+obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o
 obj-$(CONFIG_SOFTWARE_SUSPEND)	+= suspend.o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq.o
 obj-$(CONFIG_PROFILING)		+= profile.o

diff -uNr linux-2.5.47/arch/i386/kernel/entry.S linux-2.5.47.x86kexec/arch/i386/kernel/entry.S
@@ -743,6 +743,7 @@
 	.long sys_epoll_ctl	/* 255 */
 	.long sys_epoll_wait
  	.long sys_remap_file_pages
+	.long sys_kexec_load
 
 
 	.rept NR_syscalls-(.-sys_call_table)/4

diff -uNr linux-2.5.47/include/asm-i386/kexec.h linux-2.5.47.x86kexec/include/asm-i386/kexec.h
@@ -0,0 +1,25 @@
+#ifndef _I386_KEXEC_H
+#define _I386_KEXEC_H
+
+#include <asm/fixmap.h>
+
+/*
+ * KEXEC_SOURCE_MEMORY_LIMIT maximum page get_free_page can return.
+ * I.e. Maximum page that is mapped directly into kernel memory,
+ * and kmap is not required.
+ *
+ * Someone correct me if FIXADDR_START - PAGEOFFSET is not the correct
+ * calculation for the amount of memory directly mappable into the
+ * kernel memory space.
+ */
+
+/* Maximum physical address we can use pages from */
+#define KEXEC_SOURCE_MEMORY_LIMIT (FIXADDR_START - PAGE_OFFSET) 
+/* Maximum address we can reach in physical address mode */
+#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL)
+
+#define KEXEC_REBOOT_CODE_SIZE	4096
+#define KEXEC_REBOOT_CODE_ALIGN 0
+
+
+#endif /* _I386_KEXEC_H */

diff -uNr linux-2.5.47/include/asm-i386/unistd.h linux-2.5.47.x86kexec/include/asm-i386/unistd.h
@@ -262,6 +262,7 @@
 #define __NR_sys_epoll_ctl	255
 #define __NR_sys_epoll_wait	256
 #define __NR_remap_file_pages	257
+#define __NR_sys_kexec_load	258
 
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */

diff -uNr linux-2.5.47/include/linux/reboot.h linux-2.5.47.x86kexec/include/linux/reboot.h
@@ -21,6 +21,7 @@
  * POWER_OFF   Stop OS and remove all power from system, if possible.
  * RESTART2    Restart system using given command string.
  * SW_SUSPEND  Suspend system using Software Suspend if compiled in
+ * KEXEC       Restart the system using a different kernel.
  */
 
 #define	LINUX_REBOOT_CMD_RESTART	0x01234567

@@ -30,6 +31,7 @@
 #define	LINUX_REBOOT_CMD_POWER_OFF	0x4321FEDC
 #define	LINUX_REBOOT_CMD_RESTART2	0xA1B2C3D4
 #define	LINUX_REBOOT_CMD_SW_SUSPEND	0xD000FCE2
+#define LINUX_REBOOT_CMD_KEXEC		0x45584543
 
 
 #ifdef __KERNEL__

diff -uNr linux-2.5.47/kernel/Makefile linux-2.5.47.x86kexec/kernel/Makefile
@@ -21,6 +21,7 @@
 obj-$(CONFIG_CPU_FREQ) += cpufreq.o
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o
+obj-$(CONFIG_KEXEC) += kexec.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

diff -uNr linux-2.5.47/kernel/sys.c linux-2.5.47.x86kexec/kernel/sys.c
@@ -16,6 +16,7 @@
 #include <linux/init.h>
 #include <linux/highuid.h>
 #include <linux/fs.h>
+#include <linux/kexec.h>
 #include <linux/workqueue.h>
 #include <linux/device.h>
 #include <linux/times.h>

@@ -206,6 +207,7 @@
 cond_syscall(sys_lookup_dcookie)
 cond_syscall(sys_swapon)
 cond_syscall(sys_swapoff)
+cond_syscall(sys_kexec_load)
 
 static int set_one_prio(struct task_struct *p, int niceval, int error)
 {

@@ -414,6 +416,27 @@
 		machine_restart(buffer);
 		break;
 
+#ifdef CONFIG_KEXEC
+	case LINUX_REBOOT_CMD_KEXEC:
+	{
+		struct kimage *image;
+		spin_lock(&kexec_image_lock);
+		image = kexec_image;
+		if (!image || arg) {
+			spin_unlock(&kexec_image_lock);
+			unlock_kernel();
+			return -EINVAL;
+		}
+		notifier_call_chain(&reboot_notifier_list, SYS_RESTART, NULL);
+		system_running = 0;
+		device_shutdown();
+		printk(KERN_EMERG "Starting new kernel\n");
+		machine_kexec(image);
+		/* We never get here... */
+		spin_unlock(&kexec_image_lock);
+		break;
+	}
+#endif
 #ifdef CONFIG_SOFTWARE_SUSPEND
 	case LINUX_REBOOT_CMD_SW_SUSPEND:
 		if (!software_suspend_enabled) {

FYI: that patch applies cleanly to pure 2.5.47 (bk [email protected]).

The current front of the tree does not patch 100% cleanly (conflicts with recent module changes).

Results on my usual problem machine:

# ./kexec-1.5 ./kexec_test-1.5
Shutting down devices
Debug: sleeping function called from illegal context at include/asm/semaphore.h9
Call Trace: [<c011a698>]  [<c0216193>]  [<c012b165>]  [<c0132dec>]  [<c0140357> Starting new kernel
kexec_test 1.5 starting...
eax: 0E1FB007 ebx: 00001078 ecx: 00000000 edx: 00000000
esi: 00000000 edi: 00000000 esp: 00000000 ebp: 00000000
idt: 00000000 C0000000
gdt: 00000000 C0000000
Switching descriptors.
Descriptors changed.
Legacy pic setup.
In real mode.
<hang>

Sorry about the linewrap.

Same as last time, but the good news is that splitting the load and reboot operations works as expected.

Missing or inaccessible. I'll try some duct tape and the linux-2.5.44.x86kexec-hwfixes.diff and see what happens.

Confirming some earlier suspicions:
CONFIG_SMP=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y

Last time I tried to run a UP kernel (and no APIC support) on this system it wasn't pretty. I'll add that to my list of combinations to try.

And as always:
% lspci
00:00.0 Host bridge: ServerWorks CNB20LE Host Bridge (rev 06) 00:00.1 Host bridge: ServerWorks CNB20LE Host Bridge (rev 06) 00:01.0 VGA compatible controller: S3 Inc. Savage 4 (rev 04) 00:09.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08) 00:0f.0 ISA bridge: ServerWorks OSB4 South Bridge (rev 50) 00:0f.1 IDE interface: ServerWorks OSB4 IDE Controller
00:0f.2 USB Controller: ServerWorks OSB4/CSB5 OHCI USB Controller (rev 04) 01:03.0 SCSI storage controller: Adaptec AIC-7892P U160/m (rev 02) % cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 10
cpu MHz : 799.957
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1576.96
%

I will have to take a look next time a snapshot is uploaded. bk and I have not yet become friends.

Hmm. I wonder what is doing that. Do you have the semaphore problem on a normal reboot?

Yep it works until it runs into your apics that are not shutdown. That looks like one of the next things to tackle.

That is what my test machine said as well. But the confirmation is nice. And it definitely means I uploaded a working sample user space.

The .47 version is pretty much just a forward port. It is uploaded now. My apologies for not getting to it earlier.

The challenge is with the apic shutdown is that currently the apics are not in the device tree so that needs to happen before I can submit a good version for 2.5.x

I would not worry about it to much. I'm just happy my tools are good enough that with a little thinking I can figure out what the problem is. Getting there was hard.

Eric

No clue as of yet. I do not see this information during a normal reboot.

I used the linux-2.5.44.x86kexec-hwfixes.diff (it applied cleanly to pure 2.5.47 + kexec); I'll try your updated version soon if there are any major differences.

On this same system, I reconfigured and tried this:
# CONFIG_SMP is not set CONFIG_X86_GOOD_APIC=y
# CONFIG_X86_UP_APIC is not set
# CONFIG_X86_LOCAL_APIC is not set
# CONFIG_X86_IO_APIC is not set

None of the "ordinary" APIC initialization messages were output during the regular BIOS->LILO boot of this kernel.

Using kexec on this kernel to run kexec_test-1.5 stops in the same way:

    # ./kexec-1.5 --debug ./kexec_test-1.5
    Shutting down devices
    Debug: sleeping function called from illegal context at
    include/asm/semaphore.h9Call Trace: [<c0113f7c>]  [<c01ec123>] 
    [<c0120af2>]  [<c0130d5d>]  [<c0130d5d> Starting new kernel
    kexec_test 1.5 starting...
    eax: 0E1FB007 ebx: 00001078 ecx: 00000000 edx: 00000000
    esi: 00000000 edi: 00000000 esp: 00000000 ebp: 00000000
    idt: 00000000 C0000000
    gdt: 00000000 C0000000
    Switching descriptors.
    Descriptors changed.
    Legacy pic setup.
    In real mode.
    <hang>

So, does this information suggest looking somewhere other than APIC shutdown?

Andy

Doh. I must compile that debugging in when I am testing. I introduced a spin lock,
and then I called a function that might sleep... Though I am puzzled by what in the device_shutdown and reboot notifier path is actually sleeping but that is academic.

Next version will use a semaphore to be polite.
I should have asked where those addresses mapped to in your system.map.

Anyway one of the reasons I grumble about splitting it, more global variables that have to be protected, and more chances to fumble something. Oh, well.