Skip to contentSkip to conversation

4K stacks default, not a debug thing any more...?

RFC PATCH Bodo Eggert 20 participants 78 messages Jul 15, 2007 – Aug 10, 2007 8 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 11 lines · Matt Mackall
Quoted 4 lines · Rene Herman

As far as I'm aware, the actual reason for 4K stacks is that after the system has been up and running for some time getting "1 physically contiguous pages" becomes significantly easier than 2 which wouldn't be arbitrary.

If there are exactly two free pages in the system, the odds of them being buddies (ie adjacent AND properly aligned) is quite small. The available page pool has to grow quite a bit before the availability of order-1 page pairs approaches 100%.

If there are exactly two free pages in a system, the odds of starting any program are not very good. You'll have to swap, and if you do, you can swap two more pages in order to free enough RAM for the stack.

Signature · 4 lines
The secret of the universe is #@*%! NO CARRIER 

Friß, Spammer: [email protected] [email protected]
 [email protected] [email protected]
#9 lore raw
Quoted 8 lines · Bodo Eggert
Quoted 7 lines · Matt Mackall
Quoted 4 lines · Rene Herman

As far as I'm aware, the actual reason for 4K stacks is that after the system has been up and running for some time getting "1 physically contiguous pages" becomes significantly easier than 2 which wouldn't be arbitrary.

Quoted 8 lines
Quoted 4 lines

If there are exactly two free pages in the system, the odds of them being buddies (ie adjacent AND properly aligned) is quite small. The available page pool has to grow quite a bit before the availability of order-1 page pairs approaches 100%.

If there are exactly two free pages in a system, the odds of starting any program are not very good. You'll have to swap, and if you do, you can swap two more pages in order to free enough RAM for the stack.

even if you have several thousand pages the odds aren't good; or rather, they start out reasonably ok until something starts eating very deliberately at the 8k pages pool, for example the new app creating about 10 threads....

The 4K issue is "tricky", only a few selected workload/compiler combos seem to hit the dirt, yet distros like Fedora and RHEL use 4K stacks since forever, and if it gave massive problems they wouldn't do that. On the upside, especially on very-threaded workloads, it helps reliability and the VM a lot...

#10 lore raw
Quoted 23 lines · Arjan van de Ven
Quoted 8 lines · Bodo Eggert
Quoted 7 lines · Matt Mackall
Quoted 4 lines · Rene Herman

As far as I'm aware, the actual reason for 4K stacks is that after the system has been up and running for some time getting "1 physically contiguous pages" becomes significantly easier than 2 which wouldn't be arbitrary.

Quoted 8 lines
Quoted 4 lines

If there are exactly two free pages in the system, the odds of them being buddies (ie adjacent AND properly aligned) is quite small. The available page pool has to grow quite a bit before the availability of order-1 page pairs approaches 100%.

If there are exactly two free pages in a system, the odds of starting any program are not very good. You'll have to swap, and if you do, you can swap two more pages in order to free enough RAM for the stack.

even if you have several thousand pages the odds aren't good; or rather, they start out reasonably ok until something starts eating very deliberately at the 8k pages pool, for example the new app creating about 10 threads....

Quoted 5 lines

The 4K issue is "tricky", only a few selected workload/compiler combos seem to hit the dirt, yet distros like Fedora and RHEL use 4K stacks since forever, and if it gave massive problems they wouldn't do that. On the upside, especially on very-threaded workloads, it helps reliability and the VM a lot...

I guess no Fedora users run md+lvm+xfs then. That combination has quite reliably crashed any 4k-stack kernel I've ever cared to try.

Signature · 2 lines
Måns Rullgård
[email protected]
#11 lore raw
In reply to Bodo Eggert
Quoted 8 lines · Bodo Eggert
Quoted 7 lines · Matt Mackall
Quoted 4 lines · Rene Herman

As far as I'm aware, the actual reason for 4K stacks is that after the system has been up and running for some time getting "1 physically contiguous pages" becomes significantly easier than 2 which wouldn't be arbitrary.

Quoted 8 lines
Quoted 4 lines

If there are exactly two free pages in the system, the odds of them being buddies (ie adjacent AND properly aligned) is quite small. The available page pool has to grow quite a bit before the availability of order-1 page pairs approaches 100%.

If there are exactly two free pages in a system, the odds of starting any program are not very good. You'll have to swap, and if you do, you can swap two more pages in order to free enough RAM for the stack.

A thread's kernel stack is a kernel allocation. If you'd fail to allocate it you'd supposedly _already_ have swapt out everything that could be swapped out.

Moreoveover -- literally two pages free was hardly his point. The point is just that (with a page being the allocation unit) single page allocations are guaranteed to succeed if _any_ memory is free, while two adjacent (yes, and stacksize aligned) pages will be pretty hard to get by once the system has been up and running for some time.

Rene.

#12 lore raw
Quoted 14 lines · Rene Herman
Quoted 13 lines · Bodo Eggert
Quoted 11 lines · Matt Mackall
Quoted 4 lines · Rene Herman

As far as I'm aware, the actual reason for 4K stacks is that after the system has been up and running for some time getting "1 physically contiguous pages" becomes significantly easier than 2 which wouldn't be arbitrary.

If there are exactly two free pages in the system, the odds of them being buddies (ie adjacent AND properly aligned) is quite small. The available page pool has to grow quite a bit before the availability of order-1 page pairs approaches 100%.

Quoted 6 lines
Quoted 3 lines

If there are exactly two free pages in a system, the odds of starting any program are not very good. You'll have to swap, and if you do, you can swap two more pages in order to free enough RAM for the stack.

A thread's kernel stack is a kernel allocation. If you'd fail to allocate it you'd supposedly _already_ have swapt out everything that could be swapped out.

So we are in a desperate situation, we can almost make no progress, adding another task is going to push the system into an unrecoverable situation, and we make sure this task can be started.-)

Quoted 5 lines

Moreoveover -- literally two pages free was hardly his point. The point is just that (with a page being the allocation unit) single page allocations are guaranteed to succeed if _any_ memory is free, while two adjacent (yes, and stacksize aligned) pages will be pretty hard to get by once the system has been up and running for some time.

That never happened on my servers, therefore I'd opt for the little extra security of having spare 4k on the stack. (I made a patch which would printk a message if allocating a stack would ever fail).

I'm not at all opposed to letting the guys with zillions of threads benefit from having less unused kernel stack, but unless it's secure for all users, it should not be default=y.

Signature · 3 lines
A beggar walked up to a well-dressed woman shopping on Rodeo
Drive and said, "I haven't eaten anything for days."
She looked at him and said, "God, I wish I had your willpower."
#13 lore raw
Quoted 3 lines · Bodo Eggert

So we are in a desperate situation, we can almost make no progress, adding another task is going to push the system into an unrecoverable situation, and we make sure this task can be started.-)

Hnng. He was just saying that odds of two pages being buddies is quite small, not talking about literally two pages.

Quoted 6 lines
Quoted 1 line

Moreoveover -- literally two pages free was hardly his point. The point is just that (with a page being the allocation unit) single page allocations are guaranteed to succeed if _any_ memory is free, while two adjacent (yes, and stacksize aligned) pages will be pretty hard to get by once the system has been up and running for some time.

That never happened on my servers, therefore I'd opt for the little extra security of having spare 4k on the stack. (I made a patch which would printk a message if allocating a stack would ever fail).

I'm not at all opposed to letting the guys with zillions of threads benefit from having less unused kernel stack, but unless it's secure for all users, it should not be default=y.

Given that as Arjan stated Fedora and even RHEL have been using 4K stacks for some time now, and certainly the latter being a distribution which I would expect to both host a relatively large number of lvm/md/xfs and what stackeaters have you users and to be fairly conservative with respect to the chances of scribbling over kernel memory (I'm a trusting person...) it seems
there might at this stage only be very few offenders left.

Seeing as how single-page stacks are much easier on the VM so that creating those zillion threads should also be faster, at _some_ percentage we get to say "and now to hell with the rest".

Do also note that with interrupts of the process stack, available stack is definitely not halved. I don't have data (if anyone reading does, please say) but I expect that on the kinds of busy networked systems that want many-thread creation to be fastest, their many concurrent interrupt sources might mean they are not actually experiencing less stack at all. That is, that "little extra security" you speak of might very well be none at all in practice and perhaps even negative.

Getting interrupts onto their own stack(s) certainly made for better (more deterministic that is) behaviour as well -- you're then independent on how deep into the stack you already are when the interrupt comes in which is otherwise anyone's guess. Now I must say I'm not particularly sure why you couldn't still also have those even if you don't pick 4K stacks, but as far as I'm aware they're a package deal at least today.

Single page stacks are much nicer on anyone. For all I care, that single page migth be a larger soft-page. No idea how the hell you'd investigate the optimum page-size for any given system, but I quite fully expect it's larger than 4K these days _anyway_ even on x86 and for modern loads.

Since Linux doesn't yet have those that's also not very important currently though. 4K is 1024 32-bit datums which isn't all that little if recursion is limited and programmers somewhat competent. With the interrupt stacks meaning available stack might not be worse or even _better_ for some systems, well, I's argue to just go with 4K if at all possible.

Rene.

#14 lore raw
Quoted 3 lines · Rene Herman

Seeing as how single-page stacks are much easier on the VM so that creating those zillion threads should also be faster, at _some_ percentage we get to say "and now to hell with the rest".

This is the core dispute here. Stated differently, I hope you never design a bridge that I have to drive over.

Correctness first, optimization second. Introducing random and difficult to trace crashes upon an unsuspecting audience of sysadmins and users is not a viable option.

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe, or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

#15 lore raw
Quoted 4 lines

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe, or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

Any x86-32 path unsafe with 4K stacks is almost certainly unsafe with 8K stacks because the 8K stacks do not have seperate IRQ stack paths, so you have the same space but split. It might be less predictable on 8K stacks but it isn't absent.

#16 lore raw
Quoted 9 lines · Alan Cox
Quoted 4 lines

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe, or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

Any x86-32 path unsafe with 4K stacks is almost certainly unsafe with 8K stacks because the 8K stacks do not have seperate IRQ stack paths, so you have the same space but split. It might be less predictable on 8K stacks but it isn't absent.

Understood, but isn't that an argument pro interrupt stacks, rather than one against 8k? wli has a patch to break out the interrupt stack feature from the 4k/8k choice.

#17 lore raw
Quoted 10 lines · Ray Lee
Quoted 4 lines · Alan Cox

Any x86-32 path unsafe with 4K stacks is almost certainly unsafe with 8K stacks because the 8K stacks do not have seperate IRQ stack paths, so you have the same space but split. It might be less predictable on 8K stacks but it isn't absent.

Understood, but isn't that an argument pro interrupt stacks, rather than one against 8k? wli has a patch to break out the interrupt stack feature from the 4k/8k choice.

Yes but it's also an argument that the 4K stacks don't make the _current_ situation without CONFIG_4KSTACKS selected worse and given that you trust that current situation, that leaves you without your argument :-)

Rene.

#18 lore raw
Quoted 3 lines · Rene Herman

Yes but it's also an argument that the 4K stacks don't make the _current_ situation without CONFIG_4KSTACKS selected worse and given that you trust that current situation, that leaves you without your argument :-)

Heh :-). No, it's not a question of trust. First and foremost, it's that there are still users who say that they can crash a current 4k+interrupt stacks kernel, while the 8k without interrupt stacks is fine.

#19 lore raw
Quoted 9 lines · On Mon, 16 Jul 2007 16:15:28 -0700 "Ray Lee" <[email protected]> wrote:
Quoted 3 lines · Rene Herman

Yes but it's also an argument that the 4K stacks don't make the _current_ situation without CONFIG_4KSTACKS selected worse and given that you trust that current situation, that leaves you without your argument :-)

Heh :-). No, it's not a question of trust. First and foremost, it's that there are still users who say that they can crash a current 4k+interrupt stacks kernel, while the 8k without interrupt stacks is fine.

You forgot "most of the time". Its statistically less likely, which merely means its evilly hard to debug

#20 lore raw
Quoted 8 lines · Alan Cox
Quoted 4 lines · On Mon, 16 Jul 2007 16:15:28 -0700 "Ray Lee" <[email protected]> wrote:

Heh :-). No, it's not a question of trust. First and foremost, it's that there are still users who say that they can crash a current 4k+interrupt stacks kernel, while the 8k without interrupt stacks is fine.

You forgot "most of the time".

Yeah, fair enough.

Quoted 2 lines

Its statistically less likely, which
merely means its evilly hard to debug

Not being able to debug the cases that occur (and the fact that they're rare, as you're pointing out) is as much of a problem as the crashes themselves. 8k + IRQ stacks with a warning when 4k of process stack is exceeded would seem like a reasonable first step to making 4k a palatable default.

Ray

#21 lore raw
In reply to Alan Cox
Quoted 4 lines · At some point in the past, I wrote:
Quoted 4 lines

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe, or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

Quoted 4 lines · Alan Cox

Any x86-32 path unsafe with 4K stacks is almost certainly unsafe with 8K stacks because the 8K stacks do not have seperate IRQ stack paths, so you have the same space but split. It might be less predictable on 8K stacks but it isn't absent.

At hch's suggestion I rewrote the separate IRQ stack configurability patch into one making IRQ stacks mandatory and unconfigurable, and hence enabled with 8K stacks.

-- wli

#22 lore raw
Quoted 7 lines · William Lee Irwin III
Quoted 4 lines · At some point in the past, I wrote:
Quoted 4 lines

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe, or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

On Mon, Jul 16, 2007 at 11:54:38PM +0100, Alan Cox wrote:

Quoted 9 lines
Quoted 4 lines

Any x86-32 path unsafe with 4K stacks is almost certainly unsafe with 8K stacks because the 8K stacks do not have seperate IRQ stack paths, so you have the same space but split. It might be less predictable on 8K stacks but it isn't absent.

At hch's suggestion I rewrote the separate IRQ stack configurability patch into one making IRQ stacks mandatory and unconfigurable, and hence enabled with 8K stacks.

For what it's worth, that sounds good to me - like something that we would want merged.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#23 lore raw
Quoted 8 lines · Jesper Juhl
Quoted 4 lines · William Lee Irwin III

At hch's suggestion I rewrote the separate IRQ stack configurability patch into one making IRQ stacks mandatory and unconfigurable, and hence enabled with 8K stacks.

For what it's worth, that sounds good to me - like something that we would want merged.

Yes, seperate IRQ stacks make eminent sense in their own right.

Andrea Arcangeli's current thread on soft pages:

http://lkml.org/lkml/2007/7/6/346

is also interesting though in the context of 1-page stacks.

Rene.

#24 lore raw
In reply to Ray Lee
Quoted 8 lines · Ray Lee
Quoted 3 lines · Rene Herman

Seeing as how single-page stacks are much easier on the VM so that creating those zillion threads should also be faster, at _some_ percentage we get to say "and now to hell with the rest".

This is the core dispute here. Stated differently, I hope you never design a bridge that I have to drive over.

Quoted 9 lines

Correctness first, optimization second. Introducing random and difficult to trace crashes upon an unsuspecting audience of sysadmins and users is not a viable option.

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe, or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

Please note that I was not trying to remove the 8K stack option right now - heck, I didn't even add anything to feature-removal-schedule.txt
- all I wanted to accomplish with the patch that started this threas was; a) indicate that the 4K option is no longer a debug thing and
b) make 4K stacks the default option in vanilla kernel.org kernels as a gentle nudge towards getting people to start fixing the code paths that are not 4K stack safe.
Distros that currently use 8K stacks can continue to do so just fine, individuals compiling their own kernel.org kernels can as well and people using oldconfig wouldn't get any change, only people configuring a new kernel.org kernel from scratch would see a change. It was mostly meant as a hint that we want to move in the 4K stack direction over time...
In the future (perhaps far future) when all 4K unsafe codepaths are believed to have been fixed an entry could be made in feature-removal-schedule.txt stating that the 8K option would go away
in 6, 12 or whatever, months. That was my intention with the patch I
posted, I never intended to rip out 8K stacks anytime *soon*.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#25 lore raw
Quoted 7 lines · Jesper Juhl

Please note that I was not trying to remove the 8K stack option right now - heck, I didn't even add anything to feature-removal-schedule.txt
- all I wanted to accomplish with the patch that started this threas was; a) indicate that the 4K option is no longer a debug thing and
b) make 4K stacks the default option in vanilla kernel.org kernels as a gentle nudge towards getting people to start fixing the code paths that are not 4K stack safe.

Definite Acked-by: Rene Herman <[email protected]> where others can decide what that ack's worth.

Rene.

#26 lore raw
In reply to Jesper Juhl
Quoted 4 lines · Jesper Juhl

Please note that I was not trying to remove the 8K stack option right now - heck, I didn't even add anything to feature-removal-schedule.txt
- all I wanted to accomplish with the patch that started this threas was; a) indicate that the 4K option is no longer a debug thing and

Very ACK.

Quoted 3 lines

b) make 4K stacks the default option in vanilla kernel.org kernels as a gentle nudge towards getting people to start fixing the code paths that are not 4K stack safe.

That's the big NACK. It's OK for MM, where things are supposed to be in a not well-tested state, but for running possibly mission-critical systems, you should take no risk.

If you'd run a 4K stack on the NFS+XFS+LVM+dmcrypt+MD+somethingmore setup driving your loved one's life support, you may go ahead.

Signature · 2 lines
I'm a member of DNA (National Assocciation of Dyslexics).
	-- Storm in <[email protected]>
#27 lore raw
Quoted 9 lines · Bodo Eggert
Quoted 4 lines · Jesper Juhl

Please note that I was not trying to remove the 8K stack option right now - heck, I didn't even add anything to feature-removal-schedule.txt
- all I wanted to accomplish with the patch that started this threas was; a) indicate that the 4K option is no longer a debug thing and

Very ACK.

Ok, so perhaps a patch to do just that bit could go towards mainline (possibly after a short stay in -mm).

Quoted 8 lines
Quoted 3 lines

b) make 4K stacks the default option in vanilla kernel.org kernels as a gentle nudge towards getting people to start fixing the code paths that are not 4K stack safe.

That's the big NACK. It's OK for MM, where things are supposed to be in a not well-tested state, but for running possibly mission-critical systems, you should take no risk.

Hmm, I guess it would make sense to stick it in -mm for an extended period of time, but hold off on mainline until we start hearing positive noises that people can no longer break 4K stacks...

Quoted 2 lines

If you'd run a 4K stack on the NFS+XFS+LVM+dmcrypt+MD+somethingmore setup driving your loved one's life support, you may go ahead.

;-)

How about if I split it into two tiny patches
- one that just makes it no longer a debug option and moves the option out of the "kernel hacking" submenu and into "general setup" instead, but leave it to default to N.
- and then one that makes it default Y.
and then submit those two patches to Andrew for inclusion in -mm with a clear note that only the first one should go upstream anytime soon and the second one should be kept "strictly -mm" until we arrive at the point in time where people can no longer break 4K.
How does that sound?

Quoted 1 line

From my point of view that would accomplish my objective of getting 4K

stacks some more testing and the indication that long-term that's the way we probably want to move.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#28 lore raw
In reply to Bodo Eggert
Quoted 7 lines · Bodo Eggert
Quoted 4 lines

Please note that I was not trying to remove the 8K stack option right now - heck, I didn't even add anything to feature-removal-schedule.txt
- all I wanted to accomplish with the patch that started this threas was; a) indicate that the 4K option is no longer a debug thing and

Very ACK.

Quoted 7 lines
Quoted 3 lines

b) make 4K stacks the default option in vanilla kernel.org kernels as a gentle nudge towards getting people to start fixing the code paths that are not 4K stack safe.

That's the big NACK. It's OK for MM, where things are supposed to be in a not well-tested state, but for running possibly mission-critical systems, you should take no risk.

Mission-critical machines are not supposed to have kernel configured with incompetent/careless sysadmin who didn't think about config choices he made at kernel build time.
--
vda

#29 lore raw
Quoted 6 lines · Denis Vlasenko
Quoted 4 lines · Bodo Eggert
Quoted 3 lines

b) make 4K stacks the default option in vanilla kernel.org kernels as a gentle nudge towards getting people to start fixing the code paths that are not 4K stack safe.

Quoted 7 lines
Quoted 3 lines

That's the big NACK. It's OK for MM, where things are supposed to be in a not well-tested state, but for running possibly mission-critical systems, you should take no risk.

Mission-critical machines are not supposed to have kernel configured with incompetent/careless sysadmin who didn't think about config choices he made at kernel build time.

Is it careless to asume good code quality for default options? Does the 4K stack come with a big red warning about crashing the kernel? (I just checked, it does not, only benefits are listed.)
Are 4K stacks so obviously flawed nobody would use them for reliable systems? Or is each sysadmin supposed to read LKML in order to find out about the pitfalls you designed for them?

Signature · 2 lines
Top 100 things you don't want the sysadmin to say:
55. NO!  Not _that_ button!
#30 lore raw
In reply to Ray Lee
Quoted 11 lines · Ray Lee
Quoted 2 lines · Rene Herman

Seeing as how single-page stacks are much easier on the VM so that creating those zillion threads should also be faster, at _some_ percentage we get to say "and now to hell with the rest".

This is the core dispute here. Stated differently, I hope you never design a bridge that I have to drive over.

Correctness first, optimization second. Introducing random and difficult to trace crashes upon an unsuspecting audience of sysadmins and users is not a viable option.

Quite. But unfortunately you didn't actually go into the bit on how given seperate interrupt stacks, available stackspace might not actually _be_ less after selecting CONFIG_4KSTCKS nor into Fedora and RHEL shipping it already.

Quoted 2 lines

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe

I'll do that the minute you prove the current shared 8K stacks are safe. Do we have a deal?

Quoted 3 lines

or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

I'm still waiting for larger soft-pages... does anyone in this thread have a clue on their status?

Rene.

#31 lore raw
Quoted 26 lines · Rene Herman
Quoted 6 lines · Ray Lee
Quoted 3 lines · Rene Herman

Seeing as how single-page stacks are much easier on the VM so that creating those zillion threads should also be faster, at _some_ percentage we get to say "and now to hell with the rest".

Quoted 6 lines

This is the core dispute here. Stated differently, I hope you never design a bridge that I have to drive over.

Correctness first, optimization second. Introducing random and difficult to trace crashes upon an unsuspecting audience of sysadmins and users is not a viable option.

Quite. But unfortunately you didn't actually go into the bit on how given seperate interrupt stacks, available stackspace might not actually _be_ less after selecting CONFIG_4KSTCKS nor into Fedora and RHEL shipping it already.

Quoted 2 lines

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe

I'll do that the minute you prove the current shared 8K stacks are safe. Do we have a deal?

Quoted 6 lines
Quoted 3 lines

or introduce another viable alternative (such as Matt's idea for extending the stack dynamically), then removing the 8k stacks option makes sense.

I'm still waiting for larger soft-pages... does anyone in this thread have a clue on their status?

Given that most x86 users won't want anything to do with them, it's not going to help us at all here.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#32 lore raw
Quoted 7 lines · Matt Mackall
Quoted 2 lines · Rene Herman

I'm still waiting for larger soft-pages... does anyone in this thread have a clue on their status?

Given that most x86 users won't want anything to do with them, it's not going to help us at all here.

No idea why not? Is this something you expect, or know, or... ? (and who are users in this context?)

Rene.

#33 lore raw
Quoted 4 lines · Rene Herman
Quoted 1 line · Matt Mackall

On Tue, Jul 17, 2007 at 12:55:36AM +0200, Rene Herman wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I'm still waiting for larger soft-pages... does anyone in this thread have a clue on their status?

Given that most x86 users won't want anything to do with them, it's not going to help us at all here.

No idea why not? Is this something you expect, or know, or... ? (and who are users in this context?)

Larger soft pages waste tremendous amounts of memory (mostly in page cache) for minimal benefit on, say, the typical desktop. While there are workloads where it's a win, it's probably on a small percentage of machines.

So it's absolutely no help in fixing our order-1 allocation problem because we don't want to force large pages on people.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#34 lore raw
Quoted 6 lines · Matt Mackall

Larger soft pages waste tremendous amounts of memory (mostly in page cache) for minimal benefit on, say, the typical desktop. While there are workloads where it's a win, it's probably on a small percentage of machines.

So it's absolutely no help in fixing our order-1 allocation problem because we don't want to force large pages on people.

Okay. I would've expected that 4K was fairly tiny for today's loads but as usual I'm relatively data challenged so I guess I'll take your word for it. Bummer.

Rene.

#35 lore raw
In reply to Matt Mackall
Quoted 7 lines · Matt Mackall

Larger soft pages waste tremendous amounts of memory (mostly in page cache) for minimal benefit on, say, the typical desktop. While there are workloads where it's a win, it's probably on a small percentage of machines.

So it's absolutely no help in fixing our order-1 allocation problem because we don't want to force large pages on people.

I was just now looking at how much space is in fact wasted in pagecache for various pagesizes by running the attached dumb little program from a few selected directories (heavy stack recursion, never mind).

Well, hmmm. This is on a (compiled) git tree:

rene@7ixe4:~/src/linux/local$ pageslack
total	: 447350347
 4k	: 67738037 (15%)
 8k	: 147814837 (33%)
16k	: 324614581 (72%)
32k	: 724629941 (161%)
64k	: 1592785333 (356%)

Nicely constant factor 2.2 instead of the 2 one would expect but oh well. On a collection of larger files the percentages obviously drop. This is on a directory of ogg vorbis files:

root@7ixe4:/mnt/ogg/.../... # pageslack
total : 70817974
4k : 26442 (0%)
8k : 67402 (0%)
16k : 124746 (0%)
32k : 288586 (0%)
64k : 419658 (0%)

The "typical desktop" is presented by neither I guess but does involve audio and (much larger still) video and bloody huge browser apps.

Not too sure then that 8K wouldn't be something I'd want, given fewer pagefaults and all that...

Rene.

/* gcc -W -Wall -o pageslack pageslack.c */

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <dirent.h>

#define PAGE_SIZE (1UL << PAGE_SHIFT)
#define PAGE_MASK (~(PAGE_SIZE - 1))

unsigned long long total;
unsigned long long slack[5];

void do_dir(const char *name)

{
	DIR *dir; 
	struct dirent *ent;

	dir = opendir(name);
	if (!dir) {
		perror("opendir");
		exit(EXIT_FAILURE);
	}
	while ((ent = readdir(dir))) {
		struct stat buf;
		char path[PATH_MAX];

		if (!strcmp(ent->d_name, "."))
			continue;
		if (!strcmp(ent->d_name, ".."))
			continue;

		sprintf(path, "%s/%s", name, ent->d_name);
		if (stat(path, &buf)) {
			perror("stat");
			exit(EXIT_FAILURE);
		}
		if (S_ISDIR(buf.st_mode)) {
			do_dir(path);
			continue;
		}
		if (S_ISREG(buf.st_mode)) {
			int i;

			for (i = 0; i < 5; i++) {
				unsigned long PAGE_SHIFT = 12 + i;
				slack[i] += (PAGE_SIZE - (buf.st_size % PAGE_SIZE)) % PAGE_SIZE;
			}
			total += buf.st_size;
		}
	}
	if (closedir(dir)) {
		perror("closedir");
		exit(EXIT_FAILURE);
	}
}

int main(void)

{
	do_dir(".");
	printf("total\t: %llu\n", total);
	printf(" 4k\t: %llu (%llu%%)\n", slack[0], (100 * slack[0]) / total);
	printf(" 8k\t: %llu (%llu%%)\n", slack[1], (100 * slack[1]) / total);
	printf("16k\t: %llu (%llu%%)\n", slack[2], (100 * slack[2]) / total);
	printf("32k\t: %llu (%llu%%)\n", slack[3], (100 * slack[3]) / total);
	printf("64k\t: %llu (%llu%%)\n", slack[4], (100 * slack[4]) / total);
	return EXIT_SUCCESS;
}
#36 lore raw
Quoted 36 lines · Rene Herman
Quoted 7 lines · Matt Mackall

Larger soft pages waste tremendous amounts of memory (mostly in page cache) for minimal benefit on, say, the typical desktop. While there are workloads where it's a win, it's probably on a small percentage of machines.

So it's absolutely no help in fixing our order-1 allocation problem because we don't want to force large pages on people.

I was just now looking at how much space is in fact wasted in pagecache for various pagesizes by running the attached dumb little program from a few selected directories (heavy stack recursion, never mind).

Well, hmmm. This is on a (compiled) git tree:

rene@7ixe4:~/src/linux/local$ pageslack
total	: 447350347
 4k	: 67738037 (15%)
 8k	: 147814837 (33%)
16k	: 324614581 (72%)
32k	: 724629941 (161%)
64k	: 1592785333 (356%)

Nicely constant factor 2.2 instead of the 2 one would expect but oh well. On a collection of larger files the percentages obviously drop. This is on a directory of ogg vorbis files:

root@7ixe4:/mnt/ogg/.../... # pageslack
total : 70817974
4k : 26442 (0%)
8k : 67402 (0%)
16k : 124746 (0%)
32k : 288586 (0%)
64k : 419658 (0%)

Quoted 2 lines

The "typical desktop" is presented by neither I guess but does involve audio and (much larger still) video and bloody huge browser apps.

I'd be surprised if a user had substantially more than one OGG, video, or browser in memory at one time. In fact, you're likely to find only a fraction of each of those in memory at any given time.

Meanwhile, they're likely to have thousands of small browser cache, thumbnail, config, icon, maildir, etc. files in cache. And hundreds of medium-sized libraries, utilities, applications, and so on.

You can expect the distribution of file sizes to follow a gamma distribution, with a large hump towards the small end of the spectrum around 1-10K, dropping off very rapidly as file sizes grow.

Quoted 2 lines

Not too sure then that 8K wouldn't be something I'd want, given fewer pagefaults and all that...

Fewer minor pagefaults, perhaps. Readahead already deals with most of the major pagefaults that larger pages would.

Anyway, raising the systemwide memory overhead by up to 15% seems an awfully silly way to address the problem of not being able to allocate a stack when you're down to your last 1 or 2% of memory! In all likelihood, we'll fail sooner because we're completely OOM.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#37 lore raw
Quoted 3 lines · Matt Mackall

You can expect the distribution of file sizes to follow a gamma distribution, with a large hump towards the small end of the spectrum around 1-10K, dropping off very rapidly as file sizes grow.

Okay.

Quoted 4 lines
Quoted 1 line

Not too sure then that 8K wouldn't be something I'd want, given fewer pagefaults and all that...

Fewer minor pagefaults, perhaps. Readahead already deals with most of the major pagefaults that larger pages would.

Mmm, yes.

Quoted 2 lines

Anyway, raising the systemwide memory overhead by up to 15% seems an awfully silly way to address the problem of not being able to allocate a stack when you're down to your last 1 or 2% of memory!

Well, I've seen larger pagesizes submerge in more situations, specifically in allocation overhead -- ie, making the struct page's fit in lowmem for hugemem x86 boxes was the first I heard of it. But yes, otherwise (also) mostly database loads which obviously have moved to 64-bit since.

Pagecache tail-packing seems like a promising idea to deal with the downside of larger pages but I'll admit I'm not particularly sure how many _up_ sides to them are left on x86 (not -64) now that's becoming a legacy architecture (and since you just shot down the pagefaults thing).

Rene.

#38 lore raw
In reply to Matt Mackall
Quoted 2 lines · Matt Mackall

So it's absolutely no help in fixing our order-1 allocation problem because we don't want to force large pages on people.

Using kmalloc(8k) instead of alloc_page() doesn't sound a too big deal and that will solve the problem. The whole idea is to avoid the memcpy
+ pte mangling of defrag while hopefully lowering cpu utilization in allocations at the same time.

About 4k stacks I was generally against them, much better to fail in fork than to risk corruption. The per-irq stack part is great feature instead (too bad it wasn't enabled for the safer 8k stacks).

Failing in a do_no_page with variable order page size allocation is a fatal event (the task will be killed), failing in fork is graceful, userland can retry etc... Fork can fail for different reasons, ulimit itself is the most likely source of fork failures. I don't think the 8k stacks have ever been a problem, yes you will run out of stack sooner (sooner also because the 4k stacks takes less memory) but nothing is terribly wrong if the 8k allocation fails.

#39 lore raw
Quoted 3 lines

About 4k stacks I was generally against them, much better to fail in fork than to risk corruption. The per-irq stack part is great feature instead (too bad it wasn't enabled for the safer 8k stacks).

8K stacks without IRQ stacks are not "safer" so I don't understand your comment ?

#40 lore raw
Quoted 6 lines · Alan Cox
Quoted 3 lines

About 4k stacks I was generally against them, much better to fail in fork than to risk corruption. The per-irq stack part is great feature instead (too bad it wasn't enabled for the safer 8k stacks).

8K stacks without IRQ stacks are not "safer" so I don't understand your comment ?

Ouch, see the reports about 4k stack crashes. I agree they're not safe w/o irq stacks (like on x86-64), but they're generally safer.

#41 lore raw
Quoted 10 lines · Andrea Arcangeli
Quoted 6 lines · Alan Cox
Quoted 3 lines

About 4k stacks I was generally against them, much better to fail in fork than to risk corruption. The per-irq stack part is great feature instead (too bad it wasn't enabled for the safer 8k stacks).

8K stacks without IRQ stacks are not "safer" so I don't understand your comment ?

Ouch, see the reports about 4k stack crashes. I agree they're not safe w/o irq stacks (like on x86-64), but they're generally safer.

Here's a way to make forward progress on this whole thing:

Turn on irqstacks when using 8k stacks
Detect when usage with 8k stacks would overrun a 4k stack when doing
our stack switch and do a WARN_ONCE
Fix up the damn bugs

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#42 lore raw
Quoted 3 lines · Matt Mackall

Here's a way to make forward progress on this whole thing:

Turn on irqstacks when using 8k stacks

WLI: are you submitting? Makes great sense regardless of anything and they've been tested silly with 4KSTACKS already...

Quoted 2 lines

Detect when usage with 8k stacks would overrun a 4k stack when doing our stack switch and do a WARN_ONCE

Our stack switch?

Quoted 1 line

Fix up the damn bugs

DM ofcourse is fairly "layered-by-design" so I _hope_ they can be classified simple bugs...

Rene.

#43 lore raw
In reply to Matt Mackall
Quoted 1 line · Matt Mackall

Turn on irqstacks when using 8k stacks

Indeed.

Quoted 3 lines

Detect when usage with 8k stacks would overrun a 4k stack when doing
our stack switch and do a WARN_ONCE
Fix up the damn bugs

I don't think they're necessarily bugs. IMHO the WARN_ON is better off at 7k level like it is today with the current STACK_WARN. 4k for a stack for common code really is small. I doubt you're going to find obvious culprits that way, more likely you'll have to mangle the code to call kmalloc for fairly small structures which isn't necessarily a good thing in the long term. It comes to mind the folio ptes array that Hugh allocated on the stack in his large PAGE_SIZE patch of jul 2001, that thing like any other local array, would need to be kmalloced with a 4k stack. With 4k I'm afraid you better not use the stack for anything but pointers, especially if you run in common code that may invoke I/O like that.

#44 lore raw
Quoted 3 lines

I don't think they're necessarily bugs. IMHO the WARN_ON is better off at 7k level like it is today with the current STACK_WARN. 4k for a stack for common code really is small. I doubt you're going to find

You want the limit settable. On a production system you want to set the limit to somewhere appropriate for the stack size used. When debugging (eg to remove any last few bogus users of 8K stack space) you want to be able to set it to just under 4K

Alan

#45 lore raw
Quoted 8 lines · Alan Cox
Quoted 3 lines

I don't think they're necessarily bugs. IMHO the WARN_ON is better off at 7k level like it is today with the current STACK_WARN. 4k for a stack for common code really is small. I doubt you're going to find

You want the limit settable. On a production system you want to set the limit to somewhere appropriate for the stack size used. When debugging (eg to remove any last few bogus users of 8K stack space) you want to be able to set it to just under 4K

Hm, when cramming cxfs into 4k at sgi, I had a patch that did just that for debugging (warn about encroaching on 4k without actually tipping over, with a settable threshold...)

Maybe I should resurrect it & send it out...

(FWIW I think I recall that the warning itself sometimes tipped the scales enough on 4k stacks to bring the box down)

-eric

#46 lore raw
Quoted 4 lines

Maybe I should resurrect it & send it out...

(FWIW I think I recall that the warning itself sometimes tipped the scales enough on 4k stacks to bring the box down)

You can always switch stack for the printk and it probably should panic at that point and give a trace then die as that is what we are trying to prove does not occur

#47 lore raw
Quoted 1 line · Alan Cox
Quoted 1 line

Maybe I should resurrect it & send it out...

Hmm, something that hooks in not only at do_IRQ time (as the present in-mainline stackoverflow check thing)?

Quoted 6 lines
Quoted 2 lines

(FWIW I think I recall that the warning itself sometimes tipped the scales enough on 4k stacks to bring the box down)

You can always switch stack for the printk and it probably should panic at that point and give a trace then die as that is what we are trying to prove does not occur

Yes, only yesterday I saw exactly this happening DEBUG_STACKOVERFLOW when doing a udf -> pktcdvd -> cdrom -> ide_cd thing. It's one of those reproducible will-crash-4k-stacks tests, especially if you have debug stuff enabled in your build that would make on-stack structures (where such exist on the codepath) a bit heavier.

Admittedly, what seems to have happened is a bit pathological:

[  481.836378] cdrom: entering cdrom_count_tracks
[  481.844266] BUG: sleeping function called from invalid context at
include/asm/semaphore.h:98
[  481.844434] do_IRQ: stack overflow: 164
[  481.844540]  [<c0405cfe>] show_trace_log_lvl+0x19/0x2e
[  481.844707]  [<c0405dfe>] show_trace+0x12/0x14
[  481.844867]  [<c0405e14>] dump_stack+0x14/0x16
[  481.845027]  [<c0406ff6>] do_IRQ+0x7b/0xe1
[  481.845186]  [<c040583e>] common_interrupt+0x2e/0x34
[  481.845348]  [<c042b8e7>] printk+0x1b/0x1d
[  481.845507]  [<c0422c05>] __might_sleep+0x81/0xdc
[  481.845668]  [<c066d869>] __reacquire_kernel_lock+0x2d/0x4f
[  481.845833]  [<c066b09b>] schedule+0x78a/0x7a4
[  481.845996]  [<c066b538>] wait_for_completion+0x72/0x97
[  481.846160]  [<c05937a6>] ide_do_drive_cmd+0xeb/0x109
[  481.846324]  [<f89172a2>] cdrom_queue_packet_command+0x40/0xc5 [ide_cd]
[  481.846503]  [<f89175b7>] ide_cdrom_packet+0x86/0xa4 [ide_cd]
[  481.846669]  [<f8854dc1>] cdrom_get_disc_info+0x48/0x87 [cdrom]
[  481.846839]  [<f8854ec6>] cdrom_get_last_written+0x2a/0xfe [cdrom]
[  481.847009]  [<f891831b>] cdrom_read_toc+0x39d/0x3f3 [ide_cd]
[  481.847231]  [<f8918e7e>] ide_cdrom_audio_ioctl+0x130/0x1ce [ide_cd]
[  481.847414]  [<f8854123>] cdrom_count_tracks+0x5c/0x126 [cdrom]
[  481.847583]  [<f8855688>] cdrom_open+0x147/0x79c [cdrom]
[  481.847748]  [<f891799a>] idecd_open+0x75/0x8a [ide_cd]
[  481.847912]  [<c04aac0e>] do_open+0x1d1/0x284
[  481.848079]  [<c04aad89>] __blkdev_get+0x73/0x7e
[  481.848242]  [<c04aada9>] blkdev_get+0x15/0x17
[  481.848411]  [<f8b34b6b>] pkt_open+0x99/0xc6e [pktcdvd]
[  481.848583]  [<c04aaad3>] do_open+0x96/0x284
[  481.848745]  [<c04aad89>] __blkdev_get+0x73/0x7e
[  481.848910]  [<c04aada9>] blkdev_get+0x15/0x17

(... the trace cut off there, and then the box froze hard, no sysrq ...)

The mount(2) hit the wait_for_completion() in ide_do_drive_cmd(), little stack was left at this point. But then I have no idea why the __reacquire_kernel_lock() from schedule() gave a might_sleep() there, the code in sched.c and kernel_lock.c looks obviously correct -- the down(&kernel_sem) only happens with both irqs and preemption on.

Anyway, the second line of printk() in __might_sleep (the one that tells us in_atomic() and irqs_disabled()) was about to be printed when an interrupt decided to join the fun. do_IRQ() comes in, with debug stackoverflows on, it notices that only 164 bytes worth of stack is left and decides to dump_stack ... and while we were doing just that, we died. (this was 2.6.23-rc1-mm1)

#48 lore raw
Quoted 5 lines · Satyam Sharma
Quoted 1 line · Alan Cox
Quoted 1 line

Maybe I should resurrect it & send it out...

Hmm, something that hooks in not only at do_IRQ time (as the present in-mainline stackoverflow check thing)?

No, what I had did only that, so it was still a matter of probabilities...

-Eric

#49 lore raw
Quoted 1 line · Eric Sandeen

No, what I had did only that, so it was still a matter of probabilities...

How expensive would it be to allocate two , then use the MMU mark the second page unwritable? Hardware wise it should be possible, (for constant 4k pagesizes, I have not worked with variable pagesize MMUs) and since it's a per-context-switch constant operation, it would be a special case in the fault handler rather then adding another entry to the VM for every process.

Using large hardware pages to cover the kernel mapping could be worked around by leaving the area where the current process stack resides mapped via 4k pages. Of course, I haven't touched a modern PC MMU in ages, so I could be missing something fundamentally difficult.

The other issue is with the layered IO design - no matter what we configure the stack size to, it is still possible to create a set of translation layers that will cause it to crash regularly: XFS on dm_crypt on loop on XFS on dm_crypt on loop on ad infinitum.

That said, I'm missing something here - why is the stack growing? Filesystems should be issuing bios with callbacks, so they should be
back off the stack, same with dm, loop, etc. Am I missing step where
they use a wrapper function that pretends to be syncronous?

#50 lore raw
Quoted 5 lines · On Wednesday August 1, [email protected] wrote:

The other issue is with the layered IO design - no matter what we configure the stack size to, it is still possible to create a set of translation layers that will cause it to crash regularly: XFS on dm_crypt on loop on XFS on dm_crypt on loop on ad infinitum.

No, this does not use indefinite stack.

loop will schedule each request to be handled by a kernel thread, so requests to 'loop' are serialised, never stacked.

In 2.6.22, generic_make_request detects and serialises recursive calls, so unlimited recursion is not possible there either.

It is still possible to do
dm on dm on dm on dm on md on md on md on md

and calls to ->issue_flush_fn or ->unplug_fn could use an arbitrarily large amount of stack. But the stack usage of each stage is very small so it is unlikely to be a problem (though it should still be fixed).

NeilBrown

#51 lore raw
Quoted 7 lines · Neil Brown

No, this does not use indefinite stack.

loop will schedule each request to be handled by a kernel thread, so requests to 'loop' are serialised, never stacked.

In 2.6.22, generic_make_request detects and serialises recursive calls, so unlimited recursion is not possible there either.

Is that saying "before 2.6.22, a read/write on a deeply layered device would use a lot of stack?"

#52 lore raw
Quoted 4 lines · On Friday August 10, [email protected] wrote:
Quoted 2 lines · Neil Brown

No, this does not use indefinite stack.

Quoted 8 lines
Quoted 5 lines

loop will schedule each request to be handled by a kernel thread, so requests to 'loop' are serialised, never stacked.

In 2.6.22, generic_make_request detects and serialises recursive calls, so unlimited recursion is not possible there either.

Is that saying "before 2.6.22, a read/write on a deeply layered device would use a lot of stack?"

before 2.6.22, a stack of dm and/or md devices (not loop, and not md/raid0 or md/linear) would use more stack the more devices were involved. If you made a very deep stack, you could push the stack over any limit you chose.

I won't say "a lot of stack" as I haven't measured the exact amount, just "more stack as you add more devices".

NeilBrown

#53 lore raw
In reply to Dan Merillat
Quoted 2 lines · Dan Merillat

How expensive would it be to allocate two , then use the MMU mark the second page unwritable? Hardware wise it should be possible, (for

Tweaking kernel ptes is prohibitive during clone() because that's kernel memory and it would require a flush tlb all with IPIs that won't scale (IPIs are really the blocker). Basically vmalloc already does what you suggest with the gap page and yet we can't use it for performance reasons. Kernel stack should be readable by any context to allow sysrq+t kind of things, so I doubt it's feasible to do tricks to avoid ipis.

#54 lore raw
Quoted 7 lines · On Wed, 1 Aug 2007 15:33:58 +0200 Andrea Arcangeli <[email protected]> wrote:
Quoted 2 lines · Dan Merillat

How expensive would it be to allocate two , then use the MMU mark the second page unwritable? Hardware wise it should be possible, (for

Tweaking kernel ptes is prohibitive during clone() because that's kernel memory and it would require a flush tlb all with IPIs that won't scale (IPIs are really the blocker)

Agreed - except when doing debug work then its an acceptable cost. You still have to sort the debug side out because you are going to fault the kernel stack which will probably then cause a triple fault and reboot on the spot.

#55 lore raw
Quoted 10 lines · Alan Cox
Quoted 3 lines · On Wed, 1 Aug 2007 15:33:58 +0200 Andrea Arcangeli <[email protected]> wrote:

Tweaking kernel ptes is prohibitive during clone() because that's kernel memory and it would require a flush tlb all with IPIs that won't scale (IPIs are really the blocker)

Agreed - except when doing debug work then its an acceptable cost. You still have to sort the debug side out because you are going to fault the kernel stack which will probably then cause a triple fault and reboot on the spot.

I was assuming debugging work, yes. I was also thinking it wouldn't be done at clone() time, but mapped (on a single CPU) at the time of a context switch. It would eliminate IPI, but would probably make the
rest of the TLB handling much too ugly to contemplate. As an
alternative, could the TLB flush and associated IPI be deferred until
the process migrates? First migration would trigger flush/IPI,
further migration would be as now, no? I'd happily run it with
various dm/md layers underneath

Quoted 2 lines · Denis Vlasenko

Hmm, neat. Why do you need to _allocate second page_ at all? Just mark it "not present"...

Because the kernel mapping covers all physical memory contiguously, so if the page isn't allocated, it could be used by a kernel data structure you need to access. Same reason the kernel stack has to be
contiguous pages. Well, for non-highmem at least. Either way, you
don't want to mark an in-use page as inaccessable, you never know what's under there.

#56 lore raw
In reply to Andrea Arcangeli
Quoted 5 lines · On Thu, 19 Jul 2007 03:33:58 +0200 Andrea Arcangeli <[email protected]> wrote:
Quoted 2 lines

8K stacks without IRQ stacks are not "safer" so I don't understand your comment ?

Ouch, see the reports about 4k stack crashes. I agree they're not safe w/o irq stacks (like on x86-64), but they're generally safer.

Still don't follow. How is "exceeds stack space but less likely to be noticed" safer.

Alan

#57 lore raw
Quoted 2 lines · Alan Cox

Still don't follow. How is "exceeds stack space but less likely to be noticed" safer.

Statistically speaking it clearly is. The reason is probably that the irq theoretical issue happens only on large boxes with lots of reentrant irqs. Not all irqs are reentrant, not all systems runs lots of irqs at the same time etc..

#58 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 3 lines

About 4k stacks I was generally against them, much better to fail in fork than to risk corruption. The per-irq stack part is great feature instead (too bad it wasn't enabled for the safer 8k stacks).

8K stacks without IRQ stacks are not "safer" so I don't understand your comment ?

Hmm was it SuSE or RH kernels (or mainline?) I saw which had a test to defer soft IRQs if they occurred too deep in the stack for the current thread.

-Eric

#59 lore raw
Quoted 6 lines · Eric Sandeen
Quoted 2 lines

8K stacks without IRQ stacks are not "safer" so I don't understand your comment ?

Hmm was it SuSE or RH kernels (or mainline?) I saw which had a test to defer soft IRQs if they occurred too deep in the stack for the current thread.

Perhaps the "8 KB softpage" should be an option instead of 8 KB stack size?

Not sure about ABI compatibility.

Signature · 1 line
Krzysztof Halasa
#60 lore raw
In reply to Andrea Arcangeli
Quoted 6 lines · Andrea Arcangeli
Quoted 2 lines · Matt Mackall

So it's absolutely no help in fixing our order-1 allocation problem because we don't want to force large pages on people.

Using kmalloc(8k) instead of alloc_page() doesn't sound a too big deal and that will solve the problem.

How do you figure?

If you're saying that soft pages helps our 8k stack allocations, it doesn't. The memory overhead of soft pages will be higher (5-15%, mostly due to file tails in pagecache) than the level at which 8k stacks currently run into trouble (1-2% free?).

Not helpful.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#61 lore raw
Quoted 5 lines · Matt Mackall
Quoted 2 lines · Andrea Arcangeli

Using kmalloc(8k) instead of alloc_page() doesn't sound a too big deal and that will solve the problem.

Quoted 8 lines

How do you figure?

If you're saying that soft pages helps our 8k stack allocations, it doesn't. The memory overhead of soft pages will be higher (5-15%, mostly due to file tails in pagecache) than the level at which 8k stacks currently run into trouble (1-2% free?).

Not helpful.

With tail-packing it is.

Rene.

#62 lore raw
Quoted 9 lines · Rene Herman
Quoted 7 lines · Matt Mackall
Quoted 2 lines · Andrea Arcangeli

Using kmalloc(8k) instead of alloc_page() doesn't sound a too big deal and that will solve the problem.

How do you figure?

Quoted 8 lines
Quoted 6 lines

If you're saying that soft pages helps our 8k stack allocations, it doesn't. The memory overhead of soft pages will be higher (5-15%, mostly due to file tails in pagecache) than the level at which 8k stacks currently run into trouble (1-2% free?).

Not helpful.

With tail-packing it is.

Tail packing is a whole new can of worms. Especially as it's very likely to make performance suffer on small files (the common case).

On the other hand, if someone can demonstrate that tail-packed page cache doesn't suck, we should put it in mainline pronto. The poor architectures that are stuck with real 64k pages are sure to appreciate it.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#63 lore raw
In reply to Rene Herman
Quoted 6 lines · Rene Herman
Quoted 2 lines · Ray Lee

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe

I'll do that the minute you prove the current shared 8K stacks are safe. Do we have a deal?

Language barrier, I think, or perhaps I was unclear. Please read that as "4k stacks introduce no new stack bugs." And if we put wli's unconditional interrupt stacks into the kernel, it's pretty obvious that 8k stacks are at least as safe in that case as 4k stacks.

Given that there's actual, y'know, reports of people who can easily crash a 4k+interrupt stacks kernel, and not an 8k one, I think the current evidence speaks for itself.

The point remains that the burden of proof of the safety of the 4k only option is upon those people who want to remove the 8k option.

#64 lore raw
Quoted 3 lines · Ray Lee

Given that there's actual, y'know, reports of people who can easily crash a 4k+interrupt stacks kernel, and not an 8k one, I think the current evidence speaks for itself.

Where?

Quoted 2 lines

The point remains that the burden of proof of the safety of the 4k only option is upon those people who want to remove the 8k option.

Removing any such option was not the objective of this thread, just lifting 4K stacks from debug and making it the default. People fortunate enough to use workloads where some piece of crap code by accident works more often with the current shared 8K stacks then it does with the unshared 4K stacks can then still nicely not select it (or fix the code if possible).

Rene.

#65 lore raw
Quoted 8 lines · Rene Herman
Quoted 3 lines · Ray Lee

Given that there's actual, y'know, reports of people who can easily crash a 4k+interrupt stacks kernel, and not an 8k one, I think the current evidence speaks for itself.

Where?

Quoted 9 lines
Quoted 2 lines

The point remains that the burden of proof of the safety of the 4k only option is upon those people who want to remove the 8k option.

Removing any such option was not the objective of this thread, just lifting 4K stacks from debug and making it the default. People fortunate enough to use workloads where some piece of crap code by accident works more often with the current shared 8K stacks then it does with the unshared 4K stacks can then still nicely not select it (or fix the code if possible).

I may have worded the initial email a bit too heavily towards 8K removal, but that was mainly to provoke some discussion. Nothing should get removed without a fair (and long) warning in feature-removal-schedule.txt and ofcourse not before we know that the new option is at least as safe as what we intend to remove, so the patch really was just intended as a fairly harmless nudge towards getting 4K stacks into a shape where we can eventually start considering 8K removal.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#66 lore raw
In reply to Rene Herman
Quoted 6 lines · Rene Herman
Quoted 3 lines · Ray Lee

Given that there's actual, y'know, reports of people who can easily crash a 4k+interrupt stacks kernel, and not an 8k one, I think the current evidence speaks for itself.

Where?

The second message in this thread, according to my reader, from Zan Lynx. Another from Utz Lehmann a few minutes ago.

Quoted 5 lines
Quoted 2 lines

The point remains that the burden of proof of the safety of the 4k only option is upon those people who want to remove the 8k option.

Removing any such option was not the objective of this thread, just lifting 4K stacks from debug and making it the default.

True, but your messages are reading as advocacy for removing the 8k option. I'm saying that's a bad idea. If I misunderstood your position, then my bad.

Quoted 4 lines

People fortunate enough to
use workloads where some piece of crap code by accident works more often with the current shared 8K stacks then it does with the unshared 4K stacks can then still nicely not select it (or fix the code if possible).

If they even realize that it's the cause of the problem. In the meantime, we're generating more bug reports to lkml. As the general opinion is that the ones getting received now aren't getting enough attention (see regression tracking threads), setting a default that is known to break setups in hard to debug ways seems counterproductive.

#67 lore raw
Quoted 11 lines · Ray Lee
Quoted 7 lines · Rene Herman
Quoted 3 lines · Ray Lee

Given that there's actual, y'know, reports of people who can easily crash a 4k+interrupt stacks kernel, and not an 8k one, I think the current evidence speaks for itself.

Where?

The second message in this thread, according to my reader, from Zan Lynx. Another from Utz Lehmann a few minutes ago.

True enough. I'm rather wondering though why RHEL is shipping with it if it's a _real_ problem. Scribbling junk all over kernel memory would be the kind of thing I'd imagine you'd mightely piss-off enterprise customers with. But well, sure, that rather quickly becomes a self-referential argument I guess.

Quoted 5 lines
Quoted 1 line

Removing any such option was not the objective of this thread, just lifting 4K stacks from debug and making it the default.

True, but your messages are reading as advocacy for removing the 8k option. I'm saying that's a bad idea. If I misunderstood your position, then my bad.

I personally believe that CONFIG_4KSTACKS is not better only for very few users but well, no, if even some users exist, then I wouldn't want to suggest they'd be disallowed (shared or unshared) 8K stacks.

I _would_ in fact suggest there are few enough left that rather than 4K, 8K should really be the option that only those few would select, but ofcourse, given config defaults that's mostly a matter of semantics, so who cares in the end.

Quoted 5 lines

If they even realize that it's the cause of the problem. In the meantime, we're generating more bug reports to lkml. As the general opinion is that the ones getting received now aren't getting enough attention (see regression tracking threads), setting a default that is known to break setups in hard to debug ways seems counterproductive.

Well, no. "oldconfig" works fine, and other than that, all failure modes I've heard about also in this thread are MD/LVM/XFS. This is extremely widely tested stuff in at least Fedora and RHEL. "hard to debug" is simply not the case -- every one will immediately start yelling 4KSTACKS when that software-stack appears anywhere.

Don't try and hang this off generic development unease... ;-)

Rene.

#68 lore raw
Quoted 5 lines · Rene Herman

True enough. I'm rather wondering though why RHEL is shipping with it if it's a _real_ problem. Scribbling junk all over kernel memory would be the kind of thing I'd imagine you'd mightely piss-off enterprise customers with. But well, sure, that rather quickly becomes a self-referential argument I guess.

I can't speak for Fedora, but RHEL disables XFS in their kernel likely because it is known to cause problems with 4K stacks.

Quoted 3 lines

Well, no. "oldconfig" works fine, and other than that, all failure modes I've heard about also in this thread are MD/LVM/XFS. This is extremely widely tested stuff in at least Fedora and RHEL.

Again don't assume that because Fedora and RHEL have 4K stacks means that MD/LVM/XFS is widely tested.

Additionally I think I should point out that the problems pointed out so far are not the only problem areas with 4K stacks. There are out of tree drivers to consider as well, and use cases like ndiswrapper.

#69 lore raw
Quoted 2 lines · Shawn Bohrer

I can't speak for Fedora, but RHEL disables XFS in their kernel likely because it is known to cause problems with 4K stacks.

Okay. So is it fair to say it's largely XFS that's the problem? No problems with LVM/MD and say plain ext? If that's the case, I believe it could be concluded that it's not something in any sense fundamentally unfixable and the question becomes why XFS isn't fixed...

Quoted 6 lines
Quoted 3 lines

Well, no. "oldconfig" works fine, and other than that, all failure modes I've heard about also in this thread are MD/LVM/XFS. This is extremely widely tested stuff in at least Fedora and RHEL.

Again don't assume that because Fedora and RHEL have 4K stacks means that MD/LVM/XFS is widely tested.

No, quite, that specific combination was reported in this thread alone 3 times again, so that one's clear, but _other_ than that, I've heard of no other failure modes.

Quoted 3 lines

Additionally I think I should point out that the problems pointed out so far are not the only problem areas with 4K stacks. There are out of tree drivers to consider as well, and use cases like ndiswrapper.

Except these. Good to have pointed out, thanks, but as far as I'm concerned both these cases do not get a say in what's default configuration for the kernel.org kernel. They might get a say in what's removed or not removed from that kernel but that's not under discussion at the moment (nor would I expect it to be anytime soon if ever).

Rene.

#70 lore raw
Quoted 7 lines · Rene Herman
Quoted 2 lines · Shawn Bohrer

I can't speak for Fedora, but RHEL disables XFS in their kernel likely because it is known to cause problems with 4K stacks.

Okay. So is it fair to say it's largely XFS that's the problem? No problems with LVM/MD and say plain ext?

There *are* crashes from LVM and ext3. I had to change kernels to avoid them.

I had crashes with ext3 on LVM snapshot on DM mirror on SATA.

Signature · 1 line
Zan Lynx <[email protected]>
#71 lore raw
In reply to Shawn Bohrer
Quoted 2 lines

I can't speak for Fedora, but RHEL disables XFS in their kernel likely because it is known to cause problems with 4K stacks.

-was- - the SGI folks submitted patches to deal with some gcc problems with stack usage.

#72 lore raw
In reply to Rene Herman
Quoted 8 lines · Rene Herman
Quoted 4 lines · Ray Lee

On 7/16/07, Rene Herman <[email protected]> wrote:

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe

I'll do that the minute you prove the current shared 8K stacks are safe. Do we have a deal?

You claim 4k+4k is safe, therefore 8k must be safe, too. But if 8k is safe, this does not yet prove that you can store 5k+3k in 4k+4k.

Signature · 3 lines
Funny quotes:
38. Last night I played a blank tape at full blast. The mime next door went
    nuts.
#73 lore raw
Quoted 8 lines · Bodo Eggert
Quoted 5 lines · Rene Herman
Quoted 1 line · Ray Lee

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe

I'll do that the minute you prove the current shared 8K stacks are safe. Do we have a deal?

You claim 4k+4k is safe, therefore 8k must be safe, too.

No, I most certainly do not. I claim proving that 4K and seperate (per cpu) interrupt stacks are safe are exactly the same as proving unshared 8K stacks are safe. That is, you don't, no such proof exists other than in the eating of the pudding. Ray (and you) in considering !CONFIG_4KSTACKS to be "safer" than CONFIG_4KSTACKS suggest that _inevitably_ CONFIG_4KSTACKS would leave you with less available stack and I pointed out this isn't be the case.

And in fact, I shouldn't have said "exactly" the same. Unshared interrupt stacks make for more determistisc behaviour, so you'd have a harder time proven anything to some set limit of uncertainty with the shared 8K stacks than with the unshared 4K stacks.

Quoted 2 lines

But if 8k is safe, this does not yet prove that you can store 5k+3k in 4k+4k.

I really have not made any claim of the kind. The argument is that with CONFIG_4KSTACKS, availeble stack space isn't inevitably less at any point in time.

Rene.

#74 lore raw
Quoted 9 lines · Rene Herman
Quoted 8 lines · Bodo Eggert
Quoted 6 lines · Rene Herman
Quoted 2 lines · Ray Lee

If at some point one of the pro-4k stacks crowd can prove that all code paths are safe

I'll do that the minute you prove the current shared 8K stacks are safe. Do we have a deal?

Quoted 6 lines
Quoted 1 line

You claim 4k+4k is safe, therefore 8k must be safe, too.

No, I most certainly do not. I claim proving that 4K and seperate (per cpu) interrupt stacks are safe are exactly the same as proving unshared 8K stacks are safe. That is, you don't, no such proof exists other than in the eating of the pudding.

And yet you have a more strict claim than I do. If you are right, I'll be right, too, because two times less-than-4K is less tham 8K. If I'm wrong and 8K is not enough, you must be wrong, too, because you can impossibly fit more than 8K into 4K+4K. That's the law of mathematics.

Quoted 3 lines

Ray (and you) in considering !CONFIG_4KSTACKS to be "safer" than CONFIG_4KSTACKS suggest that _inevitably_ CONFIG_4KSTACKS would leave you with less available stack and I pointed out this isn't be the case.

Why do you insist on 4Kstacks being good as long as there is _one_ usevase not crashing the kernel? _All_ usecases have to be safe!

Quoted 2 lines

And in fact, I shouldn't have said "exactly" the same. Unshared interrupt stacks

, which are a completely different thing which was bundled to 4K-stacks
because you need more than 4K,

Quoted 3 lines

make for more determistisc behaviour, so you'd have a harder time proven anything to some set limit of uncertainty with the shared 8K stacks than with the unshared 4K stacks.

I don't want my stack to overflow in order to be theoretically able to prove it does not overflow. I'd rather go for 8K+4K-stacks, and if _you_ have done the proof _you_ wanted to make, we can talk again about 4K-stacks. Then I'll just add up the maximum stack usages and have the proof that 8K stacks are safe.

Quoted 6 lines
Quoted 2 lines

But if 8k is safe, this does not yet prove that you can store 5k+3k in 4k+4k.

I really have not made any claim of the kind. The argument is that with CONFIG_4KSTACKS, availeble stack space isn't inevitably less at any point in time.

I claim, you can store 5k + 3k on the 8k stack, where 5k is something like the current worst case for non-interrupt stack and 3k is plenty for interrupts. Thousands of stable systems with 8K stacks support my claim.

You claimed with 4k + 4k, there is not less available stack space. (At least for usecases you are interested in, but I'll asume you don't
want other usecases to crash.)

If you were right, I'd have enough space on 4k + 4k to store that 5k. Obviously, thousands of systems disagree by crashing with 4K-stacks. That's most simple logic.

Off cause I may be wrong and the kernels don't crash because of 4K stacks, but because of bad karma ... But even then, you'd first have to get rid of that bad karma before defaulting to 4K stacks.

Signature · 2 lines
Top 100 things you don't want the sysadmin to say:
41. OH, SH*T! (as they scrabble at the keyboard for ^c).
#75 lore raw
Quoted 9 lines · Bodo Eggert
Quoted 5 lines · Rene Herman
Quoted 1 line · Bodo Eggert

You claim 4k+4k is safe, therefore 8k must be safe, too.

No, I most certainly do not. I claim proving that 4K and seperate (per cpu) interrupt stacks are safe are exactly the same as proving unshared 8K stacks are safe. That is, you don't, no such proof exists other than in the eating of the pudding.

And yet you have a more strict claim than I do. If you are right, I'll be right, too, because two times less-than-4K is less tham 8K.

Firstly, it's not two times 4K but 4K + (4K + 4K) * NR_CPUS. Secondly, _you_ are the one making claims -- specifically that !CONFIG_4KSTACKS is "safer", happily ignoring the fact that generally speaking available process stack can be _better_ with CONFIG_4KSTACKS and there seems to exist but _one_ (één, ein, une) known situation where it's problematic.

Must there be none rather than one? In some senses maybe, if the problem is more than bad, fixable code but I doubt you know this. CONFIG_4KSTACKS is much better on the VM (and hence faster) and as such, any user not using the one nicely isolated and identified problem case benefits from it. This means it's either very close or already _at_ the point of being the best default for the kernel. Changing options is for users with special needs, as you believe you are.

I truly apologise for taking it into this direction but you're wearing me down rapidly. Every single time you insert some uninformed crap comment that shows that you both don't understand the issue and didn't understand what the other person was saying and then after being made aware of such, ignore that and follow up with the next uninformed crap comment. That is, you seem to care less about the issue then about the discussion and since for me it's quite the other way around I'm leaving it at that.

RedHat is the one with the actual data available, and they've been enabling 4KSTACKS for quite some time now (with some of their users apparently unhappy about it but not many it would seem).

Jesper also already posted how he's going to proceed: lift 4K from debug status and submit it as default for -mm. As to the latter bit, unless I remember wrong, it already _was_ default in -mm for some time a while ago so Andrew no doubt has an informed opinion on how to proceed with that.

Rene.

#76 lore raw
Quoted 11 lines · Rene Herman
Quoted 10 lines · Bodo Eggert
Quoted 8 lines · Rene Herman
Quoted 1 line · Bodo Eggert

You claim 4k+4k is safe, therefore 8k must be safe, too.

No, I most certainly do not. I claim proving that 4K and seperate (per cpu) interrupt stacks are safe are exactly the same as proving unshared 8K stacks are safe. That is, you don't, no such proof exists other than in the eating of the pudding.

Quoted 7 lines
Quoted 2 lines

And yet you have a more strict claim than I do. If you are right, I'll be right, too, because two times less-than-4K is less tham 8K.

Firstly, it's not two times 4K but 4K + (4K + 4K) * NR_CPUS. Secondly, _you_ are the one making claims -- specifically that !CONFIG_4KSTACKS is "safer", happily ignoring the fact that generally speaking available process stack can be _better_ with CONFIG_4KSTACKS

It can be better, but the worst case stays 4K + 4K - unless one CPU will walk over to the next and nicely ask for a cup of stack.

Therefore you can discuss 4K + 4K or 4K + 4K + 4K, or 4K + 4K * \inf. It won't change a thing:

1) It all can be reduced to 4K + 4K by asuming all IRQ happen on one CPU.
2) Even if the interrupts decide not to happen on one CPU, you still can't fit that possible 5K into 4K.

Having a local stack per CPU helps locality, and it's gootd, but that's about it.

Quoted 2 lines

and there seems to exist but _one_
(één, ein, une) known situation where it's problematic.

One case is reason enough not to enable 4K-stacks per default, and this is a common server setup. "server" as in "I need a reliable system".

Quoted 3 lines

Must there be none rather than one? In some senses maybe, if the problem is more than bad, fixable code but I doubt you know this. CONFIG_4KSTACKS is much better on the VM (and hence faster) and as such,

"Look how fast I crashed!" doesn't buy you anything. In order to finish first, you first got to finish.

Quoted 2 lines

any user not using the
one nicely isolated and identified problem case benefits from it.

And they can turn it on.

Quoted 4 lines

This means
it's either very close or already _at_ the point of being the best default for the kernel. Changing options is for users with special needs, as you believe you are.

If you designed a car, you would also go for breaks with a well-known problem just because they weight less and all that people not crossing mountains would be happy about the weight benefit - that is if they'd notice, wouldn't you?

Please post a list of things you have designed, so I can avoid them.

Quoted 2 lines

I truly apologise for taking it into this direction but you're wearing me down rapidly.

I put the facts onto the ground. If you're getting down, you may stumble on them. Beware

Quoted 4 lines

Every single time you insert some uninformed crap comment that shows that you both don't understand the issue and didn't understand what the other person was saying and then after being made aware of such, ignore that and follow up with the next uninformed crap comment.

So what did you say about the worst case stack size being bigger than 4K? That's correct, you choose to put it aside as a minor use case. Yea, it's just the combination you'd choose for a reliable server setup, the users won't have a problem when their systems crash ...

Was your claim about each CPU having a separate stack helping your cause? No, everybody can see it's not. That is, except for you, your CPU will just borrow some, since their neighbours have some free stack.

But let's not stop here: You claimed: "Unshared interrupt stacks make for more determistisc behaviour, so you'd have a harder time proven anything to some set limit of uncertainty with the shared 8K stacks than with the unshared 4K stacks."

So you want to tell me I can't prove 8K stacks are safe - you are right. But can you prove 4K stacks are safe? You can't either. But you want to be able to prove it. I told you to stick to your words - go and prove 4K+4K to be safe. What did you do? You chose to ignore that.

I bet you don't even consider proving 4K stacks to be correct, nor do you know anyone who would try that in the near future. And besides that, you know at least one case where your proof would fail. So why would you talk about proofs? Do you think you can trick me into believing a crashing system would work more correctly than a non-crashing one, just by mentioning the possibility of having it easier to make a proof?

Besides that, I told you you can separate unshared interrupt stacks from 4K stacks. And yet, you still argue as if 4K stacks are required for having a separate interrupt stack.

So who's ignoring facts, who is talking crap?

Quoted 3 lines

That is, you seem
to care less about the issue then about the discussion and since for me it's quite the other way around I'm leaving it at that.

I know very well you're going to turn linux into a bleeding edge system where you're supposed to get some bad surprises on an update, unless you happen not to hit the error case. That's not the way I'd like it to go. You may use -mm, you'll get the 4K stacks per default, be happy wih them.

Quoted 3 lines

RedHat is the one with the actual data available, and they've been enabling 4KSTACKS for quite some time now (with some of their users apparently unhappy about it but not many it would seem).

Would you use a distribution crashing on your servers? Or would you change the distribution and spread the word not to use redhat?

Signature · 3 lines
"'Multiple exclamation marks,' he went on, shaking his head, 'are a 
sure sign of a diseased mind.'"
	-- Terry Pratchett in "Eric"
#77 lore raw
Quoted 2 lines

1) It all can be reduced to 4K + 4K by asuming all IRQ happen on one CPU.

no it's separate stacks for soft and hard irqs, so it's really 4+4+4

another angle is that while correctness rules, userspace correctness rules as well. If you can't fork enough threads for what you need the machine for, why have the machine in the first place?

Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
#78 lore raw
Quoted 3 lines · Arjan van de Ven
Quoted 1 line

1) It all can be reduced to 4K + 4K by asuming all IRQ happen on one CPU.

no it's separate stacks for soft and hard irqs, so it's really 4+4+4

Thanks, I missed that information. Unfortunately this change still does not help if one of these stacks needs to grow beyond 4K.

Quoted 3 lines

another angle is that while correctness rules, userspace correctness rules as well. If you can't fork enough threads for what you need the machine for, why have the machine in the first place?

Userspace can't work correctly after the kernel crashed, but it can fail gracefully if it can't create enough threads.

I'd really like to be able to select 4K stacks, but as long as that stack would overflow, I can't, and it can't be default, too.

Signature · 2 lines
Top 100 things you don't want the sysadmin to say:
8. ...and after I patched the microcode...
#79 lore raw
In reply to Bodo Eggert
Quoted 1 line · Bodo Eggert

Please post a list of things you have designed, so I can avoid them.

- The ability to read
- The ability to understand

You're doing a hell of a job already.

Rene.

#80 lore raw
Quoted 8 lines · Rene Herman
Quoted 1 line · Bodo Eggert

Please post a list of things you have designed, so I can avoid them.

- The ability to read
- The ability to understand

You're doing a hell of a job already.

If you designed them like you design secure systems, that explains a lot.

Signature · 2 lines
Top 100 things you don't want the sysadmin to say:
83. Damn, and I just bought that pop...
#81 lore raw
In reply to Rene Herman
Quoted 6 lines · Rene Herman

Given that as Arjan stated Fedora and even RHEL have been using 4K stacks for some time now, and certainly the latter being a distribution which I would expect to both host a relatively large number of lvm/md/xfs and what stackeaters have you users and to be fairly conservative with respect to the chances of scribbling over kernel memory (I'm a trusting person...) it seems
there might at this stage only be very few offenders left.

I have to recompile the fedora kernel rpms (fc6, f7) with 8k stacks on my i686 server. It's using NFS -> XFS -> DM -> MD (raid1) -> IDE disks. With 4k stacks it crash (hang) within minutes after using NFS. With 8k stacks it's rock solid. No crashes within months.

utz

#82 lore raw
Quoted 9 lines · utz lehmann
Quoted 7 lines · Rene Herman

Given that as Arjan stated Fedora and even RHEL have been using 4K stacks for some time now, and certainly the latter being a distribution which I would expect to both host a relatively large number of lvm/md/xfs and what stackeaters have you users and to be fairly conservative with respect to the chances of scribbling over kernel memory (I'm a trusting person...) it seems there might at this stage only be very few offenders left.

Quoted 4 lines

I have to recompile the fedora kernel rpms (fc6, f7) with 8k stacks on my i686 server. It's using NFS -> XFS -> DM -> MD (raid1) -> IDE disks. With 4k stacks it crash (hang) within minutes after using NFS. With 8k stacks it's rock solid. No crashes within months.

Running either vanilla or Gentoo kernels, I've had no luck at all with 4k stacks. On SATA+MD(raid1)+LVM+XFS I've had numerous crashes when doing even light I/O. One particularly crash-prone scenario is writing to random locations in sparse files, thus forcing new blocks to be allocated.

I don't have dedicated testing machines, so I can't afford the time and potential data loss of testing this regularly. I have no shortage on RAM with 8k stacks, so for me the choice is quite simple.

Signature · 2 lines
Måns Rullgård
[email protected]
#83 lore raw
In reply to utz lehmann
Quoted 4 lines · utz lehmann

I have to recompile the fedora kernel rpms (fc6, f7) with 8k stacks on my i686 server. It's using NFS -> XFS -> DM -> MD (raid1) -> IDE disks. With 4k stacks it crash (hang) within minutes after using NFS. With 8k stacks it's rock solid. No crashes within months.

Okay, thanks. That's the usual offender. And only one I've heard of...

Rene.

#84 lore raw
In reply to utz lehmann
Quoted 6 lines · Rene Herman
Quoted 6 lines

Given that as Arjan stated Fedora and even RHEL have been using 4K stacks for some time now, and certainly the latter being a distribution which I would expect to both host a relatively large number of lvm/md/xfs and what stackeaters have you users and to be fairly conservative with respect to the chances of scribbling over kernel memory (I'm a trusting person...) it seems there might at this stage only be very few offenders left.

utz> I have to recompile the fedora kernel rpms (fc6, f7) with 8k utz> stacks on my i686 server. It's using NFS -> XFS -> DM -> MD utz> (raid1) -> IDE disks. With 4k stacks it crash (hang) within utz> minutes after using NFS. With 8k stacks it's rock solid. No utz> crashes within months.

Does it give any useful information when it does crash? Can you make a simple test case using ram disks instead of IDE disks and then building upon that?

I think I should try to do this myself at some point...

John

#85 lore raw
Quoted 8 lines · John Stoffel

utz> I have to recompile the fedora kernel rpms (fc6, f7) with 8k utz> stacks on my i686 server. It's using NFS -> XFS -> DM -> MD utz> (raid1) -> IDE disks. With 4k stacks it crash (hang) within utz> minutes after using NFS. With 8k stacks it's rock solid. No utz> crashes within months.

Does it give any useful information when it does crash?

No, sorry. Nearly always it lock up so hard that even sysrq didn't work anymore. Most times the console was blanked. If not, there was a line with "do_irq" or something like that (if i remember correctly). A few times it continuous oopsing (scrolling like mad).

I think it's just a stack overflow. Knowing that XFS + long IO stack have problems with 4k stacks. And i have zero crashes with the recompiled 8k stack kernels. (All kernel are the fedora ones).

Btw: In the past the server runs on slightly different hardware and without raid1 (NFS -> XFS -> DM -> IDE disk). It runs with 4k stacks. I had a few crashes, but i blame the hardware for it.

I don't want to make tests with the server. It's my main data storage and i don't want to risk it.

Quoted 3 lines

Can you make
a simple test case using ram disks instead of IDE disks and then building upon that?

Sorry, i don't think i can do this. My other computer, which i can use for tests, is x86_64 based.
And IFAIK the problem on the XFS side has something to do with looking for freespace on many AGs. So maybe a bigger and filled filesystem is needed. And 50GB ram disks are out of question.

utz