Skip to contentSkip to conversation

NoHZ and CPU isolation patches

DISCUSSION Max Krasnyansky 2 participants 2 messages Feb 14, 2012 – Feb 20, 2012 2 messages not in the archive

lore
Share & tools
lore
#2 lore raw
In reply to a message not in the archive
Quoted 5 lines · Gilad Ben-Yossef

Hi Max,

Any specific reason not to CC the LKML list? I know it is sometime noisy and I understand
you are not subscribed, but it is the Right Thing (tm) to do...

No reason really. I was just going to keep it short and basically just ask for CCs on future discussions :).
Looks like this might turn into a useful discussion. Added to the CC.

Quoted 7 lines · Max Krasnyansky
Quoted 5 lines

Gilad, Frederic,

At the time a lot of people
were totally opposed to that
from the recent threads it looks like things have changed quite a bit.

Quoted 7 lines

I think people like the idea of CPU isolation, there is just the question of what CPU
isolation means :-)

At least my personal approach is that if a task is running on an isolated CPU has caused
the kernel to need to do work on that CPU, that is fair game.

Yep. My definition is the same. ie If a task wants to avoid interference from the kernel it better not use any syscalls() or at least not the syscalls that trigger IO, mem allocations, etc.

Quoted 7 lines

As a special exception, work required by the kernel to do on the CPU by a previous task
running on the isolated CPU (cleanups) is also a fair game. (We may consider later adding
a "clean me" API so that tasks can ask the kernel to block them
until all previous cleanups
have been completed).

That's what I used CPU hotplug for. I still think it's practically perfect for doing this cleanup stuff. CPU hotplug code provides logic for migrating stuff off of the CPU that is going offline. So as part of the "isolation" prep work I setup cpusets, irq affinity masks, etc and bring the CPU offline. Then once we bring it online it nice and clean with no pending work, timers, etc.

Quoted 7 lines

What we're trying to do, IMHO, is only to keep the kernel from interfering with the user
application due to no action of the application - either because of activity of a task
on another CPU which is not directly related to our task or due to kernel normal routine.

Quoted 8 lines

As a special exception, interference caused by specific, well defined,
rare user actions
are OK as well, so long as the user has a very clear expectation that
this action will cause interference. Thus, stop_machine during module unload is fine (user controlled and rare), but stop_machine for RCU expedited sync is not (not user controlled, not rare enough).

I think that makes life easier for us :-)

As far as I remember there are other stop_machine uses that were not as obvious to the user as module unload. ftrace for example calls stop_machine to patch text segments. For now I've simply disabled dynamic ftrace in my kernels.

Quoted 7 lines
Quoted 2 lines

In the mean time I'm going to start playing with your trees. I'm curious for example if you found solutions for stop_machine and things.

Frederic's patch set deals with the scheduler timer tick and related functionality.
I took to IPI causes first.

Quoted 6 lines

stop_machine is not dealt with and as I explained, it is not always a problem.

I keep a tracking list of interference sources and remedies here:
https://github.com/gby/linux/wiki

I'd love to get feedback for it.

Sounds good. I'll take a look and get back to you.

Max

#3 lore raw
Quoted 22 lines · Max Krasnyansky
Quoted 5 lines · Gilad Ben-Yossef

Hi Max,

Any specific reason not to CC the LKML list? I know it is sometime noisy and I understand
you are not subscribed, but it is the Right Thing (tm) to do...

No reason really. I was just going to keep it short and basically just ask for CCs on future discussions :).
Looks like this might turn into a useful discussion. Added to the CC.

Quoted 11 lines · Max Krasnyansky
Quoted 5 lines

Gilad, Frederic,

At the time a lot of people
were totally opposed to that
from the recent threads it looks like things have changed quite a bit.

I think people like the idea of CPU isolation, there is just the question of what CPU
isolation means :-)

Quoted 6 lines
Quoted 3 lines

At least my personal approach is that if a task is running on an isolated CPU has caused
the kernel to need to do work on that CPU, that is fair game.

Yep. My definition is the same. ie If a task wants to avoid interference from the kernel it better not use any syscalls() or at least not the syscalls that trigger IO, mem allocations, etc.

Right but we want a solution that works for everyone: those who need to issue syscalls from time to time (IO or whatever), those who don't issue syscalls at all, etc...

So unplugging the CPU might work for pure isolation (no syscalls, no irq, no exception) but there may be lighter kind of isolation, with rare syscalls, exceptions or irqs, etc...

If the task is disturbed for useless work, then lets fix that. This may be useful for isolation and even much more globally.

Note that unplugging the CPU also prevents the task from any return to the kernel. This include the do_exit() path. And I believe we don't want to deal with a CPU that is not in the online mask when we enter such a complicated path.