sched: Implement BPF extensible scheduler class

Bah, I got it backwards indeed. But in this case, don't you also need something in pick_task() -- the whole core scheduling thing does much the same.

Hello,

Yes, indeed we do, but because we're dispatching from the balance path, the cpu_acquire method is being called from there. Because who was running on the CPU before us is less interesting, @prev is not passed into cpu_acquire() but if that becomes necessary, it's already available there too.

Thanks.

tejun

I suppose I need to read more, because I'm not knowing what cpu_acquire is :/ I do know I don't much like the asymmetry here, but maybe it makes sense, dunno.

Hello,

The sched_ext ops are symmetric - ops.cpu_release() is called when SCX loses CPU to a higher priority sched class and ops.cpu_acquire() when the CPU returns to SCX afterwards. Where they hook into is not symmetric. The class which picks the next task already knows the previous task, so there's no need to add anything. However, without the new sched_class->switch_class(), the previous class has no way of knowing, so they're a bit different.

Thanks.

tejun

Linus!

Yes we do.

There are too many places which add scx_***() invocations. That's what I asked to be cleaned up and to be generalized so it becomes uniform over the scheduler classes.

Sure one could argue that some of these calls are at places where some other scheduling class dropped already one, but that's wrong because accumulating bad code just creates more technical debt.

If cleaned up then the already existing hacks vanish into proper class callbacks, which improves the existing code and allows to drop in sched ext more naturally at the end.

One example I very explicitely mentioned back then is the dance around fork(). It took me at least an hour last year to grok the convoluted logic and it did not get any faster when I stared at it today again.

fork()
sched_fork()

    scx_pre_fork()
      percpu_down_rwsem(&scx_fork_rwsem);

    if (dl_prio(p)) {
    	ret = -EINVAL;
        goto cancel; // required to release the semaphore
    }

  sched_cgroup_fork()
    return scx_fork();

  sched_post_fork()
    scx_post_fork()
      percpu_up_rwsem(&scx_fork_rwsem);

Plus the extra scx_cancel_fork() which releases the scx_fork_rwsem in case that any call after sched_fork() fails.

My head still spins from deciphering this once more.

What has scx_fork() to do with sched_cgroup_fork()? It's completely non-obvious and the lack of comments does not help either. The changelog is handwaving at best.

scx_pre_fork() takes the semaphore unconditionally independent of the scheduler class of the forking task and even in the case that no BPF scheduler is loaded or active. Why? Neither the changelog nor the lack of comments give any hint, which is also not a new complaint from me.

A proper cleanup would just eliminate the unconditional down(), the dl_prio() cancel logic plus the whole if/elseif dance including the #ifdef SCHED_EXT. It's not rocket science to come up with the obvious:

       ret = p->sched_class->pre_fork();
       if (ret)
       		return ret;

That's just proper engineering which is what some famous programmer tells people to do for a very long time:

"I’m a huge proponent of designing your code around the data, rather
than the other way around, ... I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

And that's not only proper engineering it's also the other approach the same famous programmer tells people to do:

"I want them to lull me into a safe and cozy world where the stuff they
are pushing is actually useful to mainline people _first_."

IOW, give me something which is useful to me _first_ so that you can add your particular flavor of crazy on top without bothering me.

You obviously can complain now about the crazy people who actually listen to what that famous programmer is saying. :)

But the above is not only true for that famous programmer personally, that's equally true for any maintainer who has to deal with the result of a submission for a long time.

I'm still not seeing the general mainline people benefit of all this, so I have to trust you that there is one which is beyond my comprehension skills.

That said, I'm more than wary about the hidden locking scheme of that percpu semaphore in the fork path, but that's a different design question to be debated on the way.

There are some other technical details which need to be sorted including the interaction with other pending code, but that's something which can be solved as we move forward.

Ideally this can be shaped in a way so that the scheduler becomes closer to being modular, which would be the real useful thing for research and not just the advertisment version of it.

But wait a moment, that can't happen as pluggable schedulers have been rejected in the past:

"I absolutely *detest* pluggable schedulers."

Guess which famous programmer said that.

Well spotted.

Which is exactly the point. If the existing code does not let your new feature fall into place, then refactor it so it does. Working around the short comings at some other place is patently wrong and that's not something new either.

Requesting such refactoring is not an undue burden because the people who maintain the code will have to deal with the result.

Unwrapping this stuff after the fact is the worst thing to do and I definitely have an expert opinion on this.

None of this is rocket science and could have been done long ago even without me holding hands.

The difference is:

1) Peter is back and the capacity problem is less bad than it was

2) As I explained before I unfortunately did not have cycles in the past _seven_ months, but I can make the cycles available now and drive this forward. That's possible when both sides are willing to cooperate. I'm sure there is enough incentive to do so.

3) I'm going to focus on the integration and interaction aspect and grudgingly leave the cgroup trainwreck out of it.

If someone from the crowd who caused it actually had the courtesy to mop this up, that would be a great signal for everyone and it can be done in parallel.

Not that I'm holding my breath, but I'm still a hopeless optimist.

If you don't trust me on that, then we have a very different problem.

Thanks,

Thomas

I'e also said that I find schedulers in general boring. I think CPU scheduling is just not very interesting.

Look at the 0.01 scheduler, and the comment above it:

* 'schedule()' is the scheduler function. This is GOOD CODE! There
* probably won't be any reason to change this, as it should work well
* in all circumstances (ie gives IO-bound processes good response etc).
* The one thing you might take a look at is the signal-handler code here.

Which shows just how much I knew (or cared).

Things change. Back then, you could have a maximum of 63 processes (plus the idle task).

And the "I detest pluggabnle schedulers" has been long superseded by "I detest people who complain about our one scheduler because they have special loads that only they care about".

But Thomas - that's *MY* point.

If this code stays out of tree, the goal is always that "don't integrate, make the patch easy to apply".

This whole "keep things out until they are perfect" is a DISEASE.

It's a disease because it's counter-productive. First off, things will never be "perfect" because you have people with different goals in the first place.,

But secondly, the "keep things out" is itself counter-productive.

Linus

Linus!

You are seriously trying to lecture me of all people about out of tree code and how it's supposed to be merged?

I really don't know whether I should laugh, cry or just walk away.

That's absolutely not the goal. The goal is to do a proper and good integration so that all involved parties can move on.

That's what I clearly said and nothing else.

Nobody says that things need to be perfect. I'm very much a proponent of "perfect is the enemy of good".

Says the one who kept asking me repeatedly whether I can't keep the remaining stuff of RT out of tree forever. The last time you asked that was not that long ago.

Aside of that you are completely ignoring my point.

Collaborative integration is the right thing to do no matter what. I've integrated and helped many others to integrate tons of controversial features in the past 20 years so I definitely know what I'm talking about. But I've also learned my lessons in the past 20 years.

The whole notion of "let's get this in and sort it out later" only works when people collaborated to begin with and worked on making things more palatable in the first place. At some point they just agreed to get over it. That has nothing to do with perfect and keep stuff out of tree forever.

In all other cases the stuff got merged and the "sort it out later" was left to those who had the least interest in the so exiting new feature. They had to mop up the mess to make progress with their own work. I wasted a substantial portion of my life mopping up the mess which was declared to be sorted out later and you know that.

If that is your vision of a working and healthy community, then we have to agree that we disagree.

The issue at hand is clearly both a technical and a social problem. Contrary to you I don't hate people and my goal is to bridge the rift which clearly exists. That rift is definitely not helpful to get things sorted out later in a constructive and collaborative way.

I've offered _my_ time to help and sort this out. That's all I can do.

If you don't care, so be it.

Thanks,

tglx

Thomas, you are starting to just make made-up arguments now.

Did I require that the RT patches be done right over two decades? Hell yes. But those RT patches didn't change some single subsystem, they made fundamental changes to the most core things out there.

The RT patches made something as fundamental and core as "disable interrupts for spinlocks" be a special magical thing that normal people weren't allowed to do, and that turned into a sleeping lock. They affected _everything_. They very literally affected subsystems and rules that had been there since linux-0.01 (not the spinlocks - they came later - but things like tty / printk were in fact some of the very first things written).

And don't get me wrong - I'm not complaining about the RT patches. I think they improved things enormously in the end. They've been great.

I'm just saying that they are _not_ the norm to compare against.

The sched_ext patches? Not even remotely in the same class. The sched_ext patches are more like fuse - another filesystem, just a slightly odd one that has that big user space component.

And yes, fuse caused worries too because of the whole "filesystem development in user space" outside the confines of the normal kernel development model. It mostly just was (and is) a shim-layer that asks user space for policy, while still using the kernel infrastructure for most real work. Did those worries actually materialize? No. No they did not.

Now, fuse to some degree was easier, because while it was merged about two decades ago, even by that time we already had over a decade of pluggable filesystems. So the VFS layer had a lot of pluggability already, and it wasn't very hardcoded.

The scheduler also has the scheduler class pluggability, but it *is* fairly hardcoded. So in addition to the kinds of issues FUSE had (with all the "impedance matching" between the kernel interface and the user level interfaces), the sched_ext patches have some of that hardcoded pluggability that it needs to fix up.

So maybe a better comparison would be autofs - but autofs was added _so_ long ago that I only vaguely remember the odd pain points for waiting for mount points. Over the years, all of that special sauce that no other filesystem needs has become such an integral part of the vfs layer that people don't even think about it any more.

You can most definitely still see the effects of autofs - as opposed to "regular" filesystems - on the vfs layer if you go look (things like "finish_automount()" and friends), but it's been integrated for so long that it's a non-issue. But back in the day, it all needed what at the time was some special glue.

Anyway, what I'm saying is that you trying to equate this with the RT patches is absolutely laughable and intellectually dishonest.

Look, ignoring the actual sched_ext code itself (and the examples, documentation and test-cases which I certainly hope nobody would object to), the actual core footprint ot fht sched_ext patches is this:

 MAINTAINERS                       |  13 +++
 Makefile                          |   8 +-
 drivers/tty/sysrq.c               |   1 +
 include/asm-generic/vmlinux.lds.h |   1 +
 include/linux/cgroup.h            |   4 +-
 include/linux/sched.h             |   5 +
 include/linux/sched/task.h        |   3 +-
 include/uapi/linux/sched.h        |   1 +
 init/init_task.c                  |  12 +++
 kernel/Kconfig.preempt            |  24 +++++
 kernel/fork.c                     |  17 +++-
 kernel/sched/build_policy.c       |  10 ++
 kernel/sched/core.c               | 187 ++++++++++++++++++++++++++++++--------
 kernel/sched/debug.c              |   3 +
 kernel/sched/ext.h                | 114 +++++++++++++++++++++++
 kernel/sched/fair.c               |  21 ++---
 kernel/sched/idle.c               |   2 +
 kernel/sched/sched.h              |  75 ++++++++++++++-
 kernel/sched/syscalls.c           |  26 ++++++
 lib/dump_stack.c                  |   1 +
 20 files changed, 464 insertions(+), 64 deletions(-)

and honestly, I went through it all. None of it looks really objectionable. Some of it is trivial cleanups and makes the code look better (the "refactor" parts).

And yes, some of it is "We have a new scheduler class that wants more".

Realistically, of the above, I think the *only* parts that are even at all halfway interesting are these:

 kernel/sched/core.c               | 187 ++++++++++++++++++++++++++++++--------
 kernel/sched/ext.h                | 114 +++++++++++++++++++++++

and that ext.h is on that "interesting" list only because of for_each_active_class(), and I think it should probably just be in core.c.

See why I do not think that this is AT ALL comparable to something like the RT patches.

And yes, I do think code should be cleaned up, but the two cleanups that struck me personally were literally just

(a) scx_next_task_picked, where I slept on it, came up with a
three-line alternative suggestion, and Tejun sent the "real patch" almost immediately

(b) the for_each_active_class() thing that I think would actually be
better off just being done explicitly in sched/core.c, but probably only makes sense after integration

and it really strikes me as "neither of these issues were worth nine months of delay".

Although hopefully the nine months weren't entirely unproductive, and maybe the patches have improved. Knock wood.

Because *most* of the two files above are actually normal stuff. Things that other scheduler classes already do. Sometimes just with an #ifdef around it (although many of the ifdef's are basically hidden as "inline function that is empty if disabled" - which is how we tend to do things in the kernel in general.

So it's not even "200+ lines of objectionable code". No. It's all fairly small.

Arguably much of the strangeness comes from "it can be enabled / disabled both statically and dynamically".

The "dynamically disable" code may look particularly odd because some of that is the "scx_switched_all()" thing that basically disables some of the CFS special cases. Is that pretty? No, but in most cases I'd say that the cause of said issue was that the CFS rules were hardcoded.

So it's pretty small, it's pretty self-contained, and it only affects one single subsystem.

See where I'm coming from?

Yes. And am willing to say "ok, if three more months make this more amicable, I'll delay merging for another release".

Amicable resolutions are obviously preferred. And I am certainly willing to going back and telling people "ok, it's going to be 6.12, not 6.11". Some people are going to be disappointed in me. That's my job, and I don't think this needs to be *rushed*.

But really, in the very next sentence of "I don't think this needs to be *rushed*", I do want to re-iterate that I feel like this pointless delay this has seen seems unnecessary.

I hope I have explained above why I think this whole thing has not been worth the brouhaha and pain that it has caused. It's really not that big, and the issues I see have either had small clean trivial solutions, or have very much looked to me like "ok, it's not integrated, but I see why an external patch would try to do it that way".

And yes, I may be missing some big ugly point. But I really _have_ been going through those roughly 200 lines of diff that actually seem to be relevant to this whole argument. Multiple times.

So if I don't see it, please point it out very very explicitly, and using small words to make me understand.

Linus

Hello,

Just posted a patchset to integrate sched_ext a bit better. I open coded for_balance_class_range() and moved for_each_active_class() and friends to kernel/sched/sched.h. If something else would look better, please let me know.

http://lkml.kernel.org/r/[email protected]

Thanks.

tejun

Linus!

I really let the scheduler people bring up their pain points now. There is no value in me being the mouthpiece when I want to achieve that folks talk to each other (again). It's not a me against you thing. If they fail to explain, so be it.

So I just use one trivial example to illustrate the approach and the message it emits, i.e. the technical and the social problem which makes all of this so unpleasant.

It struck my eyes a few days ago when I deciphered the fork maze:

struct sched_ext_entity {
       .....
	/* must be the last field, see init_scx_entity() */
	struct list_head	tasks_node;
};

void init_scx_entity(struct sched_ext_entity *scx)

{
	/*
	 * init_idle() calls this function again after fork sequence is
	 * complete. Don't touch ->tasks_node as it's already linked.
	 */
	memset(scx, 0, offsetof(struct sched_ext_entity, tasks_node));

...

I immediately asked myself the obvious question: Why?

It took me less than 10 minutes to figure out that the double invocation of __sched_fork() is bogus.

fork_idle()
copy_process()

      sched_fork()
	__sched_fork()
   init_idle()
     __sched_fork()

There is only one other call site of init_idle():

     sched_init()
       init_idle()

to initialize the idle task of the boot CPU.

Another 10 minutes later I had the obvious patch for this booted and validated. So overall that took me just 20 minutes and that's not because I'm the deep scheduler expert, it's because I care.

It's really not the job of the maintainer to point at that or figure it out especially not with the knowledge that our maintainer resources are anything else than abundant. Even if I can't figure it out on my own, then pointing it out and asking would have been the right thing to do.

Instead of working around it and conveying the message "shrug, it's not in the scope of my project, why should I care?", it would have:

1) removed technical debt

2) not added more technical debt

3) followed what documentation asks people to do

4) told the scheduler people "hey, we care about working with you"

It's a trivial detail, but it illustrates the tiny bits which contributed to the overall rift.

Here I really pick RT, not for comparison, but for reference. RT was certainly exposed to resistance, ignorance and outright denial. We sorted it out by working with the people, by going the extra mile of mopping up technical debt. Setting this as expectation is not asked too much.

As I said before: This is both a technical and a social problem.

As the social problem became prevalent over time the technical problem solving got nowhere.

As a matter of fact both sides contributed to that, and that cannot be resolved par ordre du mufti.

The result of that would be a even deeper rift which turns 'work it out in tree' to 'work around each other in tree' or worse.

There is interest from both sides to get this sorted. At least that's what was conveyed to me. I'll make sure to hear it from the source.

Thanks,

tglx

---

@@ -4344,7 +4344,8 @@ int wake_up_state(struct task_struct *p,
  * Perform scheduler related setup for a newly forked process p.
  * p is forked by current.
  *
- * __sched_fork() is basic setup used by init_idle() too:
+ * __sched_fork() is basic setup which is also used by sched_init() to
+ * initialize the boot CPU's idle task.
  */
 static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 {

@@ -7579,8 +7580,6 @@ void __init init_idle(struct task_struct
 	struct rq *rq = cpu_rq(cpu);
 	unsigned long flags;
 
-	__sched_fork(0, idle);
-
 	raw_spin_lock_irqsave(&idle->pi_lock, flags);
 	raw_spin_rq_lock(rq);
 

@@ -7594,12 +7593,7 @@ void __init init_idle(struct task_struct
 	kthread_set_per_cpu(idle, cpu);
 
 #ifdef CONFIG_SMP
-	/*
-	 * It's possible that init_idle() gets called multiple times on a task,
-	 * in that case do_set_cpus_allowed() will not do the right thing.
-	 *
-	 * And since this is boot we can forgo the serialization.
-	 */
+	/* No validation and serialization required at boot time. */
 	set_cpus_allowed_common(idle, &ac);
 #endif
 	/*

@@ -8407,6 +8401,7 @@ void __init sched_init(void)
 	 * but because we are the idle thread, we just pick up running again
 	 * when this runqueue becomes "idle".
 	 */
+	__sched_fork(0, current);
 	init_idle(current, smp_processor_id());
 
 	calc_load_update = jiffies + LOAD_FREQ;

Linus!

Thanks!

I'm not comparing them.

I was just pointing out that you repeatedly asked me whether the nasty parts could stay out of tree forever, which I always found odd. But now in the context of your out of tree lecture this question struck me even more strange. Understandably, no?

See how communication between people fails?

I might have misinterpreted your question to keep RT out of tree and you interpreted my answer as a comparison, which was not my intention at all.

If I want to compare another out of tree project with sched ext, then I surely do not pick RT but DPDK. The network people rejected the DPDK approach as they wanted to have things fixed and done in tree instead of letting everyone create their own sand pit. It worked out as it made people think and come up with XDP and other things which gives the dataplane people a proper tool while having the general stuff work nicely in the same context.

In other words, that forced people to really collaborate and sort it out for the benefit of everyone. I might be missing something crucial, but I fail to see the same benefit coming from sched ext.

Coming back to what you said in an earlier mail:

I agree with that sentiment. I don't agree with the "solution".

The sad truth is that everyone involved admired the problem for a decade and kept complaining in the one way or the other.

Google dropping out of scheduler development was not because of scheduler people being hard to work with. Peter and Paul worked perfectly fine together and the hierarchical cgroup scheduling muck was merged under the premise "We work it out in tree". It just never happened because the people who added it vanished in a black hole for reasons which have nothing to do with the kernel scheduler community.

At last years OSPM everyone in the room, including the sched ext folks, agreed that the main problem is that the scheduler does not have enough information about the requirements and properties of applications, which is not a Facebook/Google specific thing. That applies to all sorts of problems including power, thermal and capacity constraints.

That's nothing new. The academic scheduler research identified that in the late 90s already and came up with specific solutions to prove their point. That effort fell short to be generalized.

So sched_ext does exactly this by putting requirements and properties of workloads into the BPF scheduler and the related user space portion.

I completely agree that this is a nice tool for doing research to identify what needs to be done to make this a generalized approach.

I disagree that providing it as an official workaround will result in more collaboration and a better result for everyone in the very end. Quite the contrary it is going to foster fragmentation way beyond the Google/Facebook space.

The whole notion of 'my workload is so special and therefore we need special sauce' is a strawman. We've debunked a lot of 'my thing is so special' claims over the years by making people sit down and come up with generalized solutions for the benefit of everyone.

I'm not saying we debunked all. Some of them failed because people refused to work it out and opted for keeping their stuff out of tree forever. But in the vast majority of cases it worked out pretty well.

I recently watched a talk about sched ext which explained how to model an execution pipeline for a specific workload to optimize the scheduling of the involved threads and how innovative that is. I really had a good laugh because that's called explicit plan scheduling and has been described and implemented in the early 2000s by academics already.

Innovative or not, that's not the point. The point is that none of this resulted in the promised feed back to the scheduler proper. As this runs in production already, it would have been a great talk at OSPM24 to follow up on the 'requirements and properties' discussion to at least provide the insights of this in the form of data to work from.

I can see your benefit that the detesting complaining will stop, but I fail to map that into a general benefit for everyone else. Some enlightment would be appreciated.

Thanks,

tglx

Not to derail this discussion, but this is not quite the case.

It never "worked out", DPDK has continued to grow in importance since XDP was invented because XDP/etc are not a replacement for DPDK. It is still the case that places using DPDK really don't have any performant alternative.

DPDK growth has been significant, in fact, it is quite likely this message traversed some DPDK in the internals of the internet on its way to you. It is an important and necessary project for certain applications.

DPDK is very good at what it does, best in class in fact, and fully supported by in-tree Linux. Just not via netdev.

The discussion on these topics created animosity from netdev toward DPDK and that has created some community/collaboration problems in its wake.

IMHO forced collaboration doesn't work unless both sides can gain some benifit - in this case it was not clear what the advantage really was for DPDK. For instance DPDK-XDP exists, but it is not widely used because it is slower.

I don't think there was a benefit for everyone here. People remain split based on their use case and there is little actual collaboration. DPDK people know not to talk to netdev :)

Part of collaboration is to be able to also know when collaboration is not going to be valuable or feasible.

Not sure there is a learning here for sched_ext..

Regards,
Jason

Hello,

This part is actually tricky. sched_cgroup_fork() part is mostly just me trying to find the right place among existing hooks. We can either just rename sched_cgroup_fork() to a more generic name or separate out the SCX hook in the fork path.

When a BPF scheduler attaches, it needs to establish its base operating condition - ie. allocate per-task data structures, change sched class, and so on. There is trade-off between how fine-grained the synchronization can be and how easy it is for the BPF schedulers and we really do wanna make it easy for the BPF schedulers.

So, the current approach is just locking things down while attaching which makes things a lot easier for the BPF schedulers. The locking is through a percpu_rwsem, so it's super heavy on the writer side but really light on the reader (fork) side. Maybe the overhead can be further reduced by guarding it with static_key but the difference won't be much and I doubt it'd make any noticeable difference in the fork path.

Thanks.

tejun

I'm confused. Once you've loaded the BPF thing, 'all' tasks you care about should already be in the bpf class. So any fork() thereafter should not need to switch classes.

This means we can have this rwsem be strictly for the bpf tasks as Thomas suggested.

What are we missing?

Hello, Peter.

Maybe I am confused but let's say the BPF scheduler gets unloaded and reloaded. What would prevent a forking thread which didn't acquire the read lock from racing against the second loading?

Also, let's say we can make it conditional but would the extra complication be justifiable? percpu_down_read()'s hot path is one likely() cond test followed by this_cpu_inc() wrapped in preempt_disable(). I'm not really sure eliding that can justify much.

That said, as Thomas pointed out, the dl cancel path is silly. Let me clean that up.

Thanks.

tejun

From: Tejun Heo <[email protected]>

sched_fork() returns with -EAGAIN if dl_prio(@p). a7a9fc549293 ("sched_ext:
Add boilerplate for extensible scheduler class") added scx_pre_fork() call before it and then scx_cancel_fork() on the exit path. This is silly as the dl_prio() block can just be moved above the scx_pre_fork() call.

Move the dl_prio() block above the scx_pre_fork() call and remove the now unnecessary scx_cancel_fork() invocation.

---
Hello,

As this is pretty trivial, if no one objects, I'll apply this to sched_ext/for-6.11 after a bit.

Thanks.

 kernel/sched/core.c |   14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

@@ -4583,8 +4583,6 @@ late_initcall(sched_core_sysctl_init);
  */
 int sched_fork(unsigned long clone_flags, struct task_struct *p)
 {
-	int ret;
-
 	__sched_fork(clone_flags, p);
 	/*
 	 * We mark the process as NEW here. This guarantees that

@@ -4621,12 +4619,12 @@ int sched_fork(unsigned long clone_flags
 		p->sched_reset_on_fork = 0;
 	}
 
+	if (dl_prio(p->prio))
+		return -EAGAIN;
+
 	scx_pre_fork(p);
 
-	if (dl_prio(p->prio)) {
-		ret = -EAGAIN;
-		goto out_cancel;
-	} else if (rt_prio(p->prio)) {
+	if (rt_prio(p->prio)) {
 		p->sched_class = &rt_sched_class;
 #ifdef CONFIG_SCHED_CLASS_EXT
 	} else if (task_should_scx(p)) {

@@ -4652,10 +4650,6 @@ int sched_fork(unsigned long clone_flags
 	RB_CLEAR_NODE(&p->pushable_dl_tasks);
 #endif
 	return 0;
-
-out_cancel:
-	scx_cancel_fork(p);
-	return ret;
 }
 
 int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)

Applying to sched_ext/for-6.11.

Thanks.

tejun

Hello, Thomas.

On Thu, Jun 20, 2024 at 04:35:08AM +0200, Thomas Gleixner wrote:
...

More on the in-person discussion later, but can you please point to the many technical arguments that have been ignored? I addressed all the review points that PeterZ raised on the first RFC patchset and responded to most of the arguments that were raised. There haven’t been any technical feedbacks since then. If there are things that I missed, please point them out, I'd be happy to respond.

I would characterize that part of the discussion more nebulous than deep. You cited a really high number for where SCX is hooking into the scheduler core and then made wide-ranging suggestions including refactoring all the schedulers, which seemed vague and out of scope. I tried to probe and we didn't get anywhere concrete, which is fine. It's difficult to hash out details in such settings.

However, this is not true. During the discussion, I asked you multiple times to review the patches and point out the parts that are problematic so that they can be addressed and the discussion can become more concrete. You promised you would but didn't.

This patch series has been up in one form or another for almost two years. It took forcing the discussion in Richmond to get any responses from you and Peter, and what we got wasn't feedback on the patches, but verbal suggestions about SCX itself, and suggestions that X for Y would help us. When we attempted to follow up with you afterwards, we got no responses.

Now that Linus said he would pull it, there all of a sudden are discussions about the code. It seems likely that we wouldn't have that without Linus' email last week. The raised issues seem resolvable and mostly stem from trying to minimize changes to sched core. I don’t see a reason why this would warrant yet another delay. Why can’t we work it out in tree like any other problem?

Thanks.

tejun

Tejun!

It's not that nebulous. And fine if you tried and got nowhere, but did you give feedback to those failed attempts or started a discussion with anyone on the scheduler side? No.

Yes, I said I will do two things:

- Talk to Peter
- Review the stuff again

I talked to Peter to come up with a proper plan in order to give feedback. It was unfortunate that Peter vanished from the planet at that time due to his shoulder incident.

As I said to Linus, I'm sorry that I afterwards dropped the ball because I got dragged into other things and ran out of cycles.

I also asked you to reach out to the scheduler folks and work with them to get things moving again. Are you really claiming that you couldn't do that without me holding your hands?

I just checked and found three private mails from you which ended up in the wrong filter dated Feb 1, Feb 9, Feb 16. My bad that I dropped them, but definitely not because of desinterest or malice.

You can of course say you tried and I ignored you, but seriously?

If you really wanted to get my attention then you exactly know how to get it like everyone else who is working with me for decades.

Thanks,

tglx

I'll be honest, the only clear and consistent communication we've gotten about sched_ext was "no, please go away". You certainly did engage with face to face discussions, but at the end of the day/week/month the overall message didn't change.

I think we made a long list of private and public attempts to collaborate, and at some point we just had to be willing to take no for an answer.

So, yes, seriously. It's certainly not hard to find times where I ignored patches or emails, and I've obviously handed off almost all of my maintainership work to focus on other kernel topics, but I'm never comfortable when I watch maintainers try to push 100% of the collaboration burden onto everyone else around them.

The good news is that every merge window is a new chance to argue with each other, and I know Linus doesn't want us to save all the arguing for the start of the merge window, but saying every day is a new chance to argue sounded less optimistic somehow.

At any rate, I think sched_ext has a good path forward, and I know we'll keep working together however we can.

-chris

Chris!

The only time _I_ really told you "go away" was at OSPM 2023 when you approached everyone in the worst possible way. I surely did not even say "please" back then.

The message people (not only me) perceived was:

"The scheduler sucks, sched_ext solves the problems, saves us millions
and Google is happy to work with us [after dropping upstream scheduler development a decade ago and leaving the opens for others to mop up]."

followed by:

"You should take it, as it will bring in fresh people to work on the
scheduler due to the lower entry barrier [because kernel hacking sucks]. This will result in great new ideas which will be contributed back to the scheduler proper."

That was a really brilliant marketing stunt and I told you so very bluntly.

It was presumably not your intention, but that's the problem of communication between people. Though I haven't seen an useful attempt to cure that.

After that clash, the room got into a lively technical discussion about the real underlying problem, i.e. that a big part of scheduling issues comes from the fact, that there is not enough information about the requirements and properties of an application available. Even you agreed with that, if I remember correctly.

sched_ext does not solve that problem. It just works around it by putting the requirements and properties of an application into the BPF scheduler and the user space portion of it. That works well in a controlled environment like yours, but it does not even remotely help to solve the underlying general problems. You acknowlegded that and told: But we don't have it today, though sched_ext is ready and will help with that.

The concern that sched_ext will reduce the incentive to work on the scheduler proper is not completely unfounded and I've yet to see the slightest evidence which proves the contrary.

Don't tell me that this is impossible because sched_ext is not yet upstream. It's used in production successfully as you said, so there clearly must be something to learn from which could be shared at least in form of data. OSPM24 would have been a great place for that especially as the requirements and properties discussion was continued there with a plan.

At all other occasions, I sat down with people and discussed at a technical level, but also clearly asked to resolve the social rift which all of this created.

I thereby surely said several times: "I wish it would just go away and stay out of tree", but that's a very different message, no?

Quite some of the questions and concerns I voiced, which got also voiced by others on the list, have not been sorted out until today. Just to name a few from the top of my head:

- How is this supposed to work with different applications requiring different sched_ext schedulers?

- How are distros/users supposed to handle this especially when applications start to come with their own optimized schedulers?

- What's the documented rule for dealing with bugs and regressions on a system where sched_ext is active?

"We'll work it out in tree" is not an answer to that. Ignoring it and let the rest of the world deal with the fallout is not a really good answer either.

I'm not saying that this is all your and the sched_ext peoples fault, the other side was not always constructive either. Neither did it help that I had to drop the ball.

For me, Linus telling that he will merge it no matter what, was a wakeup call to all involved parties. One side reached out with a clear message to sort this out amicably and not making the situation worse.

Carefully avoiding the perception trap, may I politely ask what this is supposed to tell me?

Thanks,

tglx

Looping back to where I jumped into this thread, the context was you suggesting that if we'd just asked one more time, real collaboration might have started. I'm not trying to change the message by snipping this out of context, so if I've got this wrong, please do correct me.

I really don't object to the scheduler maintainers disliking sched_ext. Pluggable scheduling isn't a problem you wanted to solve, and bpf probably isn't how you would have solved it. We could have talked every day for the last 18 months, and by now we'd have a huge library of sonnets and haikus for each other, but sched_ext still wouldn't be merged.

I do object to rewriting history to claim that if we'd just used the secret handshake, things would somehow be different than they are now.

I joined a conference I'd never been to before and brought a laundry list of problems to the table. So, there's definitely truth to the perception that I came with an agenda and pushed it.

But, if I left anyone with the impression I thought the scheduler sucked, that wasn't my goal. Like every part of the kernel, there are problems the scheduler creates and problems it doesn't solve, and my goal is/was to invest in discussing and fixing both.

It's not a surprise that google and meta have a lot of problems in common. For us, collaborating with google is really rewarding and important for a bunch of subsystems, scheduler included.

Google is full of smart people, and carrying private patches is both expensive and wildly boring, and I'm always interested in why smart people use different strategies to solve problems we have in common.

It's part of a discussion we get into internally. Why does our kernel team exist at all? Are we just here to stabilize and ship Linus's kernel? Or are we here to try and advance our infrastructure by developing in the kernel?

Those are two pretty different paths, and I know that we optimize for things others don't care about, like contorting ourselves to make things easier to ship to production. But, we also optimize for feedback loops with workload owners that other distros and kernel developers would really envy.

It's a mixed bag, but I can say with certainty that adding features and optimizations to the upstream kernel is one of the least efficient ways to improve infra. Some of this is for really good reason, nobody wants all the tech debt that would come out of upstreaming every bad idea we've ever had. But there's a balance.

Before anyone gets upset with me, the upstream kernel can be the best kernel on the planet and still be a really inefficient way to land features and optimizations for applications.

Yeah, I'd say all of those things again, and I think I repeated some of it in this email too. It's one of my favorite topics of conversation so I won't bore everyone here (even more than I already have), but I'm always trying to find ways to improve the feedback loops between workloads and the kernel developers.

sched_ext has been really effective at that so far, both inside meta and for others in the community.

I still do! In a private email a few months ago I promised you that my one true workload modeling project was just a few months away. It still is just a few months away, but I do find the topic really interesting.

But, I disagree that we should stop sched_ext development until we find the perfect way to model the properties and requirements of applications. I'm really glad that eevdf landed with more of a iterate-in-the-kernel approach.

For me, the underlying general problems get solved with frequent experiments and tight feedback loops. It's all about iteration and cooperation with the application developers, and sched_ext absolutely does provide that.

Quoting from another email of yours in this thread

"I recently watched a talk about sched ext which explained how to model an execution pipeline for a specific workload to optimize the scheduling of the involved threads and how innovative that is. I really had a good laugh because that's called explicit plan scheduling and has been described and implemented in the early 2000s by academics already."

This is kind of exactly my point. We do agree that there are lots of well understood solutions to well understood problems that are missing from the kernel.

Linus answered this pretty effectively, and I don't see the need to expand on his comments.

No, it's really not a different message. The kernel tree is where kernel development happens best. Linus covered the comparison with RT as well, but I definitely do understand you've had to carry a few patches out of tree.

I'll let Tejun pitch in on this one.

Having worked for two or three distros (I'd count meta, we have customers too), distros pick and choose what to support based on what their customers need and pay for, and different distros will make different choices. I'd assume we'll have a spectrum:

- sched_ext is unsupported, talk to the vendor
- sched_ext is unsupported, but we'll give debugging a shot
- sched_ext is supported when you're using $supported schedulers

Vendors might provide optimized schedulers, but they have to support a huge range of distros and gloriously crusty enterprise kernels, so I can't see anyone making it a requirement.

It's not different from any other new kernel component, or old kernel component for that matter. What's the documented rule for dealing with bugs and regressions when a usb nic driver is loaded? If you're asking about bpf ABI, that's been covered in many other threads.

This last part is where you lost me. I've only seen a clear message to delay for any and every reason you can make stick. I know it's a jerk thing to say and I'm sorry, but that's how it feels from my end.

I was shooting for optimism here...we've all known each other a long time. We'll find ways to keep working together.

-chris

Hello,

On Mon, Jun 24, 2024 at 12:42:01PM -0400, Chris Mason wrote:
...

Long term, the tentative plan is to support a hierarchy of schedulers where the intermediate schedulers are responsible for granting CPUs to leaf schedulers which are responsible for scheduling tasks. Barret Rhoden has a framework called flux on top of ghost which already implements this albeit with compile time composition. Nothing is set in stone yet but it's likely that I'll follow what Barret is doing in many parts.

Taking a step back, because sched_ext currently supports a single system-wide scheduler, many of the techniques that the current crop of schedulers are playing with are pretty generic, at least to a class of problems - e.g. gaming.

Even for scx_layered which is the least generic in a sense, while we it's also used for a really specific ML setup internally too, what it's more widely used for is experimenting with things like soft affinity where e.g. workloads that are not latency sensitive are grouped into few hot running CPUs which are dynamically scaled to keep caches cleaner (and other effects too) for the latency sensitive parts of the system. Dan Schatzberg is trying to generalize that with scx_mitosis.

So, the summry is that there are plans to support a tree of schedulers but we're currently mostly focusing on more generic single scheduler experiments.

Thanks.

tejun

On Mon, Jun 24, 2024 at 11:11 AM Tejun Heo <[email protected]> wrote:
[...]

[...]

[...]

[...]

Yes, as I've just mentioned in another message in this thread, we plan to explore building UMCG-like per-process scheduling infra on top of sched_ext once it is merged into the mainline kernel. This is not a promise to actually do that (build such an infra); but rather a claim that we believe it is possible to do that and that we plan to look into the technical details once sched_ext is merged.

Thomas,

Respectfully, you have seriously misrepresented the facts of how everything has played out over the last 18 months of this feature being proposed upstream. I think it's important to set the record straight here, so let's get concrete and look at the timeline for sched_ext's progress since its inception.

1. We actually first met at LPC in 2022, at which time I mentioned this project to you and asked if you had any thoughts about it. I understood your response to be, essentially, , "I don't like BPF because it introduces UAPI constraints on the entire kernel." My response was that struct_ops programs and kfuncs aren't beholden to UAPI constraints as they're exclusively kernel APIs.

This was a random off-the-cuff discussion, and we had never met before, so I don't fault you for not engaging on this specific point in future discussions. I do, however, think it's important to highlight that we've been trying to engage with you on this in one form or another for about 18 months at this point.

2. The initial RFC [0] is sent out in late 2022. Peter to his credit did leave quite a few constructive comments that result in material improvements to the patch set, but the main takeaway was the thread in [1], which essentially boiled down to, "Hard no to this whole patch set, stop trying to push BPF into more places and ship random debug patches."

[0]: https://lore.kernel.org/all/[email protected]/
[1]: https://lore.kernel.org/all/[email protected]/

3. v2 [2] of the patch set is sent to the list in late January 2023. Nobody from the scheduler community responds.

[2]: https://lore.kernel.org/bpf/[email protected]/

4. v3 [3] is sent in mid March. Again, no engagement from the scheduler community.

[3]: https://lore.kernel.org/all/[email protected]/

5. Peter sends this [4] email in a totally separate thread (which you were cc'd on, and we were not) 2 weeks before OSPM:

[4]: https://lore.kernel.org/lkml/[email protected]/

Ok, it's LKML and people say things, but let's be honest here. You're accusing _us_ of being malicious and not collaborating with anybody, while you guys are not only ignoring our patch sets, but also slinging ad-hominems about the project in completely unrelated threads without even cc'ing us.

6. Also before OSPM, Peter sends the initial EEVDF patch set [5]. I replied [6] (again, before OSPM) explaining that it severely regresses one of our largest and most critical workloads, and got no response from Peter.

Anecdotally, a lot of people have complained to me in private discussions at various conferences that they're very unhappy about EEVDF, and have asked me to chime in and say something to prevent it from being merged. I decided not to, both because I didn't want to waste my time (Peter didn't respond when I engaged before), and because I thought it would be in poor taste to publicly try and derail Peter's project while at the same time expecting him to engage with us on a project that I knew he wasn't happy about.

[5]: https://lore.kernel.org/lkml/[email protected]/
[6]: https://lore.kernel.org/lkml/20230410031350.GA49280@maniforge/

Anyways, with all that in mind, when you say this:

It's really hard to assume good intent here. If we raise issues on the list, we're ignored (yes, ignored, not oops it fell through my filter). If we bring them up in in-person discussions, we're made out to be complainers who have no intention of contributing anything. My perception from having attended these conferences is that there's literally nothing we can say that will make you guys happy (beyond meekly agreeing to just scrap everything).

On that note, let's talk about OSPM:

7. Chris and I presented [7] at OSPM in mid-April. There was definitely some useful discussion that took place, but the basic message we received from both you and Peter was again, "No, go away." My recollection is that we were essentially given the same non-technical talking point of, "People will use this and it will fragment the scheduler space". By the way, Peter also applied this "people shouldn't use it" perspective to the debugfs knobs like migration_cost_ns, which we've also used to optimize our workloads by several %.

I'm not sure exactly what we're supposed to do with the feedback of, (paraphrasing) "Just like the debugfs knobs, people will use this thing even though they shouldn't be." From my perspective, we were just being told to not do something that gives us massive performance wins, with no other suggestions or attempts at collaboration. And no, I wouldn't count your suggestion of "have user space send more information to the kernel scheduler" as something we could practically use or apply. More on that specific point below.

[7]: https://www.youtube.com/watch?v=w5bLDpVol_M

Now, I do agree with you that in general I could have delivered our message a bit better. Yes, I did try to make the argument that sched_ext would benefit fair.c (which I really do still believe), but Juri also gave me feedback afterwards that my talk probably would have gone over better if I'd first submitted patches to fair.c to show some good will. Fair enough. I was still very new to the Linux kernel world at that point, so I'll take the blame for not really understanding how things are done. Mea culpa. That said, I did attempt to submit a patch set that applies a lesson we learned from sched_ext to fair.c, but ultimately got nowhere with it (more on that below).

Well, we agreed with you that it might be a good path forward, but you were also proposing a hand-wavey idea with nothing concrete and no code behind it. It felt like a deflection at the time, and it feels like one now. Also, let's point out that you apparently didn't feel the need to say anything about how applications should tell the scheduler about their needs for EEVDF. I'd love to know why it wasn't relevant then (again, just two weeks before OSPM).

I think there is a ton of evidence which proves the contrary (XDP, FUSE, etc), but given that Linus already covered this I don’t think we need to repeat ourselves.

Anyways, let's continue going over the timeline.

9. An RFC [8] for the shared wakequeue (later called shared runqueue) patches is sent in June 2023. This patch set was based on experiments conducted in sched_ext, and I decided it was important to prioritize this based on the feedback I was given at OSPM. Peter gave a lot of helpful feedback on this patch set.

[8]: https://lore.kernel.org/lkml/[email protected]/

10. v2 [9] of the SHARED_RUNQ patch set is sent in July 2023. Peter again gives a lot of useful feedback. The environment in general feels very productive and collaborative, but the patch set isn't quite ready yet.

[9]: https://lore.kernel.org/lkml/[email protected]/

9. On the same day as the SHARED_RUNQ patches, v4 [10] of the sched_ext patch set is sent. After two weeks of silence, Peter decides to respond [11] to this one with an official NAK, again with no technical or actionable feedback.

[10]: https://lore.kernel.org/lkml/[email protected]/
[11]: https://lore.kernel.org/lkml/[email protected]/

10. SHARED_RUNQ v3 [12] is sent in early August. No response from Peter, despite requesting his input on one or two of the patches. This is an example of why contributing to the core scheduler is such a pain. I spent at least 3-4 weeks of time on this patch set, and it ended up going nowhere, partly (but not entirely) due to Peter disappearing. Frankly, it seems like it got even more scrutiny than EEVDF did. Eventually, EEVDF ended up causing the feature to not work as well on hackbench [13], so I stopped bothering.

[12]: https://lore.kernel.org/lkml/[email protected]/
[13]: https://lore.kernel.org/lkml/[email protected]/

11. I presented on sched_ext at Kernel Recipes [14] in September 2023, which you attended. In a side-channel conversation that you and I had, you reiterated your point that you thought we were pushing the completely wrong message by saying that we think this will help fair.c, and made the request that we do more to make it clear that there won't be a maintenance burden on scheduler maintainers and distros. In particular, you asked me to make it more obvious when a sched_ext scheduler is loaded in the event of a system issue so that scheduler maintainers and distros could ignore any bug reports that come in for those scenarios. If we did that, you said, you would work with Peter on coming up with an amicable solution that left everybody happy, and you would chime in on the list (just like you said you would to Tejun at the Maintainer's Summit) to make forward progress.

[14]: https://kernel-recipes.org/en/2023/schedule/sched_ext-pluggable-scheduling-in-the-linux-kernel/

After Kernel Recipes, I implemented your request in [15] (see [16] for the patch in the latest patch set). So your claim that we never did anything to meet you guys half way on anything is not true. Not only did we actually implement one of your requests, but our one request to you (to chime in on the list), you never did.

You've said in other threads that you didn't have cycles for 7 months. Ok, it happens and ultimately we’re all volunteers when it comes to upstream work, but frankly I find it very hard to believe that you had literally no time in a 7 month window to review the patch set. Hearing you say that, while also at the same time trying to accuse us of being non-collaborative and malicious, feels a bit hypocritical to say the least.

[15]: https://github.com/sched-ext/sched_ext/pull/66
[16]: https://lore.kernel.org/bpf/[email protected]/

12. v5 [17] of the series is sent in early November 2023. Once again, we get no feedback from anyone in the scheduler community.

[17]: https://lore.kernel.org/bpf/[email protected]/

13. Maintainers Summit 2023 [18] happens. You, Tejun, and Alexei discussed the current situation with sched_ext. You raise some issues you have with integration, and you agree to bring the discussion to the list, which as we all know at this point, never happened. You also request that we fix the cgroup hierarchical scheduling mess, even though our only involvement was updating the existing CPU controller to use cgroup v2 APIs. This was proposed as a trade for you talking to Peter and letting sched_ext go in. While we didn’t feel great about a quid-pro-quo for getting sched_ext merged, we agreed to discuss it with Google and get back to you.

[18]: https://lwn.net/Articles/951847/

14. After maybe 6-ish weeks, we aligned with Google about dedicating resources to fixing the cgroup hierarchical scheduling mess, purely as a token of goodwill to you and Peter. At this point we started trying to send you private pings to coordinate (obviously we weren’t about to sink a ton of time into it without circling back with you first). We sent you several private pings between this point and when v6 landed, with no responses.

15. You can't be serious.

Firstly, sched_ext was discussed at OSPM 2024. Andrea Righi presented [19] on scx_rustland. I wasn't able to attend because I had other obligations, but it was certainly discussed. Also, if you had replied to any of our private pings and asked to meet at OSPM 2024, we could have absolutely made time for it.

[19]: https://www.youtube.com/watch?v=HQRHo8E_4Ks

But regardless, let's take a moment to reflect on what you're trying to claim here and in your other emails about our supposed lack of collaboration. You’re saying we should have used sched_ext to help solve the underlying problems in the scheduler, and that it was a mistake to not attend OSPM 2024? Here's a list of what we have done over the last 18 months:

- I've been attending Steven Rostedt's monthly scheduler meeting regularly, and have discussed sched_ext at length with many people there; such as Juri, Daniel Bristot de Oliveira, Joel Fernandes, Steven, and Youssef Esmat. We’ve also discussed EEVDF, and possible improvements that could be implemented following fruitful sched_ext experiments.
- I've attended and presented at a multitude of other conferences, including OSPM 2023, LSFMM (multiple years), KR, and LPC
- Tejun attended the Maintainers Summit in 2023
- We cc'd Peter on every single patch set
- We sent you many private emails

Yet, you’re trying to claim that we should have attended OSPM 2024 and shared some data that could have been used to improve the scheduler, and because we didn’t, _we’re_ the ones who don’t want to collaborate? Sorry, but any perceived lack of data sharing on our part is 100% due to your guys’ lack of effort or desire to interact with us in literally any medium, or at any location. It really feels like you just picked OSPM 2024 because you realized it was the one conference that neither Tejun nor I could attend. For the record, you didn’t reach out to either of us to discuss meeting there. I would have made it work if it was important to you guys. Well, come to think of it, you hadn’t communicated with us in literally any capacity until Linus agreed to take the series in this patch set, so I guess it goes without saying that you didn’t ping us for OSPM 2024.

16. We send out v6 [20], and get public support for the project from two distros, Valve, Google, ChromeOS, etc. Linus decides that it's time to merge the project, and now all of a sudden you come out of the woodwork and start slinging mud and accusing us of not collaborating. And here we are now.

[20]: https://lore.kernel.org/bpf/[email protected]/

QED. _That’s_ what's actually happened over the last 18 months. We've made repeated attempts to collaborate, even going so far as agreeing to your private request that we fix the cgroup hierarchical mess, in a desperate bid to try and somehow make you guys happy and enable us to work collaboratively. Yet only now do you join the conversation, after countless private pings and private agreements that you didn't honor, once Linus _forced your hand_, only to accuse us of being unwilling to cooperate?

If I sound indignant, it’s because I am. You guys made the decision to approach every single conversation with the singular purpose of trying to get the project derailed. Fine, I understand that you don't like it, and that you probably wouldn't have implemented pluggable scheduling with BPF if you had a choice. But to now come in at the 11th hour and try to blame _us_ for not collaborating with you, when it was you who ignored emails, slung mud, and failed to honor spoken agreements, is pretty brazen.

All of that said, we of course remain committed to all the things we've said about working together with the community upstream. I actually totally agree with you that it would be a good idea to clean up the integration points. As we've said before, we didn't do that originally because we were trying to have as small of a footprint as possible in code the that you guys would have to deal with (which by the way was also in line with the feedback you gave me at KR). But no worries, now that the record is cleared, we’re happy to move forward and work with you. It’s been our goal the entire time.

As mentioned above, this was discussed in person, but you never met us half way. There's only so much we can do if you choose to ignore all of our private email pings and ghost us for 7 months (actually closer to 10 months if you count our discussion at KR 2023).

Chris responded to the rest of your email, so I'll cut my already excessively long reply here. The one last thing that I do want to say that I really hope we can eventually put this ugliness behind us. I admire how you think about and approach software engineering, and I would love for your input on how we can do things better. I'm sorry that this reply had to be so serious and accusatory, but you forced our hand by approaching this entire conversation this way, and by being blatantly dishonest about our private discussions and private efforts to reach out to you to collaborate. Hopefully we can have beers in Vienna, and move on.

Thanks,
David

On Wed, Jun 19, 2024 at 7:35 PM Thomas Gleixner <[email protected]> wrote:

[...]

Hi Thomas,

I'm sorry I missed this callout re: UMCG last week.

The efforts were not dropped on our side, I assure you. For example, I posted this last year:
https://lore.kernel.org/lkml/CAPNVh5eNEBu+gcex9pAej-_YN3zMKkG+rruXhopqS6EaG-izVQ@mail.gmail.com/T/
and I got no indication re: how I should proceed.

There were several earlier LKML posts that similarly did not result in any actionable feedback. Based on this clear (maybe just perceived? if so, I apologize) lack of interest in making UMCG ready for upstream/mainline, we've decided to wait for sched_ext to get merged; sched_ext already existed at the time, e.g.
https://lore.kernel.org/lkml/[email protected]/.

I believe that sched_ext is flexible enough that, once merged, it will allow us to build UMCG-like per-process schedulers on top of it, so I see no reason in pushing both UMCG and sched_ext, given the difficulty of getting anything merged. If I had to choose between UMCG and sched_ext getting upstream (and I do not see both UMCG _and_ sched_ext getting merged together any time soon), I'd choose sched_ext, because it naturally opens up more opportunities to tailor scheduling to different workloads.

Again, I appreciate the initial help and feedback you and Peter provided re: UMCG; but then things stalled and I was not getting any clear indication how to proceed; and given that UMCG can be built on top of sched_ext (or ghost), and a clear (or perhaps also just perceived) preference by sched maintainers to avoid competing solutions, I now believe that sched_ext should be merged first.

Thanks,
Peter

[...]