Skip to contentSkip to conversation

Kernel 2.2: tq_scheduler functions scheduling and waiting

DISCUSSION Arthur Naseef 2 participants 5 messages May 28, 2001 – May 30, 2001

lore
Share & tools
lore
#0 lore raw

All:

I have been diagnosing kernel panics for over a week and I have concerns with the use of tq_scheduler for which I was hoping I could get some assistance.

Is it considered acceptable for functions in the tq_scheduler task list to call schedule? Is it acceptable for such functions to wait on wait queues? What limitations exist?

As near as I can determine, the TTY driver code makes use of the tq_scheduler list for such purposes.

In my testing, I am running with 96 TTY devices (talking to a high-density modem card) and I consistently achieve kernel panics when the system is under heavy swapping. I am continuing to diagnose the problem. The kernel panics are triggered mostly in goodness() and del_from_runqueue(), as indicated by ksym_oops and gdb, and I suspect the run queue is getting corrupted.

In spite of this testing, I believe that I have an argument against tq_scheduler functions waiting on wait queues, but I have not thoroughly convinced myself that (a) this was not already known, and (b) this is already happening in existing kernel code.

Any help is greatly appreciated.

-art

Arthur Naseef

P.S. If this information is availed through existing documentation,
searches, or other widely available resources, I would greatly appreciate references to this material. All of my searches to date have yielded few results and nothing definitive.

#1 lore raw
Quoted 10 lines · Arthur Naseef

All:

I have been diagnosing kernel panics for over a week and I have concerns with the use of tq_scheduler for which I was hoping I could get some assistance.

Is it considered acceptable for functions in the tq_scheduler task list to call schedule? Is it acceptable for such functions to wait on wait queues? What limitations exist?

When a task wants to exit, it cleans up all its stuff, sets its state to TASK_ZOMBIE and then calls schedule(). The scheduler takes it off the runqueue and the task is never again executed. It's just a couple of stack pages which are waiting for someone in wait4() to release.

But imagine what happens if the TASK_ZOMBIE task hits schedule() and finds a tq_scheduler task to run. And that task calls schedule(). In state TASK_ZOMBIE. Messy.

At the very least, the schedule() call will never return.

If the tq_scheduler task sets current->state to TASK_[UN]INTERRUPTIBLE (as it should) before calling schedule() then it has overwritten TASK_ZOMBIE and the task which is trying to exit has become magically resurrected. As far as I can tell, the "dead" task will run again, do the `fake_volatile' thing in do_exit() and try to go zombie again.

It would be very interesting to change the test in schedule():

        sti();
-       if (tq_scheduler)
+       if (tq_scheduler && current->state != TASK_ZOMBIE)
                goto handle_tq_scheduler;

It's all rather unpleasant, and tq_scheduler was killed in 2.4. I suggest you take a look at all the serial drivers in 2.4, see how I converted them to use schedule_task(). Someone kindly ported schedule_task() to 2.2.recent, so you should be able to use that in the same way.

-

#2 lore raw

Andrew:

Excellent. I will look at the 2.4 sources.

In addition to the TASK_ZOMBIE issue you mention, I believe there is an issue of false termination of wait queues. Consider this:

- Task places itself on a wait queue
- Calls schedule()
- tq_scheduler function does the same

Now, there are two events which could place the task in TASK_RUNNING and no clear way to differentiate. And, since most of the kernel code does not check that the wait condition was actually met, this could lead to all types of problems, right?

-art

-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Andrew Morton
Sent: Monday, May 28, 2001 10:28 PM
To: Arthur Naseef
Cc: [email protected]
Subject: Re: Kernel 2.2: tq_scheduler functions scheduling and waiting

Quoted 10 lines · Arthur Naseef

All:

I have been diagnosing kernel panics for over a week and I have concerns with the use of tq_scheduler for which I was hoping I could get some assistance.

Is it considered acceptable for functions in the tq_scheduler task list to call schedule? Is it acceptable for such functions to wait on wait queues? What limitations exist?

When a task wants to exit, it cleans up all its stuff, sets its state to TASK_ZOMBIE and then calls schedule(). The scheduler takes it off the runqueue and the task is never again executed. It's just a couple of stack pages which are waiting for someone in wait4() to release.

But imagine what happens if the TASK_ZOMBIE task hits schedule() and finds a tq_scheduler task to run. And that task calls schedule(). In state TASK_ZOMBIE. Messy.

At the very least, the schedule() call will never return.

If the tq_scheduler task sets current->state to TASK_[UN]INTERRUPTIBLE (as it should) before calling schedule() then it has overwritten TASK_ZOMBIE and the task which is trying to exit has become magically resurrected. As far as I can tell, the "dead" task will run again, do the `fake_volatile' thing in do_exit() and try to go zombie again.

It would be very interesting to change the test in schedule():

        sti();
-       if (tq_scheduler)
+       if (tq_scheduler && current->state != TASK_ZOMBIE)
                goto handle_tq_scheduler;

It's all rather unpleasant, and tq_scheduler was killed in 2.4. I suggest you take a look at all the serial drivers in 2.4, see how I converted them to use schedule_task(). Someone kindly ported schedule_task() to 2.2.recent, so you should be able to use that in the same way.

-

#3 lore raw
In reply to Andrew Morton
Quoted 8 lines · Arthur Naseef

Andrew:

Excellent. I will look at the 2.4 sources.

In addition to the TASK_ZOMBIE issue you mention, I believe there is an issue of false termination of wait queues. Consider this:

Quoted 9 lines

- Task places itself on a wait queue
- Calls schedule()
- tq_scheduler function does the same

Now, there are two events which could place the task in TASK_RUNNING and no clear way to differentiate. And, since most of the kernel code does not check that the wait condition was actually met, this could lead to all types of problems, right?

Yes. The situation where one task is on two waitqueues is rare, but does happen. And yes, there is code out there which does a bare schedule() and *assumes* that once the schedule has returned, the thing it was waiting for has indeed occurred.

Generally this is poor practice - it's safer to loop over the schedule() call until the condition you're sleeping on has been tested.

You really shouldn't be sleeping in this way on tq_scheduler if there's any way in which the sleep can take an extended period of time. You may end up putting important kernel tasks to sleep.

Best to use schedule_task(), or an independent kernel thread.

-

#4 lore raw

I have tested with a kernel thread running the tq_scheduler and it is much more stable. The kernel still ran into a problem in n_tty.c in which the compiler optimized-out the check "if (!tty)" in n_tty_set_termios(); I am still investigating the right solution to this.

As a long term fix, I will review the 2.4 and latest 2.2 sources.

Quoted 9 lines

Yes. The situation where one task is on two waitqueues is rare, but does happen. And yes, there is code out there which does a bare schedule() and *assumes* that once the schedule has returned, the thing it was waiting for has indeed occurred.

Generally this is poor practice - it's safer to loop over the schedule() call until the condition you're sleeping on has been tested.

I see your point. It would prevent this type of problem if all code waiting for conditions made certain those conditions were met. However, given the way the kernel works, it is not necessary to check unless the task specifically expects more than one condition to awaken it - at least it wasn't until tq_scheduler was introduced. Actually, that is not fair either - only when functions in tq_scheduler starting "blocking" did this become a problem.

It would help me tremendously if these types of limitations and requirements for working in the kernel were well documented. It takes significant effort to determine the requirements, and to verify that my understanding is correct.

Quoted 5 lines

You really shouldn't be sleeping in this way on tq_scheduler if there's any way in which the sleep can take an extended period of time. You may end up putting important kernel tasks to sleep.

I agree. In addition, even if the tq_scheduler function did check for its own condition, a problem still exists when the task returns to the code using the first wait queue before its condition is met; since the code using the second wait queue would set the task state to running and would not set it back (which it couldn't without knowing the conditions to check).

Quoted 4 lines

Best to use schedule_task(), or an independent kernel thread.

-