MAZE: Mazed processes monitor

Hi all.

This patch provides the mazed process monitor,named MAZE.
MAZE's purpose is to notify or terminate registered processes when they are mazed. The definition of mazed process is a process using excessive CPU cycles, that is long time keeping TASK_RUNNING state.
MAZE detects mazed processes and sends specified signals to them. This implements a CGL (Carrier Grade Linux) requirement (AVL.14.0).

Possible uses:
* High-Availability system
* system using many Real-Time processes such as embedded

Please any comments!

Hirofumi Nakagawa

--- Usage
Add monitoring process.

% echo "1234 10000 20000 24 9" > /proc/maze/entries

The numeric values are "pid","soft limit [msec]","hard limit [msec]", "soft signal" and "hard signal".

Get monitoring process list.

% cat /proc/maze/entries

--- Example
As follows example express the feature of MAZE function.

--
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])

{
    int state,i;
    char cmd[128];

    sprintf(cmd, "echo \"%d 10000 20000 24 9\" > /proc/maze/entries", getpid());
    system(cmd);

    if (!fork()) {
        printf("Mazed process start.\n");
        for(;;);
    }

    if (!fork()) {
        printf("Not mazed process start.\n");
        for(i = 0;i < 20; i++)
            sleep(1);
        printf("Not mazed process finish.\n");
        exit(0);
    }

    for (i = 0;i < 2; i++)
         wait(&state);

    return 0;
}

--
---

Please include (and maintain) all the above changelog information with the patch itself rather than separately. Because if I were to merge this patch, I'd just have to copy-n-paste it all over anwyay.

Please send the text of AVL.14.0 (or a sufficient part of it for us to understand the requirement) and maintain that within the changelog also.

The immediate quesstion is, of course, "why does this need to be done in-kernel?". I am unable to answer that from the provided changelog information, and I should be able to do that.

Thanks.

Hi Andrew,

Thank you for your comment and review.

It needs to be done in kernel because excessive CPU cycle usage is directly detected watching kernel internal task_struct.state. It's hard to get that information outside the kernel.

Additionally, this patch's aim is to implement a
CGL (Carrier Grade Linux) requirement (AVL.14.0).

There are specs that can not be implemented outside the kernel. These are;
- Communication between the monitoring process and the kernel
- Ability to define policy based on process events including process/thread creation and exit, and,
- Ability to set the CPU cycle threshold to a resolution of one millisecond.

Thanks.
Hirofumi Nakagawa

Isn't that very similar to RLIMIT_CPU? The main difference seems to be that they're regularly reset and that they can be more fine grained than seconds.

How about you implement a way to change the RLIMIT_CPU limit of a running task (and possibly fix it to use a finer grained unit if you need <1s resolution).

Then you could run a user space daemon running with lower priority that just regularly resets the RLIMIT_CPUs of all running processes.

If some RT process uses so much time that the user daemon cannot keep up its cpu time limit will be eventually exceeded and it will be killed.

I think that would be a far cleaner and generic way to implement this.

-Andi

Thank you for your comments and feedbacks.
I got valuable comments.
I'll study your comments closely and post take2.

Thanks.
Hirofumi Nakagawa

Hi Andi,

Thank you for your comment and proposal.

I also watch normal processes, which run into infinite loop but cleanly scheduled out by sched when a time slice expires. In this case, your deamon still can reset the counter.

The differences with MAZE and rlimit are as follows.
- MAZE detects excessive CPU cycle usage, but rlimits limits total amount of CPU usage.
MAZE can safely handle CPU intensive but correctly running processes.

- User processes can add watched processes in MAZE.

- MAZE allows users to choose a way how to act on the process, selecting which signal to send.

Following example shows the differences of features MZE and rlimit

----
void foo(void)

{
    for(;;);
}

----
This code receives a signal shortly,
if registered under either rlimit or MAZE.

----
void foo(void)

{
    for(;;) {
        sleep(1);
    }
}

----
This code receives a signal in case of rlimit.
But, it doesn't receive a signal under MAZE.
There are cases when you need to distinguish these two types.

Thanks.
Hirofumi Nakagawa