udevd: don't use alarm() for timeouts
PATCH v2 1/1 udevd: don't use alarm() for timeouts
alarm() is per-process; we can't use it for timeouts in a multi-threaded udevd. We need to explicitly track the total time spent waiting for each event.
There is an issue here if the timeout expires in run_program(). If run_program() returns without calling wait(), the process it forked will become a zombie when it finally exits. Currently, udev-event is implemented as a process, so the zombie will be reparented to init and reaped once the event is finished. But that won't work for threads.
The solution is to fork twice:
udevd (event thread)
udevd child process
command
When the timeout expires, the event thread can return immediately. The child process will stay blocked in wait(), until the command finally finishes (or is killed). We'll get a nice process tree showing that the the hung process was started by udevd :-).
The child process passes the exit status of the command over a pipe. This also provides a solution to the lack of a timeout in any of the standard wait() system calls. We can set a timeout on select() when waiting on the pipe.
---
v2: don't use the timeout from udev_device_get_timeout(), it's completely
separate from the udev event timeout.
udev/test-udev.c +0 -7
@@ -69,9 +69,6 @@ int main(int argc, char *argv[])sigaction (SIGINT , &act,NULL );sigaction (SIGTERM , &act,NULL ); -/* trigger timeout to prevent hanging processes */ -alarm (UDEV_EVENT_TIMEOUT ); - action =getenv ("ACTION" ); devpath =getenv ("DEVPATH" ); subsystem =getenv ("SUBSYSTEM" );
@@ -98,10 +95,6 @@ int main(int argc, char *argv[]) event =udev_event_new (dev); err =udev_event_execute_rules (event, rules); -/* rules may change/disable the timeout */ -if (udev_device_get_event_timeout (dev) >=0 ) -alarm (udev_device_get_event_timeout (dev)); -if (err =0 && !event->ignore_device &&udev_get_run (udev))udev_event_execute_run (event);
udev/udev-event.c +321 -4
382 lines not on this page · raw message
udev/udev-rules.c +25 -21
119 lines not on this page · raw message
udev/udev-util.c +0 -211
214 lines not on this page · raw message
udev/udev.h +5 -2
@@ -28,6 +28,7 @@#define DEFAULT_FAKE_PARTITIONS_COUNT 15#define UDEV_EVENT_TIMEOUT 180 +#define USECS_PER_SECOND (1000 * 1000)#define UDEV_CTRL_SOCK_PATH "@" UDEV_PREFIX "/org/kernel/udev/udevd"
@@ -64,6 +65,7 @@ struct udev_event {uid_t uid;gid_t gid;struct udev_list_node run_list; +int timeout_usecs;pid_t pid;int exitstatus;time_t queue_time;
@@ -99,6 +101,9 @@ int udev_event_execute_run(struct udev_event *event);size_t udev_event_apply_format (struct udev_event *event,const char *src,char *dest,size_t size);int udev_event_apply_subsys_kernel (struct udev_event *event,const char *string,char *result,size_t maxsize,int read_value); +int udev_event_run_program (struct udev_event *event,const char *command, +char *result,size_t ressize,size_t *reslen); +/* udev-watch.c */ extern int inotify_fd;
@@ -120,8 +125,6 @@ int util_delete_path(struct udev *udev, const char *path);int util_unlink_secure (struct udev *udev,const char *filename);uid_t util_lookup_user (struct udev *udev,const char *user);gid_t util_lookup_group (struct udev *udev,const char *group); -int util_run_program (struct udev *udev,const char *command,char **envp, -char *result,size_t ressize,size_t *reslen);int util_resolve_subsys_kernel (struct udev *udev,const char *string,char *result,size_t maxsize,int read_value);
udev/udevd.c +0 -7
@@ -155,16 +155,9 @@ static void event_fork(struct udev_event *event)sigaction (SIGCHLD , &act,NULL );sigaction (SIGHUP , &act,NULL ); -/* set timeout to prevent hanging processes */ -alarm (UDEV_EVENT_TIMEOUT ); -/* apply rules, create node, symlinks */ err =udev_event_execute_rules (event, rules); -/* rules may change/disable the timeout */ -if (udev_device_get_event_timeout (event->dev) >=0 ) -alarm (udev_device_get_event_timeout (event->dev)); -/* execute RUN= */ if (err =0 && !event->ignore_device &&udev_get_run (event->udev))udev_event_execute_run (event);
Quoted 12 lines · Alan Jenkins
alarm()is per-process; we can't use it for timeouts in a multi-threaded udevd. We need to explicitly track the total time spent waiting for each event.There is an issue here if the timeout expires in
run_program(). Ifrun_program()returns without callingwait(), the process it forked will become a zombie when it finally exits. Currently, udev-event is implemented as a process, so the zombie will be reparented to init and reaped once the event is finished. But that won't work for threads.The solution is to fork twice:
Quoted 8 lines
udevd (event thread)
udevd child process
commandWhen the timeout expires, the event thread can return immediately. The child process will stay blocked in
wait(), until the command finally finishes (or is killed). We'll get a nice process tree showing that the the hung process was started by udevd :-).
The event runs all programs serialized, one after the other, can't we just kill the program that does not return in time, and wait for it to cleanup the process, instead of just exiting the event process?
Thanks,
Kay
Quoted 24 lines · Kay Sievers
Quoted 12 lines · Alan Jenkins
alarm()is per-process; we can't use it for timeouts in a multi-threaded udevd. We need to explicitly track the total time spent waiting for each event.There is an issue here if the timeout expires in
run_program(). Ifrun_program()returns without callingwait(), the process it forked will become a zombie when it finally exits. Currently, udev-event is implemented as a process, so the zombie will be reparented to init and reaped once the event is finished. But that won't work for threads.The solution is to fork twice:
Quoted 9 lines
udevd (event thread)
udevd child process
commandWhen the timeout expires, the event thread can return immediately. The child process will stay blocked in
wait(), until the command finally finishes (or is killed). We'll get a nice process tree showing that the the hung process was started by udevd :-).
Quoted 4 lines
The event runs all programs serialized, one after the other, can't we just kill the program that does not return in time, and wait for it to cleanup the process, instead of just exiting the event process?
That makes sense if you also change the timeout model - from per-event timeouts, to per-command timeouts.
The second fork is cheap, because it's a vfork. But if you like per-command timeouts, I would be happy to see it go away.
I'm not sure about killing. Do we need to escalate to SIGKILL? Do we e.g. allow half the timeout before sending SIGTERM, then another half before sending SIGKILL? The process could even be unkillable - is it ok to block after SIGKILL, or do we need another timeout?
I could keep a list of timed out processes instead, and reap them on SIGCHLD. Would that be better?
There are other workarounds for the lack of a timeout in sys_wait(), so I don't think that's a problem. We can require that commands close stdout & stderr pipes when they exit - i.e. do not pass them on to a long-running child. (At the moment there's a debian script "net.agent" which has to do this for debug mode - it would need fixing to always do it).
Alan
Quoted 6 lines · Alan Jenkins
Quoted 3 lines
The event runs all programs serialized, one after the other, can't we just kill the program that does not return in time, and wait for it to cleanup the process, instead of just exiting the event process?
That makes sense if you also change the timeout model - from per-event timeouts, to per-command timeouts.
We can still track how long we a running and use that timeout when waiting for the actual program to finish, right?
Quoted 2 lines
The second fork is cheap, because it's a vfork. But if you like per-command timeouts, I would be happy to see it go away.
I guess, it's simpler to understand and vfork() is something that even POSIX removed from the spec. :)
Quoted 3 lines
I'm not sure about killing. Do we need to escalate to SIGKILL? Do we e.g. allow half the timeout before sending SIGTERM, then another half before sending SIGKILL?
I guess a few seconds would be good enough before we try to finally KILL it.
Quoted 2 lines
The process could even be unkillable - is it ok to block after SIGKILL, or do we need another timeout?
Hmm, yeah, maybe we just block.
I don't remember any problems with the timeouts, they are mainly there to prevent bad things that make udev operations unreliable like people running apps with user interfaces from udev rules, which we see every other month. So I guess killing would be the right thing to do anyway. :)
Quoted 2 lines
I could keep a list of timed out processes instead, and reap them on SIGCHLD. Would that be better?
You mean as an alternative to blocking the event process?
Quoted 5 lines
There are other workarounds for the lack of a timeout in
sys_wait(), so I don't think that's a problem. We can require that commands close stdout & stderr pipes when they exit - i.e. do not pass them on to a long-running child. (At the moment there's a debian script "net.agent" which has to do this for debug mode - it would need fixing to always do it).
I don't think we can really assume anything from called programs. :)
Thanks,
Kay
Kay Sievers wrote:
<Snip. Apologies for snipping all your questions. I think it's become a little confused and this is the most important issue.>
[Could run_program() avoid forking twice in a threaded udevd, and kill timed-out commands so they don't become zombies when they eventually finish?]
Quoted 9 lines · Alan Jenkins
Quoted 7 lines
There are other workarounds for the lack of a timeout in
sys_wait(), so I don't think that's a problem. We can require that commands close stdout & stderr pipes when they exit - i.e. do not pass them on to a long-running child. (At the moment there's a debian script "net.agent" which has to do this for debug mode - it would need fixing to always do it).
Quoted 5 lines
I don't think we can really assume anything from called programs. :)
Thanks,
Kay
Sounds like the voice of experience.
The main constraint I'm working around is the lack of a timeout in sys_wait*(). Forking twice allows the parent process to block on a pipe instead, and use a select() timeout.
If we don't want to fork twice, and we can't rely on EOF on the command's stdout pipe, I think the only alternative is to wait for SIGCHLD. The signal handler would check for finished PIDs with WNOHANG, and look them up in a list to see which thread needs waking up. In that case, it's not essential to kill timed out commands. The signal handler can reap them just as easily as it reaps a command which is being waited for by an event.
Forking twice is simpler though. It's more direct, and it also easily works with threading disabled. At the moment my thread-specific code is reasonably contained. I have a ./configure option which switches back to using separate processes, in case pthreads support is missing or broken on some platforms.
Thanks
Alan