An autrace that follows forks

I plan to write a version of autrace that follows forks. When run in follow forks mode, it will ptrace the child. Each time the child or one of its descendents creates a new process, the program will use the API provided by libaudit to add an audit rule for the freshly created process. If I hear of interest in this capability, I'll prepare a patch against the most recent audit source release, and share.

Last summer, someone else in my company made a simple version of a fork following tracer based on the audit library; however it does not compile on my FC5 machine. The reason is he was including the kernel's headers for ptrace, which defines PTRACE_O_TRACEFORK and other symbols not part of the standard ptrace header. I don't like the idea of requiring kernel headers to compile the extended version autrace.

I studied the strace source code, in particular, strace.c and process.c. I ran the preprocessor on these files using the options created on an FC5 machine, and my comments refer to the preprocessed version of these source files.

When asked to follow forks, in addition to calling ptrace with the PTRACE_TRACEME option in the child, strace uses procfs to find the task ID of the process PID in the directory /proc/PID/task. Apparently, it's the only file name in the directory that parses as a positive number. Using ptrace to attach to the task ID allows strace to follow forks. I'm guessing that I have to set a ptrace option so that the parent is signaled on every descendent system call, and inspect each system call to see if a new process was created. I bet following more of the strace logic will allow me to figure out these details.

I'm worried that blindly following strace's ways may lead me astray. Is there someplace I can find high-level documentation on when to use ptrace and when to use the procfs? Should I also study the GDB sources?

John

This is a problem that requires a kernel side implementation. We've talked about it in the past and I don't think we've got code yet. We did add audit by ppid which helps a little. But we need a generic way to say that we want to audit a specific program and all its descendants.

-Steve

Do you mean this is a problem that requires a kernel side implementation to do it well? Ptracing the descendants has the down side of changing the behavior of an application due to all the tracing signals, but until a kernel side implementation is available, the ptracing solution seems to me to be the only way to get the audit data we desire. Or do you mean the idea of using ptrace to follow forks is flawed for some reason, and will not work?

One quick question, I notice autrace.c invokes /sbin/auditctl to change audit rules, but shouldn't it being using audit_add_rule and friends instead? I'll implement this change if you want me to.

John

It was using auditctl just for expedience when the program was written. I can add the code to switch it over.

-Steve

I wrote code that uses libaudit to add rules. After performing "yum install audit-libs-devel", I was pleased to find a manual page for audit_add_rule, but the description of the return value says to refer to audit_send. However, I found no manual page for audit_send. Studying code made me realize the error condition is <= 0, not < 0, so it's important not to guess.

John

static int audit_sys; /* File descriptor for socket to audit system */

static int /* Create a rule for a pid */
init_rule(struct audit_rule *r, pid_t pid)
{ /* Returns zero on success */
char field[PID_FIELD_SIZE];

  if (snprintf(field, PID_FIELD_SIZE, "pid=%d", pid) >= PID_FIELD_SIZE) {
    fprintf(stderr, "Internal error in init_rule\n");
    exit(1);
  }
  memset(r, 0, sizeof(*r));
  if (audit_rule_syscallbyname(r, "all") < 0) {
    fprintf(stderr, "Illegal syscall name for audit rule for %d\n", pid);
    return 1;
  }
  if (audit_rule_fieldpair(r, field, AUDIT_FILTER_ENTRY)) {
    fprintf(stderr, "Cannot add field to audit rule for %d\n", pid);
    return 1;

}
return 0;
}

/* Equivalent to '/sbin/auditctl -a entry,always -F pid=%d -S all' */ static int
add_rule(pid_t pid) /* Returns zero on success */
{
int i;
for (i = 0; i < nchildren; i++)

    if (pid == children[i])	/* Rule already present */
      return 0;
  if (nchildren >= MAX_CHILDREN) {
    fprintf(stderr, "Too many children\n");
    return 1;
  }
  struct audit_rule r[1];
  if (init_rule(r, pid))
    return 1;
  if (audit_add_rule(audit_sys, r, AUDIT_FILTER_ENTRY, AUDIT_ALWAYS) <= 0) {
    fprintf(stderr, "Cannot add an audit rule for %d\n", pid);
    return 1;

}
children[nchildren++] = pid;
return 0;
}

Note that although the FC5 kernel-headers package might well not be
updated, it's very likely that FC5 already has a kernel that supports the PTRACE_O_ options[1]. So, personally, I'd just have something like:

#ifndef PTRACE_O_TRACEFORK

# define PTRACE_O_TRACEFORK 0x00000002 /* from kernel-2.6.x */
#endif

...and assuming you handle the error return from PTRACE_SETOPTIONS, it should mostly just work on all relevant systems.

[1] git-annotate says that flag was added on: 2005-04-16 15:20:36

James Antill - <[email protected]>
setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, ...);
setsockopt(fd, IPPROTO_TCP, TCP_DEFER_ACCEPT, ...);
setsockopt(fd, SOL_SOCKET,  SO_ATTACH_FILTER, ...);

Ah, and also do the same for PTRACE_EVENT_FORK and PTRACE_GETEVENTMSG. I'm guessing using these options makes it so that signals are only sent when a process is forked, not at every system call.

I grep'd the kernel documentation directory but didn't find much of interest on ptrace. I haven't looked at the kernel sources yet. Googling on PTRACE_EVENT_FORK and the like keeps leading me to GDB, so I plan to look at the sources.

http://www.cygwin.com/ml/gdb-patches/2003-08/msg00274.html

Thanks for your help.

John

The file gdb/linux_nat.c shows that GDB uses this trick, and many other useful ones. Time to cut-and-paste.

John

I realize others on this list aren't interested in a version of autrace that follows forks using ptrace, but I thought I'd share tricks I learned for following forks. Once the traced child is started, the parent goes into a loop waiting for signals. The tricky part seems to be that the SIGTRAP generated by the parent's immediate child has to be converted to a SIGSTOP before continuing the child.

static int /* Watch all children */
watch(pid_t pid) /* This process' child is pid */
{ /* Function returns an exit code */
if (add_rule(pid))

    return 1;
  for (;;) {
    int status;
    pid = wait_for_it(&status);
    if (pid < 0) {
      if (errno == ECHILD)	/* No children to wait for */
	return 0;		/* Declare success */
      perror("wait");
      return 1;
    }
    if (WIFSTOPPED(status)) {
      int signal = WSTOPSIG(status);
      if (signal == SIGTRAP) {	/* Tracing causes this signal */
	unsigned long msg;
	if (geteventmsg(pid, &msg) < 0) {
	  perror("ptrace(PTRACE_GETEVENTMSG, ...)");
	  return 1;
	}
	pid_t child = (pid_t)msg;
	if (child) {
	  /* The child of each traced fork is noted here */
	  if (add_rule(child))
	    return 1;
	}
	/* Only this process' child gets to this location, and just
	   one time */
	else if (setoptions(pid, PTRACE_O_TRACEFORK) < 0) {
	  perror("ptrace(PTRACE_SETOPTIONS, ...)");
	  return 1;
	}
	else
	  signal = SIGSTOP;
      }
      if (restart(pid, signal) < 0) {
	perror("ptrace(PTRACE_CONT, ...)");
	return 1;
      }
    }

}
}

Some of these tricks were learned from code written by someone who until recently worked at my company. There are no comments in his code describing the origin of these ideas.

John

If you trace a shell, you find out you must always convert a SIGTRAP to a SIGSTOP.

John

        signal = SIGSTOP;

/* Wrong

        */