perf: Container-aware tracing support

The RFC patch set supports filtering container specific events when perf tool is executed inside a container. However, unlike the previous approach [1], this requires containers to be created with a new namespace "perf-namespace" (introduced in patch 1).

The basic idea is analogous to other namespaces: if event isolation is required when running perf tool inside a container, then the container should be created with perf-namespace.

Based on the feedback on the previous approach for container-aware tracing [1] in Plumbers 2015 Containers Microconference [2] we have come-up with a new prototype. The new prototype introduces a new namespace "perf-namespace" (namespace name is just a placeholder. name suggestions are welcome). The perf tool executed inside a container created with perf-namespace reports only those events that are triggered within the container. For example "perf record -a" inside a container reports container-wide events; the "-a" flag which stands for system-wide event collection takes the meaning of container-wide event collection when executed inside a container. We have tested perf-record, perf-stat and perf-top and we are currently working on getting perf-probe (kprobe/uprobe) working.

Our previous prototype [1] required the container to be created with PID namesapce and further required processes inside the container grouped into a single perf_event cgroup. However, during the discussion in Plumbers it was mentioned that the requirement of PID namespace is insufficient for containers that need access to the host PID namespace [3].

Work in progress:

- Get perf-probe working for containers. We are planning to enhance tracefs to have multiple probe lists instances and then mount different instance of tracefs (using add newinstance mount option) for every perf namespace.

Patches are based on 4.7.0-rc2.

Suggestions and feedback are welcome.

[1] https://lkml.org/lkml/2015/7/15/192
[2] http://linuxplumbersconf.org/2015/ocw/sessions/2667.html
[3] Notes for container-aware tracing:
https://etherpad.openstack.org/p/LPC2015_Containers

Regards,
Aravinda & Hari
---

Aravinda Prasad (1):
perf: Filter events based on perf-namespace

Hari Bathini (1):
perf/namespaces: Add a new namespace for isolated tracing

 fs/proc/namespaces.c           |    4 
 include/linux/nsproxy.h        |    2 
 include/linux/perf_event.h     |    8 +
 include/linux/perf_namespace.h |   58 +++++++
 include/linux/proc_ns.h        |    2 
 include/uapi/linux/sched.h     |    1 
 init/Kconfig                   |    7 +
 kernel/Makefile                |    1 
 kernel/events/core.c           |  347 ++++++++++++++++++++++++++++++++++++++++
 kernel/fork.c                  |    3 
 kernel/nsproxy.c               |   20 ++
 kernel/perf_namespace.c        |  132 +++++++++++++++
 12 files changed, 581 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/perf_namespace.h
 create mode 100644 kernel/perf_namespace.c

Aravinda Prasad

From: Hari Bathini <[email protected]>

This patch adds a new namespace to the kernel inline with the existing namespaces like pid, uts, etc. The aim of this namespace is to support isolated tracing within the context of this new namespace.

---
 fs/proc/namespaces.c           |    4 +
 include/linux/nsproxy.h        |    2 +
 include/linux/perf_namespace.h |   52 +++++++++++++++++
 include/linux/proc_ns.h        |    2 +
 include/uapi/linux/sched.h     |    1 
 init/Kconfig                   |    7 ++
 kernel/Makefile                |    1 
 kernel/fork.c                  |    3 +
 kernel/nsproxy.c               |   20 ++++++
 kernel/perf_namespace.c        |  124 ++++++++++++++++++++++++++++++++++++++++
 10 files changed, 213 insertions(+), 3 deletions(-)
 create mode 100644 include/linux/perf_namespace.h
 create mode 100644 kernel/perf_namespace.c

index 51b8b0a..f9812fc 100644
@@ -8,6 +8,7 @@
 #include <linux/ipc_namespace.h>
 #include <linux/pid_namespace.h>
 #include <linux/user_namespace.h>
+#include <linux/perf_namespace.h>
 #include "internal.h"
 
 

@@ -31,6 +32,9 @@ static const struct proc_ns_operations *ns_entries[] = {
 #ifdef CONFIG_CGROUPS
 	&cgroupns_operations,
 #endif
+#ifdef CONFIG_PERF_NS
+	&perfns_operations,
+#endif
 };
 
 static const char *proc_ns_get_link(struct dentry *dentry,

index ac0d65b..7e83e63 100644
@@ -9,6 +9,7 @@ struct uts_namespace;
 struct ipc_namespace;
 struct pid_namespace;
 struct cgroup_namespace;
+struct perf_namespace;
 struct fs_struct;
 
 /*

@@ -35,6 +36,7 @@ struct nsproxy {
 	struct pid_namespace *pid_ns_for_children;
 	struct net 	     *net_ns;
 	struct cgroup_namespace *cgroup_ns;
+	struct perf_namespace *perf_ns;
 };
 extern struct nsproxy init_nsproxy;
 

index de0e771..c2916a7 100644
@@ -27,6 +27,7 @@ extern const struct proc_ns_operations pidns_operations;
 extern const struct proc_ns_operations userns_operations;
 extern const struct proc_ns_operations mntns_operations;
 extern const struct proc_ns_operations cgroupns_operations;
+extern const struct proc_ns_operations perfns_operations;
 
 /*
  * We always define these enumerators

@@ -38,6 +39,7 @@ enum {
 	PROC_USER_INIT_INO	= 0xEFFFFFFDU,
 	PROC_PID_INIT_INO	= 0xEFFFFFFCU,
 	PROC_CGROUP_INIT_INO	= 0xEFFFFFFBU,
+	PROC_PERF_INIT_INO	= 0xEFFFFFFAU,
 };
 
 #ifdef CONFIG_PROC_FS

index 5f0fe01..6a13d40 100644
@@ -9,6 +9,7 @@
 #define CLONE_FS	0x00000200	/* set if fs info shared between processes */
 #define CLONE_FILES	0x00000400	/* set if open files shared between processes */
 #define CLONE_SIGHAND	0x00000800	/* set if signal handlers and blocked signals shared */
+#define CLONE_NEWPERF	0x00001000	/* New perf namespace */
 #define CLONE_PTRACE	0x00002000	/* set if we want to let tracing continue on the child too */
 #define CLONE_VFORK	0x00004000	/* set if the parent wants the child to wake it up on mm_release */
 #define CLONE_PARENT	0x00008000	/* set if we want to have the same parent as the cloner */

index f755a60..e0b23f2 100644
@@ -1221,6 +1221,13 @@ config NET_NS
 	  Allow user space to create what appear to be multiple instances
 	  of the network stack.
 
+config PERF_NS
+	bool "Perf Namespaces"
+	default y if PERF_EVENTS
+	help
+	  Support perf namespaces. A namespace to provide isolated tracing
+	  support in context of this namespace.
+
 endif # NAMESPACES
 
 config SCHED_AUTOGROUP

index e2ec54e..ee94119 100644
@@ -71,6 +71,7 @@ obj-$(CONFIG_CPUSETS) += cpuset.o
 obj-$(CONFIG_UTS_NS) += utsname.o
 obj-$(CONFIG_USER_NS) += user_namespace.o
 obj-$(CONFIG_PID_NS) += pid_namespace.o
+obj-$(CONFIG_PERF_NS) += perf_namespace.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_SMP) += stop_machine.o
 obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o

index 5c2c355..d53756c 100644
@@ -1926,7 +1926,8 @@ static int check_unshare_flags(unsigned long unshare_flags)
 	if (unshare_flags & ~(CLONE_THREAD|CLONE_FS|CLONE_NEWNS|CLONE_SIGHAND|
 				CLONE_VM|CLONE_FILES|CLONE_SYSVSEM|
 				CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWNET|
-				CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWCGROUP))
+				CLONE_NEWUSER|CLONE_NEWPID|
+				CLONE_NEWCGROUP|CLONE_NEWPERF))
 		return -EINVAL;
 	/*
 	 * Not implemented, but pretend it works if there is nothing

index 782102e..b9a9831 100644
@@ -22,6 +22,7 @@
 #include <linux/pid_namespace.h>
 #include <net/net_namespace.h>
 #include <linux/ipc_namespace.h>
+#include <linux/perf_namespace.h>
 #include <linux/proc_ns.h>
 #include <linux/file.h>
 #include <linux/syscalls.h>

@@ -43,6 +44,9 @@ struct nsproxy init_nsproxy = {
 #ifdef CONFIG_CGROUPS
 	.cgroup_ns		= &init_cgroup_ns,
 #endif
+#ifdef CONFIG_PERF_NS
+	.perf_ns		= &init_perf_ns,
+#endif
 };
 
 static inline struct nsproxy *create_nsproxy(void)

@@ -103,6 +107,12 @@ static struct nsproxy *create_new_namespaces(unsigned long flags,
 		goto out_cgroup;
 	}
 
+	new_nsp->perf_ns = copy_perf_ns(flags, user_ns, tsk->nsproxy->perf_ns);
+	if (IS_ERR(new_nsp->perf_ns)) {
+		err = PTR_ERR(new_nsp->perf_ns);
+		goto out_perf;
+	}
+
 	new_nsp->net_ns = copy_net_ns(flags, user_ns, tsk->nsproxy->net_ns);
 	if (IS_ERR(new_nsp->net_ns)) {
 		err = PTR_ERR(new_nsp->net_ns);

@@ -113,6 +123,9 @@ static struct nsproxy *create_new_namespaces(unsigned long flags,
 
 out_net:
 	put_cgroup_ns(new_nsp->cgroup_ns);
+out_perf:
+	if (new_nsp->net_ns)
+		put_net(new_nsp->net_ns);
 out_cgroup:
 	if (new_nsp->pid_ns_for_children)
 		put_pid_ns(new_nsp->pid_ns_for_children);

@@ -142,7 +155,7 @@ int copy_namespaces(unsigned long flags, struct task_struct *tsk)
 
 	if (likely(!(flags & (CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC |
 			      CLONE_NEWPID | CLONE_NEWNET |
-			      CLONE_NEWCGROUP)))) {
+			      CLONE_NEWCGROUP | CLONE_NEWPERF)))) {
 		get_nsproxy(old_ns);
 		return 0;
 	}

@@ -177,6 +190,8 @@ void free_nsproxy(struct nsproxy *ns)
 		put_uts_ns(ns->uts_ns);
 	if (ns->ipc_ns)
 		put_ipc_ns(ns->ipc_ns);
+	if (ns->perf_ns)
+		put_perf_ns(ns->perf_ns);
 	if (ns->pid_ns_for_children)
 		put_pid_ns(ns->pid_ns_for_children);
 	put_cgroup_ns(ns->cgroup_ns);

@@ -195,7 +210,8 @@ int unshare_nsproxy_namespaces(unsigned long unshare_flags,
 	int err = 0;
 
 	if (!(unshare_flags & (CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC |
-			       CLONE_NEWNET | CLONE_NEWPID | CLONE_NEWCGROUP)))
+			       CLONE_NEWNET | CLONE_NEWPID |
+			       CLONE_NEWCGROUP | CLONE_NEWPERF)))
 		return 0;
 
 	user_ns = new_cred ? new_cred->user_ns : current_user_ns();

Whenever perf tool is executed inside a container, this patch restricts the events to the perf-namespace in which the perf tool is executing.

This patch is based on the existing support available for tracing with cgroups.

TODO:
- Avoid code duplication.

---
 include/linux/perf_event.h     |    8 +
 include/linux/perf_namespace.h |    6 +
 kernel/events/core.c           |  347 ++++++++++++++++++++++++++++++++++++++++
 kernel/perf_namespace.c        |    8 +
 4 files changed, 368 insertions(+), 1 deletion(-)

index 1a827ce..8d797d9 100644
@@ -43,6 +43,7 @@ struct perf_guest_info_callbacks {
 #include <linux/hrtimer.h>
 #include <linux/fs.h>
 #include <linux/pid_namespace.h>
+#include <linux/perf_namespace.h>
 #include <linux/workqueue.h>
 #include <linux/ftrace.h>
 #include <linux/cpu.h>

@@ -656,6 +657,11 @@ struct perf_event {
 	struct rcu_head			rcu_head;
 
 	struct pid_namespace		*ns;
+#ifdef CONFIG_PERF_NS
+	struct perf_namespace           *perf_ns;
+	int     perfns_defer_enabled;
+#endif
+
 	u64				id;
 
 	u64				(*clock)(void);

@@ -725,6 +731,7 @@ struct perf_event_context {
 	u64				generation;
 	int				pin_count;
 	int				nr_cgroups;	 /* cgroup evts */
+	int				nr_perfns;
 	void				*task_ctx_data; /* pmu specific data */
 	struct rcu_head			rcu_head;
 };

@@ -751,6 +758,7 @@ struct perf_cpu_context {
 
 	struct pmu			*unique_pmu;
 	struct perf_cgroup		*cgrp;
+	struct perf_namespace   *perf_ns;
 };
 
 struct perf_output_handle {

index 9713724..2aad0e9 100644
@@ -8,8 +8,14 @@
 struct user_namespace;
 extern struct user_namespace init_user_ns;
 
+struct perf_ns_info {
+	u64             time;
+	u64             timestamp;
+};
+
 struct perf_namespace {
 	struct kref kref;
+	struct perf_ns_info __percpu *info;
 	struct user_namespace *user_ns;	/* Owning user namespace */
 	struct ns_common ns;
 };

index 5b76fd8..7991a93 100644
@@ -30,6 +30,13 @@ static struct perf_namespace *create_perf_ns(struct user_namespace *user_ns)
 
 	perf_ns->ns.ops = &perfns_operations;
 	perf_ns->user_ns = get_user_ns(user_ns);
+
+	perf_ns->info = alloc_percpu(struct perf_ns_info);
+	if (!perf_ns->info) {
+		kfree(perf_ns);
+		return ERR_PTR(-ENOMEM);
+	}
+
 	return perf_ns;
 }
 

@@ -115,6 +122,7 @@ struct perf_namespace init_perf_ns = {
 	.kref = {
 		.refcount = ATOMIC_INIT(2),
 	},
+	.info = NULL,
 	.user_ns = &init_user_ns,
 	.ns.inum = PROC_PERF_INIT_INO,
 #ifdef CONFIG_PERF_NS

Can't you, at perf_event_open() time, convert a per-cpu event into a per-cpu-per-cgroup event for these namespace thingies?

That seems to immediately and completely remove all that duplication.

Yes.. that seems possible. Let me check that out.

Regards,
Aravinda

Sorry for the delay in the response.

I was looking into a way how a per-cpu event can be converted to per-cpu-per-cgroup event at perf_event_open() but could not figure out how to do this conversion. The cgroup event expects the fd of the cgroup directory in cgroupfs and in this case we don't have any fd passed in. Not sure if I am missing anything.

Regards,
Aravinda

Would not the current namespace have a link to the correct cgroup? Wasn't that the entire point of the namespace thing?

Namespaces and cgroups are completely orthogonal to one another. Also in the v1 of cgroups it's possible to have a process member of more than 1 cgroup.

Then how do you specify what your new 'root' is? Surely you must first create a cgroup and then confine yourself to that?

Yeah, so? We only care about the perf controller obviously.

I completely misread the description of this, or I would have something earlier. For some reason I thought he was talking about the perf controller.

As I recall the tricky part of this was to have tracing that was safe and usable inside of a container. If you can align a per cgroup with your container that is probably sufficient for the select of processes.

At the same time there is a real desire to have identifiers like pids translated into the appropriate form for inside of the container. Without that translation they are meaningless inside a container. Further it is necessary to be certain the trancing that is used is is safe for unprivileged users.

I don't think I ever suggested or approved of the concept of a perf namespace and that sounds a bit dubious to me.

Eric

So perf uses the pid-namespace of the event-creator to report PID/TID numbers in.

So sys_perf_event_open() -> perf_event_alloc() does get_pid_ns(task_active_pid_ns(current)) to set event->ns and then we do:
task_{tgid,pid}_nr_ns(p, event->ns) to report the PID/TID resp., see perf_event_{pid,tid}().

Aligning a cgroup with the container is sufficient if containers are created with PID namespace. The first prototype was based on that.

However, not sure if it is fair to assume that the containers are created with PID namespace and the processes inside the container are grouped into a cgroup, as containers can be created without PID namespace. In fact it was mentioned in LPC container micro-conference that some containers are created without PID namespace as they need to access host PIDs.

With the recent introduction of cgroup namespace, I think we can even take out the requirement that container should be created with PID namespace to enable safe tracing inside of a container. We are currently evaluating that.

pid is already translated inside a container as mentioned below.

Yes true this was not suggested during the discussion.

As the kernel does not have the concept of a container, we thought introduction of perf-namespace could isolate events inside a container, analogous to other namespaces.

Regards,
Aravinda

No need to create cgroup and confine processes to that cgroup. The below check during event filtering will take care:

+ if (cpuctx->perf_ns != event->perf_ns)
+ return false;

event->perf_ns is set during perf_event_alloc():

+ if (current->nsproxy->perf_ns != &init_perf_ns) {
+ /*
+ * If we are called from our own perf namespace, set
+ * event->perf_ns
+ */
+ event->perf_ns = current->nsproxy->perf_ns;
+ [...]
+ }

Regards,
Aravinda