Return-path: Received: from mga04.intel.com ([192.55.52.120]) by Galois.linutronix.de with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1fEMoD-00061g-AT for speck@linutronix.de; Fri, 04 May 2018 00:34:41 +0200 From: Dave Hansen Subject: [MODERATED] [PATCH 2/5] SSB extra 3 Date: Thu, 3 May 2018 15:29:45 -0700 Message-Id: =?utf-8?q?=3C00e5d109f9753def844137c613a841ddca17de74=2E152538?= =?utf-8?q?3411=2Egit=2Edave=2Ehansen=40intel=2Ecom=3E?= In-Reply-To: References: In-Reply-To: References: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit MIME-Version: 1.0 To: speck@linutronix.de Cc: Dave Hansen List-ID: From: Dave Hansen Now that we have hooks called when we enter/exit the BFP code, tracks when we enter/leave. We "leave" lazily. The first time we leave, we schedule some work to do the actual "leave" at some point in the future. This way, we do not thrash by enabling and disabling mitigations frequently. This means that the per-BPF-program overhead is hopefully just the cost of incrementing and decrementing a per-cpu variable. The per-cpu counter 'bpf_prog_active' looks superficially like a great mechanism to use. However, it does not track active BPF programs. It appears to just be active when eprobe BPF handlers are running. Signed-off-by: Dave Hansen Cc: Andi Kleen Cc: Tim Chen --- include/linux/filter.h | 11 ++++++++++ net/core/filter.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/include/linux/filter.h b/include/linux/filter.h index e92854d..83c1298 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -490,12 +490,23 @@ struct sk_filter { struct bpf_prog *prog; }; +DECLARE_PER_CPU(unsigned int, bpf_prog_ran); + static inline void bpf_enter_prog(const struct bpf_prog *fp) { + int *count = &get_cpu_var(bpf_prog_ran); + (*count)++; } +extern void bpf_leave_prog_deferred(const struct bpf_prog *fp); static inline void bpf_leave_prog(const struct bpf_prog *fp) { + int *count = this_cpu_ptr(&bpf_prog_ran); + if (*count == 1) + bpf_leave_prog_deferred(fp); + else + (*count)--; + put_cpu_var(bpf_prog_ran); } #define BPF_PROG_RUN(filter, ctx) ({ \ diff --git a/net/core/filter.c b/net/core/filter.c index d31aff9..ffca000 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -5649,3 +5649,61 @@ int sk_get_filter(struct sock *sk, struct sock_filter __user *ubuf, release_sock(sk); return ret; } + +/* + * 0 when no BPF code has executed on the CPU. + * Incremented when running BPF code. + * When ==1, work will be scheduled. + * When >1, work will not be scheduled because work is already + * scheduled. + * When work is performed, count will be decremented from 1->0. + */ +DEFINE_PER_CPU(unsigned int, bpf_prog_ran); +EXPORT_SYMBOL_GPL(bpf_prog_ran); +static void bpf_done_on_this_cpu(struct work_struct *work) +{ + if (!this_cpu_dec_return(bpf_prog_ran)) + return; + + /* + * This is unexpected. The elevated refcount indicates + * being in the *middle* of a BPF program, which should + * be impossible. They are executed inside + * rcu_read_lock() where we can not sleep and where + * preemption is disabled. + */ + WARN_ON_ONCE(1); +} + +DEFINE_PER_CPU(struct delayed_work, bpf_prog_delayed_work); +static __init int bpf_init_delayed_work(void) +{ + int i; + + for_each_possible_cpu(i) { + struct delayed_work *w = &per_cpu(bpf_prog_delayed_work, i); + + INIT_DELAYED_WORK(w, bpf_done_on_this_cpu); + } + return 0; +} +subsys_initcall(bpf_init_delayed_work); + +/* + * Must be called with preempt disabled + * + * The schedule_delayed_work_on() is relatively expensive. So, + * this way, someone doing a bunch of repeated BPF calls will + * only pay the cost of scheduling work on the *first* BPF call. + * The subsequent calls only pay the cost of incrementing a + * per-cpu variable, which is cheap. + */ +void bpf_leave_prog_deferred(const struct bpf_prog *fp) +{ + int cpu = smp_processor_id(); + struct delayed_work *w = &per_cpu(bpf_prog_delayed_work, cpu); + unsigned long delay_jiffies = msecs_to_jiffies(10); + + schedule_delayed_work_on(cpu, w, delay_jiffies); +} +EXPORT_SYMBOL_GPL(bpf_leave_prog_deferred); -- 2.9.5