bpf: two fixes for bpf iterators
Subsystems (5)
v2 cover bpf: two fixes for bpf iterators
Patch #1 fixed a rcu stall warning when traversing large number of tasks/files without overflowing seq_file buffer. The method is to control the number of visited objects in bpf_seq_read() so all bpf iterators will benefit.
Patch #2 calculated tid properly in a namespace in order to avoid visiting the name task multiple times.
Patch #3 handled read() error code EAGAIN properly in bpftool bpf_iter userspace code to collect pids. The change is needed due to Patch #1.
Yonghong Song (3):
bpf: fix a rcu_sched stall issue with bpf task/task_file iterator
bpf: avoid visit same object multiple times
bpftool: handle EAGAIN error code properly in pids collection
kernel/bpf/bpf_iter.c | 15 ++++++++++++++- kernel/bpf/task_iter.c | 3 ++- tools/bpf/bpftool/pids.c | 2 ++ 3 files changed, 18 insertions(+), 2 deletions(-)
Signature · 1 line
2.24.1
PATCH v2 1/3 bpf: fix a rcu_sched stall issue with bpf task/task_file iterator
In our production system, we observed rcu stalls when 'bpftool prog` is running.
rcu: INFO: rcu_sched self-detected stall on CPU rcu: \x097-....: (20999 ticks this GP) idle=302/1/0x4000000000000000 softirq=1508852/1508852 fqs=4913 \x09(t=21031 jiffies g=2534773 q=179750) NMI backtrace for cpu 7 CPU: 7 PID: 184195 Comm: bpftool Kdump: loaded Tainted: G W 5.8.0-00004-g68bfc7f8c1b4 #6 Hardware name: Quanta Twin Lakes MP/Twin Lakes Passive MP, BIOS F09_3A17 05/03/2019 Call Trace: <IRQ> dump_stack+0x57/0x70 nmi_cpu_backtrace.cold+0x14/0x53 ? lapic_can_unplug_cpu.cold+0x39/0x39 nmi_trigger_cpumask_backtrace+0xb7/0xc7 rcu_dump_cpu_stacks+0xa2/0xd0 rcu_sched_clock_irq.cold+0x1ff/0x3d9 ? tick_nohz_handler+0x100/0x100 update_process_times+0x5b/0x90 tick_sched_timer+0x5e/0xf0 __hrtimer_run_queues+0x12a/0x2a0 hrtimer_interrupt+0x10e/0x280 __sysvec_apic_timer_interrupt+0x51/0xe0 asm_call_on_stack+0xf/0x20 </IRQ> sysvec_apic_timer_interrupt+0x6f/0x80 asm_sysvec_apic_timer_interrupt+0x12/0x20 RIP: 0010:task_file_seq_get_next+0x71/0x220 Code: 00 00 8b 53 1c 49 8b 7d 00 89 d6 48 8b 47 20 44 8b 18 41 39 d3 76 75 48 8b 4f 20 8b 01 39 d0 76 61 41 89 d1 49 39 c1 48 19 c0 <48> 8b 49 08 21 d0 48 8d 04 c1 4c 8b 08 4d 85 c9 74 46 49 8b 41 38 RSP: 0018:ffffc90006223e10 EFLAGS: 00000297 RAX: ffffffffffffffff RBX: ffff888f0d172388 RCX: ffff888c8c07c1c0 RDX: 00000000000f017b RSI: 00000000000f017b RDI: ffff888c254702c0 RBP: ffffc90006223e68 R08: ffff888be2a1c140 R09: 00000000000f017b R10: 0000000000000002 R11: 0000000000100000 R12: ffff888f23c24118 R13: ffffc90006223e60 R14: ffffffff828509a0 R15: 00000000ffffffff task_file_seq_next+0x52/0xa0 bpf_seq_read+0xb9/0x320 vfs_read+0x9d/0x180 ksys_read+0x5f/0xe0 do_syscall_64+0x38/0x60 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f8815f4f76e Code: c0 e9 f6 fe ff ff 55 48 8d 3d 76 70 0a 00 48 89 e5 e8 36 06 02 00 66 0f 1f 44 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 52 c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 RSP: 002b:00007fff8f9df578 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 RAX: ffffffffffffffda RBX: 000000000170b9c0 RCX: 00007f8815f4f76e RDX: 0000000000001000 RSI: 00007fff8f9df5b0 RDI: 0000000000000007 RBP: 00007fff8f9e05f0 R08: 0000000000000049 R09: 0000000000000010 R10: 00007f881601fa40 R11: 0000000000000246 R12: 00007fff8f9e05a8 R13: 00007fff8f9e05a8 R14: 0000000001917f90 R15: 000000000000e22e
Note that bpftool prog actually calls a task_file bpf iterator program to establish an association between prog/map/link/btf anon files and processes.
In the case where the above rcu stall occured, we had a process having 1587 tasks and each task having roughly 81305 files. This implied 129 million bpf prog invocations. Unfortunwtely none of these files are prog/map/link/btf files so bpf iterator/prog needs to traverse all these files and not able to return to user space since there are no seq_file buffer overflow.
This patch fixed the issue in bpf_seq_read() to limit the number of visited objects. If the maximum number of visited objects is reached, no more objects will be visited in the current syscall. If there is nothing written in the seq_file buffer, -EAGAIN will return to the user so user can try again.
The maximum number of visited objects is set at 1 million. In our Intel Xeon D-2191 2.3GHZ 18-core server, bpf_seq_read() visiting 1 million files takes around 0.18 seconds.
We did not use cond_resched() since for some iterators, e.g., netlink iterator, where rcu read_lock critical section spans between consecutive seq_ops->next(), which makes impossible to do cond_resched() in the key while loop of function bpf_seq_read().
--- kernel/bpf/bpf_iter.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
kernel/bpf/bpf_iter.c +14 -1
@@ -67,6 +67,9 @@ static void bpf_iter_done_stop(struct seq_file *seq) iter_priv->done_stop = true; } +/* maximum visited objects before bailing out */ +#define MAX_ITER_OBJECTS 1000000 +/* bpf_seq_read, a customized and simpler version for bpf iterator. * no_llseek is assumed for this file. * The following are differences from seq_read():
@@ -79,7 +82,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, {struct seq_file *seq = file->private_data;size_t n, offs, copied =0 ; -int err =0 ; +int err =0 , num_objs =0 ;void *p;mutex_lock (&seq->lock);
@@ -135,6 +138,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,while (1 ) {loff_t pos = seq->index; + num_objs++; offs = seq->count; p = seq->op->next (seq, p, &seq->index);if (pos == seq->index) {
@@ -153,6 +157,15 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,if (seq->count >= size)break ; +if (num_objs >=MAX_ITER_OBJECTS ) { +if (offs ==0 ) { + err = -EAGAIN ; + seq->op->stop (seq, p); + goto done; + } +break ; + } + err = seq->op->show (seq, p);if (err >0 ) {bpf_iter_dec_seq_num (seq);
Signature · 1 line
2.24.1
Re: [PATCH bpf v2 1/3] bpf: fix a rcu_sched stall issue with bpf task/task_file iterator
Quoted 5 lines · Yonghong Song
We did not use
cond_resched()since for some iterators, e.g., netlink iterator, where rcuread_lockcritical section spans between consecutiveseq_ops->next(), which makes impossible to docond_resched()in the key while loop of functionbpf_seq_read().
but after this patch we can, right?
Quoted 27 lines
kernel/bpf/bpf_iter.c +5
-1+/* maximum visited objects before bailing out */ +#define MAX_ITER_OBJECTS 1000000 +/* bpf_seq_read, a customized and simpler version for bpf iterator. * no_llseek is assumed for this file. * The following are differences from seq_read(): @@ -79,7 +82,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, {struct seq_file *seq = file->private_data;size_t n, offs, copied =0 ; -int err =0 ; +int err =0 , num_objs =0 ;void *p;mutex_lock (&seq->lock);@@ -135,6 +138,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,while (1 ) {loff_t pos = seq->index; + num_objs++; offs = seq->count; p = seq->op->next (seq, p, &seq->index);if (pos == seq->index) {@@ -153,6 +157,15 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,if (seq->count >= size)break ;
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 168
Quoted 10 lines
kernel/bpf/bpf_iter.c +9
-0+if (num_objs >=MAX_ITER_OBJECTS ) { +if (offs ==0 ) { + err = -EAGAIN ; + seq->op->stop (seq, p); + goto done; + } +break ; + } +
should this block be after op->show() and error processing? Otherwise bpf_iter_inc_seq_num() will be incorrectly incremented?
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 171
Quoted 3 lines
kernel/bpf/bpf_iter.c +0
-0err = seq->op->show (seq, p);if (err >0 ) {bpf_iter_dec_seq_num (seq);
After op->stop() we can do cond_resched() in all cases, since rhashtable walk does rcu_unlock in stop() callback, right? I think copy_to_user() and mutex_unlock() don't do cond_resched() equivalent work.
Re: [PATCH bpf v2 1/3] bpf: fix a rcu_sched stall issue with bpf task/task_file iterator
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Yonghong Song
We did not use
cond_resched()since for some iterators, e.g., netlink iterator, where rcuread_lockcritical section spans between consecutiveseq_ops->next(), which makes impossible to docond_resched()in the key while loop of functionbpf_seq_read().but after this patch we can, right?
We can do cond_resched() after seq->op->stop(). See more below.
Quoted 29 lines
Quoted 29 lines
kernel/bpf/bpf_iter.c +5
-1+/* maximum visited objects before bailing out */ +#define MAX_ITER_OBJECTS 1000000 +/* bpf_seq_read, a customized and simpler version for bpf iterator. * no_llseek is assumed for this file. * The following are differences from seq_read(): @@ -79,7 +82,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, {struct seq_file *seq = file->private_data;size_t n, offs, copied =0 ; -int err =0 ; +int err =0 , num_objs =0 ;void *p;mutex_lock (&seq->lock);@@ -135,6 +138,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,while (1 ) {loff_t pos = seq->index; + num_objs++; offs = seq->count; p = seq->op->next (seq, p, &seq->index);if (pos == seq->index) {@@ -153,6 +157,15 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,if (seq->count >= size)break ;
Quoted 12 lines
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 168
Quoted 9 lines
kernel/bpf/bpf_iter.c +9
-0+if (num_objs >=MAX_ITER_OBJECTS ) { +if (offs ==0 ) { + err = -EAGAIN ; + seq->op->stop (seq, p); + goto done; + } +break ; + } +should this block be after
op->show()and error processing? Otherwisebpf_iter_inc_seq_num()will be incorrectly incremented?
The purpose of op->next() is to calculate the "next" object position, stored in the seq private data. So for next read() syscall, start() will try to fetch the data based on the info in seq private data.
This is true for conditions "if (seq->count >= size) break" in the above so next op->start() can try to locate the correct object. The same is for this -EAGAIN thing.
Quoted 7 lines
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 171
Quoted 4 lines
kernel/bpf/bpf_iter.c +0
-0err = seq->op->show (seq, p);if (err >0 ) {bpf_iter_dec_seq_num (seq);After
op->stop()we can docond_resched()in all cases, since rhashtable walk doesrcu_unlockinstop()callback, right?
Yes, we can. I am thinking since we return to user space, cond_resched() might not be needed since returning to user space will trigger some kind of scheduling. This patch fixed the rcu stall issue. But if my understanding is incorrect, I am happy to add cond_reched().
Quoted 3 lines
I think
copy_to_user()andmutex_unlock()don't docond_resched()equivalent work.
Re: [PATCH bpf v2 1/3] bpf: fix a rcu_sched stall issue with bpf task/task_file iterator
Quoted 65 lines · Yonghong Song
Quoted 8 lines · Alexei Starovoitov
Quoted 5 lines · Yonghong Song
We did not use
cond_resched()since for some iterators, e.g., netlink iterator, where rcuread_lockcritical section spans between consecutiveseq_ops->next(), which makes impossible to docond_resched()in the key while loop of functionbpf_seq_read().but after this patch we can, right?
We can do
cond_resched()afterseq->op->stop(). See more below.Quoted 29 lines
Quoted 29 lines
kernel/bpf/bpf_iter.c +9
-1+/* maximum visited objects before bailing out */ +#define MAX_ITER_OBJECTS 1000000 +/* bpf_seq_read, a customized and simpler version for bpf iterator. * no_llseek is assumed for this file. * The following are differences from seq_read(): @@ -79,7 +82,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, {struct seq_file *seq = file->private_data;size_t n, offs, copied =0 ; -int err =0 ; +int err =0 , num_objs =0 ;void *p;mutex_lock (&seq->lock);@@ -135,6 +138,7 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,while (1 ) {loff_t pos = seq->index; + num_objs++; offs = seq->count; p = seq->op->next (seq, p, &seq->index);if (pos == seq->index) {@@ -153,6 +157,15 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size,if (seq->count >= size)break ; +if (num_objs >=MAX_ITER_OBJECTS ) { +if (offs ==0 ) { + err = -EAGAIN ; + seq->op->stop (seq, p);Quoted 8 lines
kernel/bpf/bpf_iter.c static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · line 168
Quoted 5 lines
kernel/bpf/bpf_iter.c +5
-0+ goto done; + } +break ; + } +should this block be after
op->show()and error processing? Otherwisebpf_iter_inc_seq_num()will be incorrectly incremented?The purpose of
op->next()is to calculate the "next" object position, stored in the seq private data. So for nextread()syscall,start()will try to fetch the data based on the info in seq private data.This is true for conditions "if (
seq->count>= size) break" in the above so nextop->start()can try to locate the correct object. The same is for this-EAGAINthing.Quoted 5 lines
Quoted 4 lines
kernel/bpf/bpf_iter.c +0
-0err = seq->op->show (seq, p);if (err >0 ) {bpf_iter_dec_seq_num (seq);
Quoted 8 lines
Quoted 2 lines
After
op->stop()we can docond_resched()in all cases, since rhashtable walk doesrcu_unlockinstop()callback, right?Yes, we can. I am thinking since we return to user space,
cond_resched()might not be needed since returning to user space will trigger some kind of scheduling. This patch fixed the rcu stall issue. But if my understanding is incorrect, I am happy to addcond_reched().
ahh. you're correct on both counts. Applied all three patches to bpf tree. thanks!
PATCH v2 2/3 bpf: avoid visit same object multiple times
In reply to Yonghong SongCurrently when traversing all tasks, the next tid is always increased by one. This may result in visiting the same task multiple times in a pid namespace.
This patch fixed the issue by seting the next tid as pid_nr_ns(pid, ns) + 1, similar to funciton next_tgid().
--- kernel/bpf/task_iter.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/bpf/task_iter.c +2 -1
@@ -29,8 +29,9 @@ static struct task_struct *task_seq_get_next(struct pid_namespace *ns,rcu_read_lock (); retry: - pid =idr_get_next (&ns->idr, tid); + pid =find_ge_pid (*tid, ns);if (pid) { + *tid =pid_nr_ns (pid, ns); task =get_pid_task (pid,PIDTYPE_PID );if (!task) { ++*tid;
Signature · 1 line
2.24.1
PATCH v2 3/3 bpftool: handle EAGAIN error code properly in pids collection
In reply to Yonghong SongWhen the error code is EAGAIN, the kernel signals the user space should retry the read() operation for bpf iterators. Let us do it.
--- tools/bpf/bpftool/pids.c | 2 ++ 1 file changed, 2 insertions(+)
tools/bpf/bpftool/pids.c +2 -0
@@ -134,6 +134,8 @@ int build_obj_refs_table(struct obj_refs_table *table, enum bpf_obj_type type)while (true) { ret =read (fd, buf,sizeof (buf));if (ret <0 ) { +if (errno ==EAGAIN ) +continue ; err = -errno;p_err ("failed to read PID iterator output: %d" , err); goto out;
Signature · 1 line
2.24.1
Re: [PATCH bpf v2 3/3] bpftool: handle EAGAIN error code properly in pids collection
Quoted 7 lines · Yonghong Song
When the error code is EAGAIN, the kernel signals the user space should retry the
read()operation for bpf iterators. Let us do it.Signed-off-by Yonghong Song <[email protected]>---
LGTM.
Quoted 11 lines · Yonghong Song
Patch #1 fixed a rcu stall warning when traversing large number of tasks/files without overflowing
seq_filebuffer. The method is to control the number of visited objects inbpf_seq_read()so all bpf iterators will benefit.Patch #2 calculated tid properly in a namespace in order to avoid visiting the name task multiple times.
Patch #3 handled
read()error code EAGAIN properly in bpftoolbpf_iteruserspace code to collect pids. The change is needed due to Patch #1.
Sorry. Missed the changelog below.
Changelogs:
- v1 -> v2:
. do ratelimiting in bpf_seq_read() with a counter.
cond_resched() doesn't work as some iterators (e.g., tcp, netlink, etc.) has rcu read critical section across consecutive seq_ops->next() functions.
Quoted 11 lines
Yonghong Song (3):
bpf: fix arcu_schedstall issue with bpf task/task_file iterator
bpf: avoid visit same object multiple times
bpftool: handle EAGAIN error code properly in pids collectionkernel/bpf/bpf_iter.c | 15 ++++++++++++++- kernel/bpf/task_iter.c | 3 ++- tools/bpf/bpftool/pids.c | 2 ++ 3 files changed, 18 insertions(+), 2 deletions(-)