Experience on full stack profiling using eBPF.
Subsystems (8)
cover Experience on full stack profiling using eBPF.
Hi Alexei,
We are working on full stack system profiling tools and found your eBPF is an amazing start point for us. However, during our prototype designing and implementing we also found some limitations on it. I'd like to share our experience with you in this mailing list, and I hope you and others can provide comments and suggestions for us.
Our goal is to profile production systems by analyzing end-to-end trace of particular procedures, including full IO request through sys_write() system call to underlying disk driver, full network processes through sys_sendto() to NIC driver and even full database query through client initiating to server processing to client finalizing. To avoid heavy tracing overhead and overwhelming data, we randomly sample a few out of huge amounts of procedures, only trace events caused by them at different layers and leave other procedures work normally. As a result, we make as few traces as possible for our profiling with the minimal effect to system performance.
In this series of RFC we provide a very rough implementation which traces at different layers for IO processes in a QEMU system. Patch 1/2 deploys some new tracepoints at each layer of IO stack, patch 2/2 attaches eBPF programs on them. eBPF programs select events according to their trigger conditions, and propagate these conditions to eBPF programs at lower layers. We use hash tables to record trigger conditions.
Our prototype is able to sample one sys_write() call among all write by a dd command. The result is as follow:
dd-985 [000] .... 54878.066046: iov_iter_copy_from_user_atomic: Node1 iov=00000000011c0010 page=ffffea0001eebb40
...
kworker/ [000] .... 54886.746406: ext4_bio_write_page: Node2 page=ffffea0001eebb40 bio=ffff88007c249540 kworker/ [000] .... 54886.750054: bio_attempt_back_merge: Node3 bio=ffff88007c249540 rq=ffff88007c3d4158
...
kworker/ [000] d... 54886.828480: virtscsi_add_cmd: Node4 req=ffff88007c3d4158 index=2
...
tracex5-9 [000] .Ns. 54886.840415: bio_endio: Node5 bio=ffff88007c249540
The result trace log is less than 20 lines, which will be more than 10000 lines without sampling and conditions. We can easily get a graph like this from the above result trace:
S1/2/3 are samples.
S1S2 S3
| | |
(vfs)pagecache -----------------------------------------> timeline
S1S2 S3
| | |
(block)request -----------------------------------------> timeline
S1S2 S3
| | |bio_end -----------------------------------------> timeline
During our work we found that eBPF can be improved so we can make our life easier:
1. BPF prog cannot use div/mod operator, which is useful in sampling.
2. BPF is a filter, the output trace goes to trace pipe, sometimes we may need a structured trace output like perf.data, so we can record more in filter function. BPF Array table is a good place to record data, but the index is restricted to 4 byte size.
3. Though most of events are filtered out, the eBPF program attached to tracepoints still run for every execution. We should find a way to minimize performance overhead for those events filtered out.
4. Provide a more user-friendly interface for user to use.
Thanks for your work.
He Kuang (2):
tracing: add additional IO tracepoints
samples: bpf: IO profiling example
block/bio.c | 1 + block/blk-core.c | 4 + drivers/scsi/virtio_scsi.c | 3 + fs/ext4/page-io.c | 4 + include/trace/events/block.h | 81 +++++++++++++++++ include/trace/events/ext4.h | 21 +++++ include/trace/events/filemap.h | 25 ++++++ include/trace/events/scsi.h | 21 +++++ mm/filemap.c | 2 + samples/bpf/Makefile | 4 + samples/bpf/tracex5_kern.c | 195 +++++++++++++++++++++++++++++++++++++++++ samples/bpf/tracex5_user.c | 56 ++++++++++++ 12 files changed, 417 insertions(+) create mode 100644 samples/bpf/tracex5_kern.c create mode 100644 samples/bpf/tracex5_user.c
Signature · 1 line
2.2.0.33.gc18b867
RFC 1/2 tracing: add additional IO tracepoints
Add tracepoints according to IO data stream to trace data propagate in vfs, fs, block layers.
--- block/bio.c | 1 + block/blk-core.c | 4 +++ drivers/scsi/virtio_scsi.c | 3 ++ fs/ext4/page-io.c | 4 +++ include/trace/events/block.h | 81 ++++++++++++++++++++++++++++++++++++++++++ include/trace/events/ext4.h | 21 +++++++++++ include/trace/events/filemap.h | 25 +++++++++++++ include/trace/events/scsi.h | 21 +++++++++++ mm/filemap.c | 2 ++ 9 files changed, 162 insertions(+)
block/bio.c +1 -0
@@ -1827,6 +1827,7 @@ void bio_endio(struct bio *bio, int error) }else {if (bio->bi_end_io) bio->bi_end_io (bio, error); +trace_bio_endio (bio); bio =NULL ; } }
block/blk-core.c +4 -0
@@ -1442,6 +1442,7 @@ bool bio_attempt_back_merge(struct request_queue *q, struct request *req, req->ioprio =ioprio_best (req->ioprio,bio_prio (bio));blk_account_io_start (req, false); +trace_bio_attempt_back_merge (req, bio);return true; }
@@ -1466,6 +1467,7 @@ bool bio_attempt_front_merge(struct request_queue *q, struct request *req, req->ioprio =ioprio_best (req->ioprio,bio_prio (bio));blk_account_io_start (req, false); +trace_bio_attempt_front_merge (req, bio);return true; }
@@ -1612,6 +1614,8 @@ get_rq:* Returns with the queue unlocked. */ req =get_request (q, rw_flags, bio,GFP_NOIO ); +trace_blk_queue_bio (req, bio); +if (IS_ERR (req)) {bio_endio (bio,PTR_ERR (req));/* @q is dead */ goto out_unlock;
drivers/scsi/virtio_scsi.c +3 -0
@@ -30,6 +30,8 @@#include <scsi/scsi_tcq.h> #include <linux/seqlock.h> +#include <trace/events/scsi.h> +#define VIRTIO_SCSI_MEMPOOL_SZ 64#define VIRTIO_SCSI_EVENT_LEN 8#define VIRTIO_SCSI_VQ_BASE 2
@@ -464,6 +466,7 @@ static int virtscsi_add_cmd(struct virtqueue *vq, sgs[out_num + in_num++] = in->sgl; } +trace_virtscsi_add_cmd (vq, sc->request);return virtqueue_add_sgs (vq, sgs, out_num, in_num, cmd,GFP_ATOMIC ); }
fs/ext4/page-io.c +4 -0
@@ -31,6 +31,8 @@#include "xattr.h" #include "acl.h" +#include <trace/events/ext4.h> +static struct kmem_cache *io_end_cachep;int __init ext4_init_pageio (void )
@@ -478,6 +480,8 @@ int ext4_bio_write_page(struct ext4_io_submit *io,redirty_page_for_writepage (wbc, page);break ; } +trace_ext4_bio_write_page (io, page); + nr_submitted++;clear_buffer_dirty (bh); }while ((bh = bh->b_this_page) != head);
include/trace/events/block.h +81 -0
87 lines not on this page · raw message
include/trace/events/ext4.h +21 -0
@@ -2476,6 +2476,27 @@ TRACE_EVENT(ext4_es_shrink, __entry->scan_time, __entry->nr_skipped, __entry->retried) ); +TRACE_EVENT (ext4_bio_write_page, + TP_PROTO(struct ext4_io_submit *io, struct page *page), + +TP_ARGS (io, page), + +TP_STRUCT__entry ( + __field( struct page *, page ) + __field( struct bio*, bio ) + ), + +TP_fast_assign ( + __entry->page = page; + __entry->bio = io->io_bio; + ), + +TP_printk ("Node2 page=%p bio=%p" , + __entry->page, + __entry->bio) + ); + +#endif /* _TRACE_EXT4_H */ /* This part must be outside protection */
include/trace/events/filemap.h +25 -0
@@ -11,6 +11,8 @@#include <linux/device.h> #include <linux/kdev_t.h> +#include <linux/uio.h> + DECLARE_EVENT_CLASS(mm_filemap_op_page_cache, TP_PROTO(struct page *page),
@@ -52,6 +54,29 @@ DEFINE_EVENT(mm_filemap_op_page_cache, mm_filemap_add_to_page_cache,TP_ARGS (page) ); + + +TRACE_EVENT (iov_iter_copy_from_user_atomic, + + TP_PROTO(struct page *page, struct iov_iter *i), + +TP_ARGS (page, i), + +TP_STRUCT__entry ( + __field(struct page *, page) + __field(void*, iov_base) + ), + +TP_fast_assign ( + __entry->page = page; + __entry->iov_base = i->iov->iov_base; + ), + +TP_printk ("Node1 iov=%p page=%p" , + __entry->iov_base, + __entry->page) + ); +#endif /* _TRACE_FILEMAP_H */ /* This part must be outside protection */
include/trace/events/scsi.h +21 -0
@@ -8,6 +8,7 @@#include <scsi/scsi_host.h> #include <linux/tracepoint.h> #include <linux/trace_seq.h> +#include <linux/virtio.h> #define scsi_opcode_name (opcode) { opcode, #opcode }#define show_opcode_name (val) \
@@ -359,6 +360,26 @@ TRACE_EVENT(scsi_eh_wakeup,TP_printk ("host_no=%u" , __entry->host_no) ); +TRACE_EVENT (virtscsi_add_cmd, + + TP_PROTO(struct virtqueue *vq, struct request *req), + +TP_ARGS (vq, req), + +TP_STRUCT__entry ( + __field( unsigned int, vq_index ) + __field( void *, req_base ) + ), + +TP_fast_assign ( + __entry->vq_index = vq->index; + __entry->req_base = req; + ), + +TP_printk ("Node4 req=%p index=%u" , __entry->req_base, + __entry->vq_index) + ); +#endif /* _TRACE_SCSI_H */ /* This part must be outside protection */
mm/filemap.c +2 -0
@@ -2497,6 +2497,8 @@ again:if (mapping_writably_mapped (mapping))flush_dcache_page (page); +trace_iov_iter_copy_from_user_atomic (page, i); + copied =iov_iter_copy_from_user_atomic (page, i, offset, bytes);flush_dcache_page (page);
Signature · 1 line
2.2.0.33.gc18b867
RFC 2/2 samples: bpf: IO profiling example
In reply to He KuangFor the first node, we sample and record its characteristic (eg. address), and add the result to a global hash table. For other nodes, they'll be traced when their context matches the characteristics in previous tables.
--- samples/bpf/Makefile | 4 + samples/bpf/tracex5_kern.c | 195 +++++++++++++++++++++++++++++++++++++++++++++ samples/bpf/tracex5_user.c | 56 +++++++++++++ 3 files changed, 255 insertions(+) create mode 100644 samples/bpf/tracex5_kern.c create mode 100644 samples/bpf/tracex5_user.c
samples/bpf/Makefile +4 -0
@@ -11,6 +11,7 @@ hostprogs-y += tracex1 hostprogs-y += tracex2 hostprogs-y += tracex3 hostprogs-y += tracex4 +hostprogs-y += tracex5 dropmon-objs := dropmon.o libbpf.o test_verifier-objs := test_verifier.o libbpf.o
@@ -22,6 +23,7 @@ tracex1-objs := bpf_load.o libbpf.o tracex1_user.o tracex2-objs := bpf_load.o libbpf.o tracex2_user.o tracex3-objs := bpf_load.o libbpf.o tracex3_user.o tracex4-objs := bpf_load.o libbpf.o tracex4_user.o +tracex5-objs := bpf_load.o libbpf.o tracex5_user.o # Tell kbuild to always build the programs always := $(hostprogs-y)
@@ -31,6 +33,7 @@ always += tracex1_kern.o always += tracex2_kern.o always += tracex3_kern.o always += tracex4_kern.o +always += tracex5_kern.o HOSTCFLAGS += -I$(objtree)/usr/include HOSTCFLAGS += -I$(objtree)/include/uapi/
@@ -43,6 +46,7 @@ HOSTLOADLIBES_tracex1 += -lelf HOSTLOADLIBES_tracex2 += -lelf HOSTLOADLIBES_tracex3 += -lelf HOSTLOADLIBES_tracex4 += -lelf +HOSTLOADLIBES_tracex5 += -lelf # point this to your LLVM backend with bpf support #LLC=$(srctree)/tools/bpf/llvm/bld/Debug+Asserts/bin/llc
samples/bpf/tracex5_kern.c new file +195 -0
195 lines not on this page · raw message
samples/bpf/tracex5_user.c new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.2.0.33.gc18b867