nvme-trace: Add support for fabrics command
Subsystems (2)
v6 cover nvme-trace: Add support for fabrics command
Hi,
This is 6th series to introduce target-side tracing. This series continas codes duplicated with host-side tracing, but it can make us avoid some disadvantages Christoph pointed out.
The target-side takes struct nvmet_req *req, so that those two entries have a little bit different from host-side tracing entries.
The following logs are examples of this patch series when trying to connect the loop targeted device:
nvme-2401 [006] .... 1474.570779: nvmet_req_init: nvme0: qid=0, cmdid=0, nsid=1, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_connect recfmt=0, qid=0, sqsize=31, cattr=0, kato=15000)
kworker/6:2-1067 [006] .... 1474.570933: nvmet_req_complete: nvme1: qid=0, cmdid=0, res=0x1, status=0x0
nvme-2401 [004] .... 1474.571216: nvmet_req_init: nvme1: qid=0, cmdid=6, nsid=4, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_property_get attrib=1, ofst=0x0)
kworker/4:2-1235 [004] .... 1474.571336: nvmet_req_complete: nvme1: qid=0, cmdid=6, res=0x200f0003ff, status=0x0
nvme-2401 [006] .... 1474.571440: nvmet_req_init: nvme1: qid=0, cmdid=22, nsid=0, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_property_set attrib=0, ofst=0x14, value=0x460001)
kworker/6:2-1067 [006] .... 1474.571494: nvmet_req_complete: nvme1: qid=0, cmdid=22, res=0x0, status=0x0
nvme-2401 [004] .... 1474.571668: nvmet_req_init: nvme1: qid=0, cmdid=7, nsid=4, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_property_get attrib=0, ofst=0x1c)
kworker/4:2-1235 [004] .... 1474.571767: nvmet_req_complete: nvme1: qid=0, cmdid=7, res=0x1, status=0x0
nvme-2401 [006] .... 1474.572078: nvmet_req_init: nvme1: qid=0, cmdid=23, nsid=4, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_property_get attrib=0, ofst=0x8)
kworker/6:2-1067 [006] .... 1474.572098: nvmet_req_complete: nvme1: qid=0, cmdid=23, res=0x10300, status=0x0
nvme-2401 [004] .... 1474.572268: nvmet_req_init: nvme1: qid=0, cmdid=8, nsid=4, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_property_get attrib=1, ofst=0x0)
kworker/4:2-1235 [004] .... 1474.572287: nvmet_req_complete: nvme1: qid=0, cmdid=8, res=0x200f0003ff, status=0x0
nvme-2401 [006] .... 1474.572436: nvmet_req_init: nvme1: qid=0, cmdid=24, nsid=0, flags=0x40, meta=0x0, cmd=(nvme_admin_identify cns=1, ctrlid=0)
kworker/6:2-1067 [006] .... 1474.572462: nvmet_req_complete: nvme1: qid=0, cmdid=24, res=0x0, status=0x0
nvme-2401 [004] .... 1474.572643: nvmet_req_init: nvme1: qid=0, cmdid=9, nsid=4294967295, flags=0x40, meta=0x0, cmd=(nvme_admin_get_log_page cdw10=05 00 ff 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00)
kworker/4:2-1235 [004] .... 1474.572664: nvmet_req_complete: nvme1: qid=0, cmdid=9, res=0x0, status=0x0
nvme-2401 [006] .... 1474.573001: nvmet_req_init: nvme1: qid=0, cmdid=25, nsid=4294967295, flags=0x40, meta=0x0, cmd=(nvme_admin_get_log_page cdw10=0c 01 03 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00)
kworker/6:2-1067 [006] .... 1474.573028: nvmet_req_complete: nvme1: qid=0, cmdid=25, res=0x0, status=0x0
nvme-2401 [004] .... 1474.573158: nvmet_req_init: nvme1: qid=0, cmdid=6, nsid=0, flags=0x40, meta=0x0, cmd=(nvme_admin_set_features cdw10=07 00 00 00 07 00 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00)
kworker/4:2-1235 [004] .... 1474.573179: nvmet_req_complete: nvme1: qid=0, cmdid=6, res=0x20007f007f, status=0x0
kworker/0:1H-1057 [000] .... 1474.574905: nvmet_req_init: nvme0: qid=0, cmdid=0, nsid=1, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_connect recfmt=0, qid=1, sqsize=127, cattr=0, kato=0)
kworker/0:2-2118 [000] .... 1474.575025: nvmet_req_complete: nvme1: qid=1, cmdid=0, res=0x0, status=0x0
kworker/1:1H-1867 [001] .... 1474.575236: nvmet_req_init: nvme0: qid=0, cmdid=0, nsid=1, flags=0x40, meta=0x0, cmd=(nvme_fabrics_type_connect recfmt=0, qid=2, sqsize=127, cattr=0, kato=0)
kworker/1:1-63 [001] .... 1474.575358: nvmet_req_complete: nvme1: qid=2, cmdid=0, res=0x0, status=0x0Please review.
Thanks,
Changes to V5:
- Provide trace code to the target-side instead of a common code shared between host and target to avoid disadvantages something bad for the performance like cache miss. It has been suggested by Christoph.
- Removed the third patch out of this series because that has nothing to do with this series.
- Merged the last two commits into a single commit for the review.
Changes to V4:
- Add Reviewed-by: tag from Sagi. (Thanks to Sagi)
- Consider endianness for cqe->status when assigning the value in trace
- Add more descriptions about the variable arguments in events.
Changes to V3:
- Remove additional argument from the caller level.
Changes to V2:
- Provide a common code for both host and target. (Chaitanya)
- Add support for tracing requests in target-side (Chaitanya)
- Make it simple in trace.h without branch out from nvme core module (Christoph)
Changes to V1:
- fabrics commands should also be decoded, not just showing that it's a fabrics command. (Christoph)
- do not make it within nvme admin commands (Chaitanya)
Minwoo Im (3):
nvmet: introduce nvmet_req_to_ctrl to get ctrl instance
nvme: trace: do not EXPORT_SYMBOL for a trace function
nvmet: introduce target-side trace
drivers/nvme/host/trace.c | 1 - drivers/nvme/target/Makefile | 3 + drivers/nvme/target/core.c | 8 ++ drivers/nvme/target/nvmet.h | 9 ++ drivers/nvme/target/trace.c | 179 ++++++++++++++++++++++++++++++++++ drivers/nvme/target/trace.h | 184 +++++++++++++++++++++++++++++++++++ 6 files changed, 383 insertions(+), 1 deletion(-) create mode 100644 drivers/nvme/target/trace.c create mode 100644 drivers/nvme/target/trace.h
Signature · 1 line
2.21.0
RFC PATCH v6 1/3 nvmet: introduce nvmet_req_to_ctrl to get ctrl instance
nvme host driver can get the controller instance from nvme_req(req)->ctrl. In case of target driver, it needs to get controller instance from the struct nvmet_req with sq where the request has been submitted.
--- drivers/nvme/target/nvmet.h | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/nvme/target/nvmet.h +9 -0
@@ -318,6 +318,15 @@ struct nvmet_req {u64 error_slba; }; +static inline struct nvmet_ctrl *nvmet_req_to_ctrl (struct nvmet_req *req) +{ +struct nvmet_sq *sq = req->sq; + +if (sq) +return sq->ctrl; +return NULL ; +} +extern struct workqueue_struct *buffered_io_wq;static inline void nvmet_set_result (struct nvmet_req *req,u32 result)
Signature · 1 line
2.21.0
Christoph Hellwig
maintainerRFC PATCH v6 1/3 nvmet: introduce nvmet_req_to_ctrl to get ctrl instance
Quoted 4 lines · Minwoo Im
nvme host driver can get the controller instance from nvme_req(req)->ctrl. In case of target driver, it needs to get controller instance from the struct
nvmet_reqwith sq where the request has been submitted.
Note that normally we can just dereference ->sq just fine, as it only isn't set in the low-level code before accepting the initial connect.
Maybe keep this as a helper in trace.h?
RFC PATCH v6 1/3 nvmet: introduce nvmet_req_to_ctrl to get ctrl instance
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Minwoo Im
nvme host driver can get the controller instance from nvme_req(req)->ctrl. In case of target driver, it needs to get controller instance from the struct
nvmet_reqwith sq where the request has been submitted.Note that normally we can just dereference ->sq just fine, as it only isn't set in the low-level code before accepting the initial connect.
That's good point. I just wanted to have safe-code here, but that would be fine.
Quoted 1 line
Maybe keep this as a helper in
trace.h?
That would be better to put tihs function to nvmet.h once it's needed by nvmet code later time.
Thanks for your review on this, I'll prepare the V7 series.
RFC PATCH v6 2/3 nvme: trace: do not EXPORT_SYMBOL for a trace function
In reply to Minwoo Imnvme_trace_disk_name() is now already being invoked with the function prototype in trace.h. We don't need to export this symbol at all.
The following patches are going to provide target-side trace feature with the exactly same function with this so that this patch removes the EXPORT_SYMBOL() for this function.
--- drivers/nvme/host/trace.c | 1 - 1 file changed, 1 deletion(-)
drivers/nvme/host/trace.c +0 -1
@@ -145,6 +145,5 @@ const char *nvme_trace_disk_name(struct trace_seq *p, char *name)return ret; } -EXPORT_SYMBOL_GPL (nvme_trace_disk_name);EXPORT_TRACEPOINT_SYMBOL_GPL (nvme_sq);
Signature · 1 line
2.21.0
Christoph Hellwig
maintainerRFC PATCH v6 2/3 nvme: trace: do not EXPORT_SYMBOL for a trace function
Looks fine,
RFC PATCH v6 2/3 nvme: trace: do not EXPORT_SYMBOL for a trace function
In reply to Minwoo ImLooks good to me.
Quoted 27 lines · Minwoo Im
nvme_trace_disk_name()is now already being invoked with the function prototype intrace.h. We don't need to export this symbol at all.The following patches are going to provide target-side trace feature with the exactly same function with this so that this patch removes the
EXPORT_SYMBOL()for this function.Cc Keith Busch <kbusch at kernel.org>Cc Christoph Hellwig <hch at lst.de>Cc Sagi Grimberg <sagi at grimberg.me>Signed-off-by Minwoo Im <minwoo.im.dev at gmail.com>--- drivers/nvme/host/trace.c | 1 - 1 file changed, 1 deletion(-)drivers/nvme/host/trace.c +0
-1@@ -145,6 +145,5 @@ const char *nvme_trace_disk_name(struct trace_seq *p, char *name)return ret; } -EXPORT_SYMBOL_GPL (nvme_trace_disk_name);EXPORT_TRACEPOINT_SYMBOL_GPL (nvme_sq);
RFC PATCH v6 3/3 nvmet: introduce target-side trace
In reply to Minwoo ImThis patch introduces target-side request tracing. As Christoph suggested, the trace would not be in a core or module to avoid disadvantages like cache miss:
http://lists.infradead.org/pipermail/linux-nvme/2019-June/024721.html
The target-side trace code is totally based on the Johannes's trace code from the host side. It has lots of codes duplicated, but it would be better than having advantages mentioned above.
It also traces not only fabrics commands, but also nvme normal commands. Once the codes to be shared gets bigger, then we can make it common as suggsted.
This also removed the create_sq and create_cq trace parsing functions because it will be done by the connect fabrics command.
Example:
echo 1 > /sys/kernel/debug/tracing/event/nvmet/nvmet_req_init/enable echo 1 > /sys/kernel/debug/tracing/event/nvmet/nvmet_req_complete/enable cat /sys/kernel/debug/tracing/trace
--- drivers/nvme/target/Makefile | 3 + drivers/nvme/target/core.c | 8 ++ drivers/nvme/target/trace.c | 179 ++++++++++++++++++++++++++++++++++ drivers/nvme/target/trace.h | 184 +++++++++++++++++++++++++++++++++++ 4 files changed, 374 insertions(+) create mode 100644 drivers/nvme/target/trace.c create mode 100644 drivers/nvme/target/trace.h
drivers/nvme/target/Makefile +3 -0
@@ -1,5 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 +ccflags-y += -I$(src) + obj-$(CONFIG_NVME_TARGET) += nvmet.o obj-$(CONFIG_NVME_TARGET_LOOP) += nvme-loop.o obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o
@@ -14,3 +16,4 @@ nvmet-rdma-y += rdma.o nvmet-fc-y += fc.o nvme-fcloop-y += fcloop.o nvmet-tcp-y += tcp.o +nvmet-$(CONFIG_TRACING) += trace.o
drivers/nvme/target/core.c +8 -0
@@ -10,6 +10,9 @@#include <linux/pci-p2pdma.h> #include <linux/scatterlist.h> +#define CREATE_TRACE_POINTS +#include "trace.h" +#include "nvmet.h" struct workqueue_struct *buffered_io_wq;
@@ -689,6 +692,9 @@ static void __nvmet_req_complete(struct nvmet_req *req, u16 status)if (unlikely (status))nvmet_set_error (req, status); + +trace_nvmet_req_complete (req); +if (req->ns)nvmet_put_namespace (req->ns); req->ops->queue_response (req);
@@ -848,6 +854,8 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq, req->error_loc =NVMET_NO_ERROR_LOC ; req->error_slba =0 ; +trace_nvmet_req_init (req, req->cmd); +/* no support for fused commands yet */ if (unlikely (flags & (NVME_CMD_FUSE_FIRST |NVME_CMD_FUSE_SECOND ))) { req->error_loc = offsetof(struct nvme_common_command , flags);
drivers/nvme/target/trace.c new file +179 -0
179 lines not on this page · raw message
drivers/nvme/target/trace.h new file +184 -0
184 lines not on this page · raw message
Signature · 1 line
2.21.0
Christoph Hellwig
maintainerRFC PATCH v6 3/3 nvmet: introduce target-side trace
drivers/nvme/target/trace.h line 22
Quoted 2 lines
drivers/nvme/target/trace.h new file +2
-0+#define nvme_admin_opcode_name (opcode) { opcode, #opcode } +#define show_admin_opcode_name (val) \
All these bits can probably just move to nvme.h, next to the actual definitions.
Review
Quoted 2 lines
(quoted diff) +2
-0+ __assign_disk_name(__entry->disk, req->ns ? + req->ns->bdev->bd_disk : NULL);
Tis looks it will crash badly when using the file backend. I think we need to put a neutral identifier in, e.g. the uuid or something.
RFC PATCH v6 3/3 nvmet: introduce target-side trace
Quoted 5 lines · Christoph Hellwig
drivers/nvme/target/trace.h line 22
Quoted 2 lines
drivers/nvme/target/trace.h new file +2
-0+#define nvme_admin_opcode_name (opcode) { opcode, #opcode } +#define show_admin_opcode_name (val) \All these bits can probably just move to
nvme.h, next to the actual definitions.
If it's located in <linux/nvme.h> that would be great for the host-side trace code.
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ __assign_disk_name(__entry->disk, req->ns ? + req->ns->bdev->bd_disk : NULL);Tis looks it will crash badly when using the file backend. I think we need to put a neutral identifier in, e.g. the uuid or something.
Sorry I didn't think about the file-backed request at all. Thanks for pointing it out.
Will also prepare V7 series with this updates.
Thanks,