nvme: Support user mode processing of ZNS Zone Changed AEN
v2 cover nvme: Support user mode processing of ZNS Zone Changed AEN
From: Clay Mayers <[email protected]>
This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.
Patch 1 adds non-zero NVME_AEN_DATA with the value of CQE.DW1 to all NVME_AEN uevents.
Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.
This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.
Changes since v1
- Break up into two patches
- Only inlucde AEN_DATA if CQE.DW1 is non-zero
Clay Mayers (2):
nvme: Include AEN CQE.DW1 in NVME_AEN uevents
nvme: All AENs of type notify generate an NVME_AEN uevent
drivers/nvme/host/core.c | 21 +++++++++++++-------- drivers/nvme/host/nvme.h | 1 + 2 files changed, 14 insertions(+), 8 deletions(-)
Signature · 1 line
2.27.0
PATCH v2 1/2 nvme: Include AEN CQE.DW1 in NVME_AEN uevents
From: Clay Mayers <[email protected]>
There are AENs from alternate command sets that include extra data in their AEN's CQE.DW1. For example, the ZNS Zone-Descriptor-Changed AEN uses it to indicate the NSID of the event's log page.
NVME_AEN uevent now includes the value of CQE.DW1 as a new property. It is only included when non-zero to keep previously existing uevents unmodified.
--- drivers/nvme/host/core.c | 17 ++++++++++++----- drivers/nvme/host/nvme.h | 1 + 2 files changed, 13 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +12 -5
@@ -4615,17 +4615,22 @@ static void nvme_change_uevent(struct nvme_ctrl *ctrl, char *envdata)static void nvme_aen_uevent (struct nvme_ctrl *ctrl) { -char *envp[2 ] = {NULL ,NULL }; +char *envp[3 ] = {NULL ,NULL ,NULL };u32 aen_result = ctrl->aen_result; +u32 aen_data = ctrl->aen_data; ctrl->aen_result =0 ; + ctrl->aen_data =0 ;if (!aen_result)return ; envp[0 ] =kasprintf (GFP_KERNEL ,"NVME_AEN=%#08x" , aen_result); -if (!envp[0 ]) -return ; -kobject_uevent_env (&ctrl->device->kobj,KOBJ_CHANGE , envp); +if (aen_data) + envp[1 ] =kasprintf (GFP_KERNEL ,"NVME_AEN_DATA=%#08x" , aen_data); +if (envp[0 ] && (envp[1 ] || !aen_data)) +kobject_uevent_env (&ctrl->device->kobj,KOBJ_CHANGE , envp); + +kfree (envp[1 ]);kfree (envp[0 ]); }
@@ -4799,8 +4804,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,break ; } -if (requeue) +if (requeue) { + ctrl->aen_data =le64_to_cpu (res->u64) >>32 ;queue_work (nvme_wq, &ctrl->async_event_work); + } }EXPORT_SYMBOL_GPL (nvme_complete_async_event);
drivers/nvme/host/nvme.h +1 -0
@@ -299,6 +299,7 @@ struct nvme_ctrl {u16 cctemp;u32 oaes;u32 aen_result; +u32 aen_data;u32 ctratt;unsigned int shutdown_timeout;unsigned int kato;
Signature · 1 line
2.27.0
Christoph Hellwig
maintainerRe: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents
Quoted 10 lines · On Thu, Sep 29, 2022 at 03:39:54PM -0700, [email protected] wrote:
From: Clay Mayers <[email protected]>
There are AENs from alternate command sets that include extra data in their AEN's CQE.DW1. For example, the ZNS Zone-Descriptor-Changed AEN uses it to indicate the NSID of the event's log page.
NVME_AENuevent now includes the value of CQE.DW1 as a new property. It is only included when non-zero to keep previously existing uevents unmodified.
I don't think we can actually do this. If we ever want to handle Zone Excursions or any other event where the device can chane the Zone Descriptor we need to handle this AEN in the kernel and thus control the clearing of the even by reading the associated log page, so we can't just send it on to userspace. This is a bit of a sad state of affairs but unavoidable due to the NVMe AEN design.
RE: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents
Quoted 15 lines
From: Christoph Hellwig <[email protected]>
Sent: Monday, October 17, 2022 6:24 AMQuoted 10 lines · On Thu, Sep 29, 2022 at 03:39:54PM -0700, [email protected] wrote:
From: Clay Mayers <[email protected]>
There are AENs from alternate command sets that include extra data in their AEN's CQE.DW1. For example, the ZNS Zone-Descriptor-Changed AEN uses it to indicate the NSID of the event's log page.
NVME_AENuevent now includes the value of CQE.DW1 as a new property. It is only included when non-zero to keep previously existing uevents unmodified.
Quoted 6 lines
I don't think we can actually do this. If we ever want to handle Zone Excursions or any other event where the device can chane the Zone Descriptor we need to handle this AEN in the kernel and thus control the clearing of the even by reading the associated log page, so we can't just send it on to userspace. This is a bit of a sad state of affairs but unavoidable due to the NVMe AEN design.
What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?
The limitation is, the ZDC AEN is enabled/disabled at the controller level but it's essentially a namespace level event. Once on for one namespace, zone descriptor changes have to be handled as an AEN for all namespaces.
Christoph Hellwig
maintainerRe: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents
Quoted 5 lines · Clay Mayers
What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?
The problem is how the NVMe AENs work - the are raised and then the AEN command completion is delivered and they are cleared by reading the log page. But if we deliver them to userspace, we lose the pending state of the AEN for the kernel. Right now that is not an issue, but we lose all chance of ever making use of that information in the kernel. So maybe the answer is to handle the AEN in the kernel, read the changed zones log in the kernel, and then send an uevent for all changes zones to userspace.
Sagi Grimberg
maintainerRe: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents
Quoted 7 lines
Quoted 6 lines
What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?
Quoted 8 lines
The problem is how the NVMe AENs work - the are raised and then the AEN command completion is delivered and they are cleared by reading the log page. But if we deliver them to userspace, we lose the pending state of the AEN for the kernel. Right now that is not an issue, but we lose all chance of ever making use of that information in the kernel. So maybe the answer is to handle the AEN in the kernel, read the changed zones log in the kernel, and then send an uevent for all changes zones to userspace.
That is what I was thinking... I feel we had this discussion before on a different AEN (maybe it was something that wanted to get ANA info to userspace)...
RE: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents
Quoted 17 lines
From: Sagi Grimberg <[email protected]>
Sent: Tuesday, October 25, 2022 9:00 AMQuoted 6 lines
Quoted 5 lines
What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?
Quoted 8 lines
The problem is how the NVMe AENs work - the are raised and then the AEN command completion is delivered and they are cleared by reading the log page. But if we deliver them to userspace, we lose the pending state of the AEN for the kernel. Right now that is not an issue, but we lose all chance of ever making use of that information in the kernel. So maybe the answer is to handle the AEN in the kernel, read the changed zones log in the kernel, and then send an uevent for all changes zones to userspace.
Quoted 3 lines
That is what I was thinking... I feel we had this discussion before on a different AEN (maybe it was something that wanted to get ANA info to userspace)...
If you're thinking of this: https://lore.kernel.org/linux-nvme/BL0PR13MB4291371B4FD44B93D187B2319C419@BL0PR13MB4291.namprd13.prod.outlook.com/
I share the concern of both user and kernel reading the AEN referenced
log page. The solution should look more like what Christoph outlined
above with the kernel reading the log page and routing results.
However, the ZDC AEN is not at the controller level. It supplies the NSID so only the code writing to the namespace can be notified and only it will read the log page. Host-aware zoned devices can't be partitioned so responsibility wouldn't be split across user mode, kernel or multiple applications.
Yes, the kernel doesn't get to look at my namespace's ZDC entries and It has no reason to. In the future when the kernel is processing ZDC entries for a namespace, I shouldn’t expect a uevent or to see its ZDC entries just like two apps using different namespaces.
PATCH v2 2/2 nvme: All AENs of type notify generate an NVME_AEN uevent
In reply to [email protected]From: Clay Mayers <[email protected]>
This enables user mode processing of all unhandled AENs of type NOTIFY.
--- drivers/nvme/host/core.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/nvme/host/core.c +1 -3
@@ -4753,11 +4753,9 @@ static bool nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)queue_work (nvme_wq, &ctrl->ana_work);break ;#endif -case NVME_AER_NOTICE_DISC_CHANGED : +default : ctrl->aen_result = result;break ; -default : -dev_warn (ctrl->device,"async event result %08x\n" , result); }return requeue; }
Signature · 1 line
2.27.0
Quoted 5 lines · On 9/29/22 15:39, [email protected] wrote:
From: Clay Mayers <[email protected]>
This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.
Quoted 10 lines
Patch 1 adds non-zero
NVME_AEN_DATAwith the value of CQE.DW1 to allNVME_AENuevents.Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.
This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.
Please write a testcase in the blktest to this will get tested every release.
-ck
Quoted 22 lines
From: Chaitanya Kulkarni <[email protected]>
Sent: Monday, October 3, 2022 6:49 PM
Subject: Re: [PATCH V2 0/2] nvme: Support user mode processing of ZNS Zone Changed AENQuoted 5 lines · On 9/29/22 15:39, [email protected] wrote:
From: Clay Mayers <[email protected]>
This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.
Quoted 10 lines
Patch 1 adds non-zero
NVME_AEN_DATAwith the value of CQE.DW1 to allNVME_AENuevents.Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.
This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.
Quoted 4 lines
Please write a testcase in the blktest to this will get tested every release.
-ck
Ok - I'll give it a go to implement that but AENs by definition aren't a response to a request (mostly) - they just happen. The case that I know of that creates a zone excursion AEN takes many hours to manifest and that duration would be FLASH-type specific. I don't know how you could use a real device.
This patch was tested under QEMU with our custom emulator. The current QEMU NVMe/ZNS device doesn't emulate zone excursions so I'll have to start there unless someone knows of a better way.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.
Quoted 7 lines
Quoted 4 lines
Please write a testcase in the blktest to this will get tested every release.
-ck
Ok - I'll give it a go to implement that but AENs by definition aren't a response to a request (mostly) - they just happen. The case that I know
and this is exactly the main reason we need a testcase since it is sporadic and side effect of the controller's behavior that cannot be predicted (untill it hits the condition) by the requests. Lack of the testcase will leave this code untested unlike other code paths which are getting exercised predictably.
Quoted 4 lines
of that creates a zone excursion AEN takes many hours to manifest and that duration would be FLASH-type specific. I don't know how you could use a real device.
For this using real device is not an option since everyone should be able to test that with ease in blktests.
Quoted 4 lines
This patch was tested under QEMU with our custom emulator. The current QEMU NVMe/ZNS device doesn't emulate zone excursions so I'll have to start there unless someone knows of a better way.
Please add the support our QEMU NVMe developers are very helpful to get new features in.
-ck
Sagi Grimberg
maintainerQuoted 10 lines
This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.
Patch 1 adds non-zero
NVME_AEN_DATAwith the value of CQE.DW1 to allNVME_AENuevents.Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.
Quoted 6 lines
This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.
Changes since v1
- Break up into two patches
- Only inlucdeAEN_DATAif CQE.DW1 is non-zero
What about the other comments that were given on v1?
--
Review
Quoted 7 lines
#endif(quoted diff) +1
-3- case NVME_AER_NOTICE_DISC_CHANGED: + default: ctrl->aen_result = result; break; - default: - dev_warn(ctrl->device, "async event result %08x\n", result);
I'd keep the log... and also don't know if we want to pass any unknown possible spam to userspace... not sure that being blindly forward compatible is the right choice here.
Review void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status, · line 4804
Quoted 9 lines
} return requeue;}
(quoted diff) +2
-0@@ -4799,8 +4800,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status, break; } - if (requeue) + if (requeue) { + ctrl->aen_data = le64_to_cpu(res->u64) >> 32;
Please use a helper for that.aen_data is not a great name... maybe use a union for that for the specific aen subtype?
Maybe it'd be better that zoned namespace aen is handled explicitly and passes a AEN_NSID env veriable to the uevent.
Quoted 18 lines
From: Sagi Grimberg <[email protected]>
Sent: Thursday, October 6, 2022 4:34 AM
To: Clay Mayers <[email protected]>; [email protected]
Cc: Keith Busch <[email protected]>; Jens Axboe <[email protected]>; Christoph Hellwig <[email protected]>
Subject: Re: [PATCH V2 0/2] nvme: Support user mode processing of ZNS Zone Changed AENQuoted 9 lines
This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.
Patch 1 adds non-zero
NVME_AEN_DATAwith the value of CQE.DW1 to allNVME_AENuevents.Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.
Quoted 8 lines
Quoted 6 lines
This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.
Changes since v1
- Break up into two patches
- Only inlucdeAEN_DATAif CQE.DW1 is non-zeroWhat about the other comments that were given on v1?
My mistake
Quoted 10 lines
--
Review
Quoted 7 lines
#endif(quoted diff) +1
-3- case NVME_AER_NOTICE_DISC_CHANGED: + default: ctrl->aen_result = result; break; - default: - dev_warn(ctrl->device, "async event result %08x\n", result);
Quoted 3 lines
I'd keep the log... and also don't know if we want to pass any unknown possible spam to userspace... not sure that being blindly forward compatible is the right choice here.
I'll put it back in. I went back and forth on this. My reasoning for removing is that the other types of AEN don't warn and always generate a uevent. Only the NOTICE type conditionally sets ctrl->aen_result before queueing async_event_work.
The real control over AENs is the AEN config feature. When an AEN is generated and unhandled, all the AENs of that type will no longer be sent. They can't be on by default without breaking the system so there won't be a flood of meaningless NVME_AEN uevents.
The ones to suppress are those that don't go through default and are handled in the kernel. That's unchanged.
Quoted 12 lines
Review void nvme_complete_async_event(struct
Quoted 5 lines
} return requeue;}
(quoted diff) +0
-0@@ -4799,8 +4800,10 @@ void nvme_complete_async_event(struct
nvme_ctrl*ctrl,__le16status,Review void nvme_complete_async_event(struct · line 4804
Quoted 5 lines
(quoted diff) +2
-0break; } - if (requeue) + if (requeue) { + ctrl->aen_data = le64_to_cpu(res->u64) >> 32;
Quoted 6 lines
Please use a helper for that.
aen_datais not a great name... maybe use a union for that for the specific aen subtype?Maybe it'd be better that zoned namespace aen is handled explicitly and passes a
AEN_NSIDenv veriable to the uevent.
I'll respin with ZNS specific handling and naming with a union. I think it's valuable to keep generating uevents with AEN_DATA when not ZNS or NOTICE. Some recent work has allowed all alternate command sets to be used through an ng device without changes to the core. This fills the gap of AEN processing. The down side is it's difficult to test the non-ZNS path when ZNS is special cased.