Skip to contentSkip to conversation

nvme: Support user mode processing of ZNS Zone Changed AEN

PATCH In review [email protected] 4 participants 13 messages Sep 29, 2022 – Oct 29, 2022 NVMe

lore
Share & tools
lore
#0 lore raw

v2 cover nvme: Support user mode processing of ZNS Zone Changed AEN

From: Clay Mayers <[email protected]>

This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.

Patch 1 adds non-zero NVME_AEN_DATA with the value of CQE.DW1 to all NVME_AEN uevents.

Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.

This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.

Changes since v1
- Break up into two patches
- Only inlucde AEN_DATA if CQE.DW1 is non-zero

Clay Mayers (2):
nvme: Include AEN CQE.DW1 in NVME_AEN uevents
nvme: All AENs of type notify generate an NVME_AEN uevent

 drivers/nvme/host/core.c | 21 +++++++++++++--------
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 14 insertions(+), 8 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH v2 1/2 nvme: Include AEN CQE.DW1 in NVME_AEN uevents

From: Clay Mayers <[email protected]>

There are AENs from alternate command sets that include extra data in their AEN's CQE.DW1. For example, the ZNS Zone-Descriptor-Changed AEN uses it to indicate the NSID of the event's log page.

NVME_AEN uevent now includes the value of CQE.DW1 as a new property. It is only included when non-zero to keep previously existing uevents unmodified.

---
 drivers/nvme/host/core.c | 17 ++++++++++++-----
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 13 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +12 -5
index 66446f1e06cf..4fcc7aec5a8c 100644
@@ -4615,17 +4615,22 @@ static void nvme_change_uevent(struct nvme_ctrl *ctrl, char *envdata)
 
 static void nvme_aen_uevent(struct nvme_ctrl *ctrl)
 {
-	char *envp[2] = { NULL, NULL };
+	char *envp[3] = { NULL, NULL, NULL };
 	u32 aen_result = ctrl->aen_result;
+	u32 aen_data = ctrl->aen_data;
 
 	ctrl->aen_result = 0;
+	ctrl->aen_data = 0;
 	if (!aen_result)
 		return;
 
 	envp[0] = kasprintf(GFP_KERNEL, "NVME_AEN=%#08x", aen_result);
-	if (!envp[0])
-		return;
-	kobject_uevent_env(&ctrl->device->kobj, KOBJ_CHANGE, envp);
+	if (aen_data)
+		envp[1] = kasprintf(GFP_KERNEL, "NVME_AEN_DATA=%#08x", aen_data);
+	if (envp[0] && (envp[1] || !aen_data))
+		kobject_uevent_env(&ctrl->device->kobj, KOBJ_CHANGE, envp);
+
+	kfree(envp[1]);
 	kfree(envp[0]);
 }
 
@@ -4799,8 +4804,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
 		break;
 	}
 
-	if (requeue)
+	if (requeue) {
+		ctrl->aen_data = le64_to_cpu(res->u64) >> 32;
 		queue_work(nvme_wq, &ctrl->async_event_work);
+	}
 }
 EXPORT_SYMBOL_GPL(nvme_complete_async_event);
 
drivers/nvme/host/nvme.h +1 -0
index 1bdf714dcd9e..a1bab447dc65 100644
@@ -299,6 +299,7 @@ struct nvme_ctrl {
 	u16 cctemp;
 	u32 oaes;
 	u32 aen_result;
+	u32 aen_data;
 	u32 ctratt;
 	unsigned int shutdown_timeout;
 	unsigned int kato;
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents

Quoted 10 lines · On Thu, Sep 29, 2022 at 03:39:54PM -0700, [email protected] wrote:

From: Clay Mayers <[email protected]>

There are AENs from alternate command sets that include extra data in their AEN's CQE.DW1. For example, the ZNS Zone-Descriptor-Changed AEN uses it to indicate the NSID of the event's log page.

NVME_AEN uevent now includes the value of CQE.DW1 as a new property. It is only included when non-zero to keep previously existing uevents unmodified.

I don't think we can actually do this. If we ever want to handle Zone Excursions or any other event where the device can chane the Zone Descriptor we need to handle this AEN in the kernel and thus control the clearing of the even by reading the associated log page, so we can't just send it on to userspace. This is a bit of a sad state of affairs but unavoidable due to the NVMe AEN design.

#3 lore raw

RE: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents

Quoted 15 lines

From: Christoph Hellwig <[email protected]>
Sent: Monday, October 17, 2022 6:24 AM

Quoted 10 lines · On Thu, Sep 29, 2022 at 03:39:54PM -0700, [email protected] wrote:

From: Clay Mayers <[email protected]>

There are AENs from alternate command sets that include extra data in their AEN's CQE.DW1. For example, the ZNS Zone-Descriptor-Changed AEN uses it to indicate the NSID of the event's log page.

NVME_AEN uevent now includes the value of CQE.DW1 as a new property. It is only included when non-zero to keep previously existing uevents unmodified.

Quoted 6 lines

I don't think we can actually do this. If we ever want to handle Zone Excursions or any other event where the device can chane the Zone Descriptor we need to handle this AEN in the kernel and thus control the clearing of the even by reading the associated log page, so we can't just send it on to userspace. This is a bit of a sad state of affairs but unavoidable due to the NVMe AEN design.

What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?

The limitation is, the ZDC AEN is enabled/disabled at the controller level but it's essentially a namespace level event. Once on for one namespace, zone descriptor changes have to be handled as an AEN for all namespaces.

#4 lore raw

Re: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents

Quoted 5 lines · Clay Mayers

What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?

The problem is how the NVMe AENs work - the are raised and then the AEN command completion is delivered and they are cleared by reading the log page. But if we deliver them to userspace, we lose the pending state of the AEN for the kernel. Right now that is not an issue, but we lose all chance of ever making use of that information in the kernel. So maybe the answer is to handle the AEN in the kernel, read the changed zones log in the kernel, and then send an uevent for all changes zones to userspace.

maintainer
#5 lore raw

Re: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents

Quoted 7 lines
Quoted 6 lines

What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?

Quoted 8 lines

The problem is how the NVMe AENs work - the are raised and then the AEN command completion is delivered and they are cleared by reading the log page. But if we deliver them to userspace, we lose the pending state of the AEN for the kernel. Right now that is not an issue, but we lose all chance of ever making use of that information in the kernel. So maybe the answer is to handle the AEN in the kernel, read the changed zones log in the kernel, and then send an uevent for all changes zones to userspace.

That is what I was thinking... I feel we had this discussion before on a different AEN (maybe it was something that wanted to get ANA info to userspace)...

#6 lore raw

RE: [PATCH V2 1/2] nvme: Include AEN CQE.DW1 in NVME_AEN uevents

Quoted 17 lines

From: Sagi Grimberg <[email protected]>
Sent: Tuesday, October 25, 2022 9:00 AM

Quoted 6 lines
Quoted 5 lines

What happens today is a warning is logged and the log page is left unread. The patch closes that gap allowing ZDC AENs to be enable and handled in user space for things like RocksDB's ZenFS. Kernel clients will also need a way to handle them, but can't that be a different patch series?

Quoted 8 lines

The problem is how the NVMe AENs work - the are raised and then the AEN command completion is delivered and they are cleared by reading the log page. But if we deliver them to userspace, we lose the pending state of the AEN for the kernel. Right now that is not an issue, but we lose all chance of ever making use of that information in the kernel. So maybe the answer is to handle the AEN in the kernel, read the changed zones log in the kernel, and then send an uevent for all changes zones to userspace.

Quoted 3 lines

That is what I was thinking... I feel we had this discussion before on a different AEN (maybe it was something that wanted to get ANA info to userspace)...

If you're thinking of this: https://lore.kernel.org/linux-nvme/BL0PR13MB4291371B4FD44B93D187B2319C419@BL0PR13MB4291.namprd13.prod.outlook.com/
I share the concern of both user and kernel reading the AEN referenced
log page. The solution should look more like what Christoph outlined
above with the kernel reading the log page and routing results.

However, the ZDC AEN is not at the controller level. It supplies the NSID so only the code writing to the namespace can be notified and only it will read the log page. Host-aware zoned devices can't be partitioned so responsibility wouldn't be split across user mode, kernel or multiple applications.

Yes, the kernel doesn't get to look at my namespace's ZDC entries and It has no reason to. In the future when the kernel is processing ZDC entries for a namespace, I shouldn’t expect a uevent or to see its ZDC entries just like two apps using different namespaces.

#7 lore raw

PATCH v2 2/2 nvme: All AENs of type notify generate an NVME_AEN uevent

In reply to [email protected]

From: Clay Mayers <[email protected]>

This enables user mode processing of all unhandled AENs of type NOTIFY.

---
 drivers/nvme/host/core.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/nvme/host/core.c +1 -3
index 4fcc7aec5a8c..a3d8e6476af3 100644
@@ -4753,11 +4753,9 @@ static bool nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		queue_work(nvme_wq, &ctrl->ana_work);
 		break;
 #endif
-	case NVME_AER_NOTICE_DISC_CHANGED:
+	default:
 		ctrl->aen_result = result;
 		break;
-	default:
-		dev_warn(ctrl->device, "async event result %08x\n", result);
 	}
 	return requeue;
 }
Signature · 1 line
2.27.0
#8 lore raw
In reply to [email protected]
Quoted 5 lines · On 9/29/22 15:39, [email protected] wrote:

From: Clay Mayers <[email protected]>

This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.

Quoted 10 lines

Patch 1 adds non-zero NVME_AEN_DATA with the value of CQE.DW1 to all NVME_AEN uevents.

Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.

This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.

Please write a testcase in the blktest to this will get tested every release.

-ck

#9 lore raw
Quoted 22 lines

From: Chaitanya Kulkarni <[email protected]>
Sent: Monday, October 3, 2022 6:49 PM
Subject: Re: [PATCH V2 0/2] nvme: Support user mode processing of ZNS Zone Changed AEN

Quoted 5 lines · On 9/29/22 15:39, [email protected] wrote:

From: Clay Mayers <[email protected]>

This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.

Quoted 10 lines

Patch 1 adds non-zero NVME_AEN_DATA with the value of CQE.DW1 to all NVME_AEN uevents.

Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.

This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.

Quoted 4 lines

Please write a testcase in the blktest to this will get tested every release.

-ck

Ok - I'll give it a go to implement that but AENs by definition aren't a response to a request (mostly) - they just happen. The case that I know of that creates a zone excursion AEN takes many hours to manifest and that duration would be FLASH-type specific. I don't know how you could use a real device.

This patch was tested under QEMU with our custom emulator. The current QEMU NVMe/ZNS device doesn't emulate zone excursions so I'll have to start there unless someone knows of a better way.

#10 lore raw
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.

Quoted 7 lines
Quoted 4 lines

Please write a testcase in the blktest to this will get tested every release.

-ck

Ok - I'll give it a go to implement that but AENs by definition aren't a response to a request (mostly) - they just happen. The case that I know

and this is exactly the main reason we need a testcase since it is sporadic and side effect of the controller's behavior that cannot be predicted (untill it hits the condition) by the requests. Lack of the testcase will leave this code untested unlike other code paths which are getting exercised predictably.

Quoted 4 lines

of that creates a zone excursion AEN takes many hours to manifest and that duration would be FLASH-type specific. I don't know how you could use a real device.

For this using real device is not an option since everyone should be able to test that with ease in blktests.

Quoted 4 lines

This patch was tested under QEMU with our custom emulator. The current QEMU NVMe/ZNS device doesn't emulate zone excursions so I'll have to start there unless someone knows of a better way.

Please add the support our QEMU NVMe developers are very helpful to get new features in.

-ck

maintainer
#11 lore raw
In reply to [email protected]
Quoted 10 lines

This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.

Patch 1 adds non-zero NVME_AEN_DATA with the value of CQE.DW1 to all NVME_AEN uevents.

Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.

Quoted 6 lines

This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.

Changes since v1
- Break up into two patches
- Only inlucde AEN_DATA if CQE.DW1 is non-zero

What about the other comments that were given on v1?

--

Review

Quoted 7 lines
  #endif
(quoted diff) +1 -3
-    case NVME_AER_NOTICE_DISC_CHANGED:
+    default:
          ctrl->aen_result = result;
          break;
-    default:
-        dev_warn(ctrl->device, "async event result %08x\n", result);

I'd keep the log... and also don't know if we want to pass any unknown possible spam to userspace... not sure that being blindly forward compatible is the right choice here.

Review void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status, · line 4804

Quoted 9 lines
      }
      return requeue;

}

(quoted diff) +2 -0
@@ -4799,8 +4800,10 @@ void nvme_complete_async_event(struct nvme_ctrl *ctrl, __le16 status,
          break;
      }
  -    if (requeue)
+    if (requeue) {
+        ctrl->aen_data = le64_to_cpu(res->u64) >> 32;

Please use a helper for that.
aen_data is not a great name... maybe use a union for that for the specific aen subtype?

Maybe it'd be better that zoned namespace aen is handled explicitly and passes a AEN_NSID env veriable to the uevent.

#12 lore raw
Quoted 18 lines

From: Sagi Grimberg <[email protected]>
Sent: Thursday, October 6, 2022 4:34 AM
To: Clay Mayers <[email protected]>; [email protected]
Cc: Keith Busch <[email protected]>; Jens Axboe <[email protected]>; Christoph Hellwig <[email protected]>
Subject: Re: [PATCH V2 0/2] nvme: Support user mode processing of ZNS Zone Changed AEN

Quoted 9 lines

This adds support for the ZNS Zone-Descriptor-Changes AEN, which is of type notify and uses CQE.DW1 to convey the NSID of the log page to read.

Patch 1 adds non-zero NVME_AEN_DATA with the value of CQE.DW1 to all NVME_AEN uevents.

Patch 2 generates a uevent for all unhandled AEN notify results instead of issuing a dev warning.

Quoted 8 lines
Quoted 6 lines

This support is planned to be used by both zone based applications and another unreleased device with an alternate command set.

Changes since v1
- Break up into two patches
- Only inlucde AEN_DATA if CQE.DW1 is non-zero

What about the other comments that were given on v1?

My mistake

Quoted 10 lines

--

Review

Quoted 7 lines
  #endif
(quoted diff) +1 -3
-    case NVME_AER_NOTICE_DISC_CHANGED:
+    default:
          ctrl->aen_result = result;
          break;
-    default:
-        dev_warn(ctrl->device, "async event result %08x\n", result);
Quoted 3 lines

I'd keep the log... and also don't know if we want to pass any unknown possible spam to userspace... not sure that being blindly forward compatible is the right choice here.

I'll put it back in. I went back and forth on this. My reasoning for removing is that the other types of AEN don't warn and always generate a uevent. Only the NOTICE type conditionally sets ctrl->aen_result before queueing async_event_work.

The real control over AENs is the AEN config feature. When an AEN is generated and unhandled, all the AENs of that type will no longer be sent. They can't be on by default without breaking the system so there won't be a flood of meaningless NVME_AEN uevents.

The ones to suppress are those that don't go through default and are handled in the kernel. That's unchanged.

Quoted 12 lines

Review void nvme_complete_async_event(struct

Quoted 5 lines
      }
      return requeue;

}

(quoted diff) +0 -0
@@ -4799,8 +4800,10 @@ void nvme_complete_async_event(struct

nvme_ctrl *ctrl, __le16 status,

Review void nvme_complete_async_event(struct · line 4804

Quoted 5 lines
(quoted diff) +2 -0
          break;
      }
  -    if (requeue)
+    if (requeue) {
+        ctrl->aen_data = le64_to_cpu(res->u64) >> 32;
Quoted 6 lines

Please use a helper for that.
aen_data is not a great name... maybe use a union for that for the specific aen subtype?

Maybe it'd be better that zoned namespace aen is handled explicitly and passes a AEN_NSID env veriable to the uevent.

I'll respin with ZNS specific handling and naming with a union. I think it's valuable to keep generating uevents with AEN_DATA when not ZNS or NOTICE. Some recent work has allowed all alternate command sets to be used through an ng device without changes to the core. This fills the gap of AEN processing. The down side is it's difficult to test the non-ZNS path when ZNS is special cased.