Skip to contentSkip to conversation

nvmet: make nvmet_wq visible in sysfs

PATCH In review Guixin Liu 4 participants 7 messages Oct 31, 2024 – Nov 5, 2024

Subsystems (2)
lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

PATCH v2 1/1 nvmet: make nvmet_wq visible in sysfs

In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as:
1. Docker containers for function computation (background processing).
2. Docker containers for real-time tasks.
3. Docker containers for monitoring, event handling, and management.
4. An NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. Additionally, we make the nvmet_wq visible in sysfs, allowing for tuning its attributes through sysfs, such as cpumask.

---
 drivers/nvme/target/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/core.c +1 -1
index ed2424f8a396..15b25f464e77 100644
@@ -1717,7 +1717,7 @@ static int __init nvmet_init(void)
 		goto out_free_zbd_work_queue;
 
 	nvmet_wq = alloc_workqueue("nvmet-wq",
-			WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+			WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);
 	if (!nvmet_wq)
 		goto out_free_buffered_work_queue;
 
Signature · 1 line
2.43.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 10 lines · Guixin Liu

In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as:
1. Docker containers for function computation (background processing).
2. Docker containers for real-time tasks.
3. Docker containers for monitoring, event handling, and management.
4. An NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. Additionally, we make the nvmet_wq visible in sysfs, allowing for tuning its attributes through sysfs, such as cpumask.

How about following ? no need to send V3 can be done at the time of applying the patch if you are okay with it :-

" In some complex scenarios, we deploy multiple taskson asingle machine (hybrid deployment), suchas Docker containersfor function computation (background processing), real-time tasks, monitoring,event handling, and management, alongwith an NVMe target server.

Each of these componentsis restrictedto its own CPU coresto prevent mutual interferenceand ensurestrict isolation.To achieve this level of isolation for nvmet_wq we needto use sysfs tunables such as cpumask that are currently not accessible.

Add WQ_SYSFS flag to alloc_workqueue() when creating nvmet_wq so workqueue tunables are exported in the userspace via sysfs.

with this patch :-

nvme (nvme-6.13) # ls /sys/devices/virtual/workqueue/nvmet-wq/ affinity_scope affinity_strict cpumask max_active nice per_cpu power subsystem uevent

"

With that looks good.

-ck

author
#2 lore raw
Quoted 34 lines · Chaitanya Kulkarni
Quoted 10 lines · Guixin Liu

In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as:
1. Docker containers for function computation (background processing).
2. Docker containers for real-time tasks.
3. Docker containers for monitoring, event handling, and management.
4. An NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. Additionally, we make the nvmet_wq visible in sysfs, allowing for tuning its attributes through sysfs, such as cpumask.

How about following ? no need to send V3 can be done at the time of applying the patch if you are okay with it :-

" In some complex scenarios, we deploy multiple taskson asingle machine (hybrid deployment), suchas Docker containersfor function computation (background processing), real-time tasks, monitoring,event handling, and management, alongwith an NVMe target server.

Each of these componentsis restrictedto its own CPU coresto prevent mutual interferenceand ensurestrict isolation.To achieve this level of isolation for nvmet_wq we needto use sysfs tunables such as cpumask that are currently not accessible.

Add WQ_SYSFS flag to alloc_workqueue() when creating nvmet_wq so workqueue tunables are exported in the userspace via sysfs.

with this patch :-

nvme (nvme-6.13) # ls /sys/devices/virtual/workqueue/nvmet-wq/ affinity_scope affinity_strict cpumask max_active nice per_cpu power subsystem uevent

Quoted 8 lines

"

With that looks good.

-ck

Thanks for tunning the commit message, the new content looks good,

but I see some words are joined together, "coresto" -> "cores to",

"interferenceand" -> "interference and", and so on.

Please change this when you applying the patch.

Best Regards,

Guixin Liu

#3 lore raw
Quoted 11 lines · Guixin Liu

Thanks for tunning the commit message, the new content looks good,

but I see some words are joined together, "coresto" -> "cores to",

"interferenceand" -> "interference and", and so on.

Please change this when you applying the patch.

Best Regards,

Guixin Liu

Here is the updated one :-

In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as Docker containers for function computation (background processing), real-time tasks, monitoring, event handling, and management, along with an NVMe target server.

Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. To achieve this level of isolation for nvmet_wq we need to  use sysfs tunables such as cpumask that are currently not accessible.

Add WQ_SYSFS flag to alloc_workqueue() when creating nvmet_wq so workqueue tunables are exported in the userspace via sysfs.

with this patch :-

nvme (nvme-6.13) # ls /sys/devices/virtual/workqueue/nvmet-wq/ affinity_scope  affinity_strict  cpumask  max_active  nice per_cpu power  subsystem  uevent

-ck

author
#5 lore raw
In reply to Chaitanya Kulkarni
Quoted 27 lines · Chaitanya Kulkarni
Quoted 11 lines · Guixin Liu

Thanks for tunning the commit message, the new content looks good,

but I see some words are joined together, "coresto" -> "cores to",

"interferenceand" -> "interference and", and so on.

Please change this when you applying the patch.

Best Regards,

Guixin Liu

Here is the updated one :-

In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as Docker containers for function computation (background processing), real-time tasks, monitoring, event handling, and management, along with an NVMe target server.

Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. To achieve this level of isolation for nvmet_wq we need to  use sysfs tunables such as cpumask that are currently not accessible.

Add WQ_SYSFS flag to alloc_workqueue() when creating nvmet_wq so workqueue tunables are exported in the userspace via sysfs.

Quoted 9 lines

with this patch :-

nvme (nvme-6.13) # ls /sys/devices/virtual/workqueue/nvmet-wq/ affinity_scope  affinity_strict  cpumask  max_active  nice per_cpu power  subsystem  uevent

-ck

Looks good now, my deepest gratitude for the tunning.

Best Regards,

Guixin Liu

maintainer
#6 lore raw
Quoted 1 line · Guixin Liu

Looks good now, my deepest gratitude for the tunning.

Thanks all, applied to nvme-6.13 with the updated commit message.