nvmet: make nvmet_wq visible in sysfs
Subsystems (2)
PATCH v2 1/1 nvmet: make nvmet_wq visible in sysfs
In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as:
1. Docker containers for function computation (background processing).
2. Docker containers for real-time tasks.
3. Docker containers for monitoring, event handling, and management.
4. An NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. Additionally, we make the nvmet_wq visible in sysfs, allowing for tuning its attributes through sysfs, such as cpumask.
--- drivers/nvme/target/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/core.c +1 -1
@@ -1717,7 +1717,7 @@ static int __init nvmet_init(void) goto out_free_zbd_work_queue; nvmet_wq =alloc_workqueue ("nvmet-wq" , -WQ_MEM_RECLAIM |WQ_UNBOUND ,0 ); +WQ_MEM_RECLAIM |WQ_UNBOUND |WQ_SYSFS ,0 );if (!nvmet_wq) goto out_free_buffered_work_queue;
Signature · 1 line
2.43.0
Quoted 10 lines · Guixin Liu
In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as:
1. Docker containers for function computation (background processing).
2. Docker containers for real-time tasks.
3. Docker containers for monitoring, event handling, and management.
4. An NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. Additionally, we make thenvmet_wqvisible in sysfs, allowing for tuning its attributes through sysfs, such as cpumask.
How about following ? no need to send V3 can be done at the time of applying the patch if you are okay with it :-
" In some complex scenarios, we deploy multiple taskson asingle machine (hybrid deployment), suchas Docker containersfor function computation (background processing), real-time tasks, monitoring,event handling, and management, alongwith an NVMe target server.
Each of these componentsis restrictedto its own CPU coresto prevent mutual interferenceand ensurestrict isolation.To achieve this level of isolation for nvmet_wq we needto use sysfs tunables such as cpumask that are currently not accessible.
Add WQ_SYSFS flag to alloc_workqueue() when creating nvmet_wq so workqueue tunables are exported in the userspace via sysfs.
with this patch :-
nvme (nvme-6.13) # ls /sys/devices/virtual/workqueue/nvmet-wq/ affinity_scope affinity_strict cpumask max_active nice per_cpu power subsystem uevent
"
With that looks good.
-ck
Quoted 34 lines · Chaitanya Kulkarni
Quoted 10 lines · Guixin Liu
In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as:
1. Docker containers for function computation (background processing).
2. Docker containers for real-time tasks.
3. Docker containers for monitoring, event handling, and management.
4. An NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. Additionally, we make thenvmet_wqvisible in sysfs, allowing for tuning its attributes through sysfs, such as cpumask.How about following ? no need to send V3 can be done at the time of applying the patch if you are okay with it :-
" In some complex scenarios, we deploy multiple taskson asingle machine (hybrid deployment), suchas Docker containersfor function computation (background processing), real-time tasks, monitoring,event handling, and management, alongwith an NVMe target server.
Each of these componentsis restrictedto its own CPU coresto prevent mutual interferenceand ensurestrict isolation.To achieve this level of isolation for
nvmet_wqwe needto use sysfs tunables such as cpumask that are currently not accessible.Add
WQ_SYSFSflag toalloc_workqueue()when creatingnvmet_wqso workqueue tunables are exported in the userspace via sysfs.with this patch :-
nvme (nvme-6.13) # ls
/sys/devices/virtual/workqueue/nvmet-wq/affinity_scopeaffinity_strictcpumaskmax_activeniceper_cpupower subsystem uevent
Quoted 8 lines
"
With that looks good.
Reviewed-by Chaitanya Kulkarni <[email protected]>-ck
Thanks for tunning the commit message, the new content looks good,
but I see some words are joined together, "coresto" -> "cores to",
"interferenceand" -> "interference and", and so on.
Please change this when you applying the patch.
Best Regards,
Guixin Liu
Quoted 11 lines · Guixin Liu
Thanks for tunning the commit message, the new content looks good,
but I see some words are joined together, "coresto" -> "cores to",
"interferenceand" -> "interference and", and so on.
Please change this when you applying the patch.
Best Regards,
Guixin Liu
Here is the updated one :-
In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as Docker containers for function computation (background processing), real-time tasks, monitoring, event handling, and management, along with an NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. To achieve this level of isolation for nvmet_wq we need to use sysfs tunables such as cpumask that are currently not accessible.
Add WQ_SYSFS flag to alloc_workqueue() when creating nvmet_wq so workqueue tunables are exported in the userspace via sysfs.
with this patch :-
nvme (nvme-6.13) # ls /sys/devices/virtual/workqueue/nvmet-wq/ affinity_scope affinity_strict cpumask max_active nice per_cpu power subsystem uevent
-ck
Looks good with the updated commit message:
Quoted 27 lines · Chaitanya Kulkarni
Quoted 11 lines · Guixin Liu
Thanks for tunning the commit message, the new content looks good,
but I see some words are joined together, "coresto" -> "cores to",
"interferenceand" -> "interference and", and so on.
Please change this when you applying the patch.
Best Regards,
Guixin Liu
Here is the updated one :-
In some complex scenarios, we deploy multiple tasks on a single machine (hybrid deployment), such as Docker containers for function computation (background processing), real-time tasks, monitoring, event handling, and management, along with an NVMe target server.
Each of these components is restricted to its own CPU cores to prevent mutual interference and ensure strict isolation. To achieve this level of isolation for
nvmet_wqwe need to use sysfs tunables such as cpumask that are currently not accessible.Add
WQ_SYSFSflag toalloc_workqueue()when creatingnvmet_wqso workqueue tunables are exported in the userspace via sysfs.
Quoted 9 lines
with this patch :-
nvme (nvme-6.13) # ls
/sys/devices/virtual/workqueue/nvmet-wq/affinity_scope affinity_strict cpumask max_active niceper_cpupower subsystem uevent-ck
Looks good now, my deepest gratitude for the tunning.
Best Regards,
Guixin Liu
Keith Busch
maintainerQuoted 1 line · Guixin Liu
Looks good now, my deepest gratitude for the tunning.
Thanks all, applied to nvme-6.13 with the updated commit message.