Skip to contentSkip to conversation

nvmet: make nvmet_wq visible in sysfs

PATCH Superseded Guixin Liu 3 participants 15 messages Oct 29, 2024 – Oct 31, 2024

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

PATCH 1/1 nvmet: make nvmet_wq visible in sysfs

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---
 drivers/nvme/target/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/core.c +1 -1
index ed2424f8a396..15b25f464e77 100644
@@ -1717,7 +1717,7 @@ static int __init nvmet_init(void)
 		goto out_free_zbd_work_queue;
 
 	nvmet_wq = alloc_workqueue("nvmet-wq",
-			WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+			WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_SYSFS, 0);
 	if (!nvmet_wq)
 		goto out_free_buffered_work_queue;
 
Signature · 1 line
2.43.0
author
#2 lore raw
Quoted 9 lines · Chaitanya Kulkarni
Quoted 5 lines · Guixin Liu

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---

do you happened have a usecase for this?

-ck

Sometimes, in order to respond promptly to certain events or

manage commands, we need to reserve resources and partition

the CPU cores. For example, if there are 4 cores available,

we can initially allocate them by dedicating one core for

management while the remaining 3 cores are specifically for handling IO.

Best Regards,

Guixin Liu

Quoted 1 line
#3 lore raw
Quoted 18 lines · Guixin Liu
Quoted 9 lines · Chaitanya Kulkarni
Quoted 5 lines · Guixin Liu

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---

do you happened have a usecase for this?

-ck

Sometimes, in order to respond promptly to certain events or

manage commands, we need to reserve resources and partition

the CPU cores. For example, if there are 4 cores available,

Quoted 8 lines

we can initially allocate them by dedicating one core for

management while the remaining 3 cores are specifically for handling IO.

Best Regards,

Guixin Liu

I'm aware of exposing tunables through sysfs and it's benefits, my question was do you have a setup where this setting is needed currently ?

I've always been asked to for the usecase on a patch when we expose something
out of kernel that is solving the problem in the deployment ...

-ck

Reviewed-by
#4 lore raw
Quoted 25 lines · Chaitanya Kulkarni
Quoted 16 lines · Guixin Liu
Quoted 9 lines · Chaitanya Kulkarni
Quoted 5 lines · Guixin Liu

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---

do you happened have a usecase for this?

-ck

Sometimes, in order to respond promptly to certain events or

manage commands, we need to reserve resources and partition

the CPU cores. For example, if there are 4 cores available,

Quoted 8 lines

we can initially allocate them by dedicating one core for

management while the remaining 3 cores are specifically for handling IO.

Best Regards,

Guixin Liu

Quoted 10 lines

I'm aware of exposing tunables through sysfs and it's benefits, my question was do you have a setup where this setting is needed currently ?

I've always been asked to for the usecase on a patch when we expose something
out of kernel that is solving the problem in the deployment ...

-ck

If others are okay then :-

Looks good.

-ck

author
#5 lore raw
In reply to Chaitanya Kulkarni
Quoted 25 lines · Chaitanya Kulkarni
Quoted 16 lines · Guixin Liu
Quoted 9 lines · Chaitanya Kulkarni
Quoted 5 lines · Guixin Liu

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---

do you happened have a usecase for this?

-ck

Sometimes, in order to respond promptly to certain events or

manage commands, we need to reserve resources and partition

the CPU cores. For example, if there are 4 cores available,

Quoted 8 lines

we can initially allocate them by dedicating one core for

management while the remaining 3 cores are specifically for handling IO.

Best Regards,

Guixin Liu

Quoted 8 lines

I'm aware of exposing tunables through sysfs and it's benefits, my question was do you have a setup where this setting is needed currently ?

I've always been asked to for the usecase on a patch when we expose something
out of kernel that is solving the problem in the deployment ...

-ck

I need serverve some cpu core to do other things, such as handle events

and managements, so that the nvmet_wq can't running on all cpu cores,

currently, I restrict it by setting the cpumask of nvmet_wq(that's why

I expose nvmet_wq to sysfs).

Best Regards,

Guixin Liu

Quoted 1 line
#6 lore raw
Quoted 7 lines · Guixin Liu

I need serverve some cpu core to do other things, such as handle events

and managements, so that the nvmet_wq can't running on all cpu cores,

currently, I restrict it by setting the cpumask of nvmet_wq(that's why

I expose nvmet_wq to sysfs).

Can you resend the patch with an explanation of the use case in the commit message? Thanks!

author
#7 lore raw
Quoted 10 lines · 在 2024/10/30 13:53, [email protected] 写道:
Quoted 7 lines · Guixin Liu

I need serverve some cpu core to do other things, such as handle events

and managements, so that the nvmet_wq can't running on all cpu cores,

currently, I restrict it by setting the cpumask of nvmet_wq(that's why

I expose nvmet_wq to sysfs).

Can you resend the patch with an explanation of the use case in the commit message? Thanks!

Sure.

#8 lore raw
In reply to Guixin Liu
Quoted 42 lines · Guixin Liu
Quoted 30 lines · Chaitanya Kulkarni
Quoted 16 lines · Guixin Liu
Quoted 9 lines · Chaitanya Kulkarni
Quoted 5 lines · Guixin Liu

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---

do you happened have a usecase for this?

-ck

Sometimes, in order to respond promptly to certain events or

manage commands, we need to reserve resources and partition

the CPU cores. For example, if there are 4 cores available,

Quoted 9 lines

we can initially allocate them by dedicating one core for

management while the remaining 3 cores are specifically for handling IO.

Best Regards,

Guixin Liu

I'm aware of exposing tunables through sysfs and it's benefits, my question
was do you have a setup where this setting is needed currently ?

Quoted 5 lines

I've always been asked to for the usecase on a patch when we expose something
out of kernel that is solving the problem in the deployment ...

-ck

I need serverve some cpu core to do other things, such as handle events

and managements, so that the nvmet_wq can't running on all cpu cores,

Quoted 9 lines

currently, I restrict it by setting the cpumask of nvmet_wq(that's why

I expose nvmet_wq to sysfs).

Best Regards,

Guixin Liu

Quoted 1 line

Can you please explain your setup ? e.g. transport tcp/rdma/fc, device backend file/block etc ?

so nvmet_wq's CPU consumption is so high, that it doesn't have bandwidth to handle events and managements ?

Can you please explain the workload and what kind of events and managements handling is needed where you need to restrict the nvmet_wq with CPUMASK ?

The only reason I'm asking that I've not seen this scenario so far in the many
many deployments since we've added the nvmet_wq and I'd really like to learn about scenario.

-ck

author
#9 lore raw
Quoted 58 lines · Chaitanya Kulkarni
Quoted 40 lines · Guixin Liu
Quoted 30 lines · Chaitanya Kulkarni
Quoted 16 lines · Guixin Liu
Quoted 9 lines · Chaitanya Kulkarni
Quoted 5 lines · Guixin Liu

Make nvmet_wq visible in sysfs, allowing for tuning the it's attr through sysfs.

---

do you happened have a usecase for this?

-ck

Sometimes, in order to respond promptly to certain events or

manage commands, we need to reserve resources and partition

the CPU cores. For example, if there are 4 cores available,

Quoted 9 lines

we can initially allocate them by dedicating one core for

management while the remaining 3 cores are specifically for handling IO.

Best Regards,

Guixin Liu

I'm aware of exposing tunables through sysfs and it's benefits, my question
was do you have a setup where this setting is needed currently ?

Quoted 5 lines

I've always been asked to for the usecase on a patch when we expose something
out of kernel that is solving the problem in the deployment ...

-ck

I need serverve some cpu core to do other things, such as handle events

and managements, so that the nvmet_wq can't running on all cpu cores,

Quoted 8 lines

currently, I restrict it by setting the cpumask of nvmet_wq(that's why

I expose nvmet_wq to sysfs).

Best Regards,

Guixin Liu

Can you please explain your setup ? e.g. transport tcp/rdma/fc, device backend file/block etc ?

so nvmet_wq's CPU consumption is so high, that it doesn't have bandwidth to handle events and managements ?

Can you please explain the workload and what kind of events and managements handling is needed where you need to restrict the nvmet_wq with CPUMASK ?

Quoted 6 lines

The only reason I'm asking that I've not seen this scenario so far in the many
many deployments since we've added the nvmet_wq and I'd really like to learn about scenario.

-ck

Sorry for the unclear explanation.

The transport is tcp and the backend is block.

This is just a solution level thing, in some complicated scenarios, we deploy multiple

missions on one machine(hybrid deployment), such as:

1. Dockers for function computation.

2. Real-time tasks.

3. Monitoring, and handling events and managemets.

4. And also nvme target server.

All of them are retrict to their own cpu cores to prevent mutual influence.

There is no problem if nvmet_wq running on all cpus of course, but for strict isolation,

we need to do this retriction.

I dont know if I've given enough detail.

Best Regards,

Guixin Liu

Quoted 1 line
#10 lore raw
Quoted 24 lines · Guixin Liu

Sorry for the unclear explanation.

The transport is tcp and the backend is block.

This is just a solution level thing, in some complicated scenarios, we deploy multiple

missions on one machine(hybrid deployment), such as:

1. Dockers for function computation.

2. Real-time tasks.

3. Monitoring, and handling events and managemets.

4. And also nvme target server.

All of them are retrict to their own cpu cores to prevent mutual influence.

There is no problem if nvmet_wq running on all cpus of course, but for strict isolation,

Quoted 7 lines

we need to do this retriction.

I dont know if I've given enough detail.

Best Regards,

Guixin Liu

can you please send a patch with detailed usecase ?

Also, it will be nice (not a blocker to merge this patch) if you can provide
steps similar to listed above so we can get this scenario tested, even better
if you can submit a block test, but if not I'll send one once I get steps.

-ck

author
#11 lore raw
Quoted 33 lines · Chaitanya Kulkarni
Quoted 23 lines · Guixin Liu

Sorry for the unclear explanation.

The transport is tcp and the backend is block.

This is just a solution level thing, in some complicated scenarios, we deploy multiple

missions on one machine(hybrid deployment), such as:

1. Dockers for function computation.

2. Real-time tasks.

3. Monitoring, and handling events and managemets.

4. And also nvme target server.

All of them are retrict to their own cpu cores to prevent mutual influence.

There is no problem if nvmet_wq running on all cpus of course, but for strict isolation,

Quoted 7 lines

we need to do this retriction.

I dont know if I've given enough detail.

Best Regards,

Guixin Liu

can you please send a patch with detailed usecase ?

Quoted 7 lines

Also, it will be nice (not a blocker to merge this patch) if you can provide
steps similar to listed above so we can get this scenario tested, even better
if you can submit a block test, but if not I'll send one once I get steps.

-ck

I will send the v2 with our usecase to expain why we should restrict the cpumask,

I'm concerned whether blktests can handle such complex tests, as it relies on deploying

many Docker containers and services. Should it only test the case of setting the cpumask

with fio?

Best Regards,

Guixin Liu

Quoted 1 line
#12 lore raw
Quoted 50 lines · Guixin Liu



Quoted 34 lines · Chaitanya Kulkarni
Quoted 24 lines · Guixin Liu

Sorry for the unclear explanation.

The transport is tcp and the backend is block.

This is just a solution level thing, in some complicated scenarios, we deploy multiple

missions on one machine(hybrid deployment), such as:

1. Dockers for function computation.

2. Real-time tasks.

3. Monitoring, and handling events and managemets.

4. And also nvme target server.

All of them are retrict to their own cpu cores to prevent mutual influence.

There is no problem if nvmet_wq running on all cpus of course, but for strict isolation,

Quoted 7 lines

we need to do this retriction.

I dont know if I've given enough detail.

Best Regards,

Guixin Liu

can you please send a patch with detailed usecase ?

Quoted 7 lines

Also, it will be nice (not a blocker to merge this patch) if you can provide
steps similar to listed above so we can get this scenario tested, even better
if you can submit a block test, but if not I'll send one once I get steps.

-ck

I will send the v2 with our usecase to expain why we should restrict the cpumask,

I'm concerned whether blktests can handle such complex tests, as it relies on deploying

Quoted 8 lines

many Docker containers and services. Should it only test the case of setting the cpumask

with fio?

Best Regards,

Guixin Liu

For now just cpumask and fio is sufficient so that when we upstream this patch we have some sort of testing done via sysfs.

-ck

#13 lore raw
Quoted 8 lines · Chaitanya Kulkarni
Quoted 8 lines

I will send the v2 with our usecase to expain why we should restrict the cpumask,

I'm concerned whether blktests can handle such complex tests, as it relies on deploying

many Docker containers and services. Should it only test the case of setting the cpumask

with fio?

Quoted 8 lines
Quoted 4 lines

Best Regards,

Guixin Liu

For now just cpumask and fio is sufficient so that when we upstream this patch we have some sort of testing done via sysfs.

-ck

based on my very limited understanding I've written a rough blktest for your patch see if this helps, it's bit rough and totally untested  :-

#!/bin/bash
# SPDX-License-Identifier: GPL-3.0+
# Description: Test nvmet_wq cpumask sysfs attribute with NVMe-oF and fio workload

. tests/nvme/rc

DESCRIPTION="Test nvmet_wq cpumask sysfs attribute and verify with fio on NVMe-oF device"

requires() {
    _nvme_requires
    _have_fio
    _require_nvme_trtype_is_fabrics
}

test() {
    local cpumask_path="/sys/devices/virtual/workqueue/nvmet_wq/cpumask"

    # Check if cpumask attribute exists
    if [[ ! -f "$cpumask_path" ]]; then SKIP_REASONS+=("nvmet_wq cpumask sysfs attribute not found.") return 1
    fi

    # Save original cpumask value
    local original_cpumask
    original_cpumask=$(cat "$cpumask_path")
    echo "Original cpumask: $original_cpumask"

    # Set a new cpumask (e.g., CPU 0)
    echo 1 | tee "$cpumask_path" > /dev/null
    local new_cpumask
    new_cpumask=$(cat "$cpumask_path")

    if [[ "$new_cpumask" != "1" ]]; then
        echo "Test Failed: cpumask was not set correctly" return 1
    else
        echo "Test Passed: cpumask set to $new_cpumask" fi

    # Set up NVMe-over-Fabrics target
    echo "Setting up NVMe-oF target"
    _setup_nvmet
    _nvmet_target_setup
    _nvme_connect_subsys

    # Locate the NVMe-oF namespace
    local ns
    ns=$(_find_nvme_ns "${def_subsys_uuid}")

    # Run fio with data verification on NVMe-oF device
    echo "Starting fio workload with verification on NVMe-oF device" fio --name=nvmet_wq_test --filename="/dev/$ns" --direct=1
--rw=randwrite \
        --bs=4k --size=100M --numjobs=1 --verify=crc32c --verify_fatal=1 \
        --time_based --runtime=30s --iodepth=16 --ioengine=libaio
--group_reporting

    # Disconnect and clean up NVMe-oF target
    echo "Cleaning up NVMe-oF setup"
    _nvme_disconnect_subsys
    _nvmet_target_cleanup

    # Restore original cpumask
    echo "$original_cpumask" | tee "$cpumask_path" > /dev/null restored_cpumask=$(cat "$cpumask_path")

    if [[ "$restored_cpumask" != "$original_cpumask" ]]; then echo "Failed to restore original cpumask." return 1
    else
        echo "Original cpumask restored successfully." fi
}

author
#14 lore raw
Quoted 90 lines · Chaitanya Kulkarni
Quoted 8 lines · Chaitanya Kulkarni
Quoted 8 lines

I will send the v2 with our usecase to expain why we should restrict the cpumask,

I'm concerned whether blktests can handle such complex tests, as it relies on deploying

many Docker containers and services. Should it only test the case of setting the cpumask

with fio?

Quoted 8 lines
Quoted 4 lines

Best Regards,

Guixin Liu

For now just cpumask and fio is sufficient so that when we upstream this patch we have some sort of testing done via sysfs.

-ck

based on my very limited understanding I've written a rough blktest for your patch see if this helps, it's bit rough and totally untested  :-

#!/bin/bash
# SPDX-License-Identifier: GPL-3.0+
# Description: Test nvmet_wq cpumask sysfs attribute with NVMe-oF and fio workload

. tests/nvme/rc

DESCRIPTION="Test nvmet_wq cpumask sysfs attribute and verify with fio on NVMe-oF device"

requires() {
    _nvme_requires
    _have_fio
    _require_nvme_trtype_is_fabrics
}

test() {
    local cpumask_path="/sys/devices/virtual/workqueue/nvmet_wq/cpumask"

    # Check if cpumask attribute exists
    if [[ ! -f "$cpumask_path" ]]; then SKIP_REASONS+=("nvmet_wq cpumask sysfs attribute not found.") return 1
    fi

      # Save original cpumask value
      local original_cpumask
      original_cpumask=$(cat "$cpumask_path")
      echo "Original cpumask: $original_cpumask"

      # Set a new cpumask (e.g., CPU 0)
      echo 1 | tee "$cpumask_path" > /dev/null
      local new_cpumask
      new_cpumask=$(cat "$cpumask_path")

    if [[ "$new_cpumask" != "1" ]]; then
        echo "Test Failed: cpumask was not set correctly" return 1
    else
        echo "Test Passed: cpumask set to $new_cpumask" fi

    # Set up NVMe-over-Fabrics target
    echo "Setting up NVMe-oF target"
    _setup_nvmet
    _nvmet_target_setup
    _nvme_connect_subsys

      # Locate the NVMe-oF namespace
      local ns
      ns=$(_find_nvme_ns "${def_subsys_uuid}")

    # Run fio with data verification on NVMe-oF device
    echo "Starting fio workload with verification on NVMe-oF device" fio --name=nvmet_wq_test --filename="/dev/$ns" --direct=1
--rw=randwrite \
        --bs=4k --size=100M --numjobs=1 --verify=crc32c --verify_fatal=1 \
        --time_based --runtime=30s --iodepth=16 --ioengine=libaio
--group_reporting

    # Disconnect and clean up NVMe-oF target
    echo "Cleaning up NVMe-oF setup"
    _nvme_disconnect_subsys
    _nvmet_target_cleanup

      # Restore original cpumask
      echo "$original_cpumask" | tee "$cpumask_path" > /dev/null
      restored_cpumask=$(cat "$cpumask_path")
Quoted 7 lines

    if [[ "$restored_cpumask" != "$original_cpumask" ]]; then echo "Failed to restore original cpumask." return 1
    else
        echo "Original cpumask restored successfully." fi
}

The script looks good, thank you for adding the new test to my patch.

Best Regards,

Guixin Liu