Skip to contentSkip to conversation

scsi: scsi_host_queue_ready: increase busy count early

PATCH In review [email protected] 7 participants 27 messages Jan 20, 2021 – Mar 11, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scsi: scsi_host_queue_ready: increase busy count early

From: Martin Wilck <[email protected]>

Donald: please give this patch a try.

Commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq") contained this hunk:

-       busy = atomic_inc_return(&shost->host_busy) - 1;
        if (atomic_read(&shost->host_blocked) > 0) {
-               if (busy)
+               if (scsi_host_busy(shost) > 0)
                        goto starved;

The previous code would increase the busy count before checking host_blocked. With 6eb045e092ef, the busy count would be increased (by setting the SCMD_STATE_INFLIGHT bit) after the if clause for host_blocked above.

Users have reported a regression with the smartpqi driver [1] which has been shown to be caused by this commit [2].

It seems that by moving the increase of the busy counter further down, it could happen that the can_queue limit of the controller could be exceeded if several CPUs were executing this code in parallel on different queues.

This patch attempts to fix it by moving setting the SCMD_STATE_INFLIGHT before the host_blocked test again. It also inserts barriers to make sure scsi_host_busy() on once CPU will notice the increase of the count from another.

[1]: https://marc.info/?l=linux-scsi&m=160271263114829&w=2
[2]: https://marc.info/?l=linux-scsi&m=161116163722099&w=2

Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")
Cc Ming Lei <[email protected]>
Cc Don Brace <[email protected]>
Cc Kevin Barnett <[email protected]>
Cc Donald Buczek <[email protected]>
Cc John Garry <[email protected]>
Cc Paul Menzel <[email protected]>
---
 drivers/scsi/hosts.c    | 2 ++
 drivers/scsi/scsi_lib.c | 8 +++++---
 2 files changed, 7 insertions(+), 3 deletions(-)
drivers/scsi/hosts.c +2 -0
index 2f162603876f..1c452a1c18fd 100644
@@ -564,6 +564,8 @@ static bool scsi_host_check_in_flight(struct request *rq, void *data,
 	int *count = data;
 	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
 
+	/* This pairs with set_bit() in scsi_host_queue_ready() */
+	smp_mb__before_atomic();
 	if (test_bit(SCMD_STATE_INFLIGHT, &cmd->state))
 		(*count)++;
 
drivers/scsi/scsi_lib.c +5 -3
index b3f14f05340a..0a9a36c349ee 100644
@@ -1353,8 +1353,12 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
 	if (scsi_host_in_recovery(shost))
 		return 0;
 
+	set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
+	/* This pairs with test_bit() in scsi_host_check_in_flight() */
+	smp_mb__after_atomic();
+
 	if (atomic_read(&shost->host_blocked) > 0) {
-		if (scsi_host_busy(shost) > 0)
+		if (scsi_host_busy(shost) > 1)
 			goto starved;
 
 		/*
@@ -1379,8 +1383,6 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
 		spin_unlock_irq(shost->host_lock);
 	}
 
-	__set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
-
 	return 1;
 
 starved:
Signature · 1 line
2.29.2
#1 lore raw
Quoted 13 lines · On 20/01/2021 18:45, [email protected] wrote:

From: Martin Wilck <[email protected]>

Donald: please give this patch a try.

Commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq") contained this hunk:

(quoted diff) +1 -2
-       busy = atomic_inc_return(&shost->host_busy) - 1;
         if (atomic_read(&shost->host_blocked) > 0) {
-               if (busy)
+               if (scsi_host_busy(shost) > 0)
                         goto starved;
Quoted 7 lines

The previous code would increase the busy count before checking host_blocked. With 6eb045e092ef, the busy count would be increased (by setting the SCMD_STATE_INFLIGHT bit) after the if clause for host_blocked above.

Users have reported a regression with the smartpqi driver [1] which has been shown to be caused by this commit [2].

Please note these, where potential issue was discussed before:
https://lore.kernel.org/linux-scsi/[email protected]/
https://lore.kernel.org/linux-scsi/20200714104437.GB602708@T590/

Quoted 4 lines

It seems that by moving the increase of the busy counter further down, it could happen that the can_queue limit of the controller could be exceeded if several CPUs were executing this code in parallel on different queues.

Quoted 8 lines

This patch attempts to fix it by moving setting the SCMD_STATE_INFLIGHT before the host_blocked test again. It also inserts barriers to make sure scsi_host_busy() on once CPU will notice the increase of the count from another.

[1]: https://marc.info/?l=linux-scsi&m=160271263114829&w=2
[2]: https://marc.info/?l=linux-scsi&m=161116163722099&w=2

Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")

For failing test, it would be good to know:
- host .can_queue
- host .nr_hw_queues
- number of LUNs in test and their queue depth

All can be got from lsscsi, apart from nr_hw_queues, which can be got from scsi_host sysfs for kernel >= 5.10

Cheers,
John

Quoted 54 lines
Cc Ming Lei <[email protected]>
Cc Don Brace <[email protected]>
Cc Kevin Barnett <[email protected]>
Cc Donald Buczek <[email protected]>
Cc John Garry <[email protected]>
Cc Paul Menzel <[email protected]>
---
  drivers/scsi/hosts.c    | 2 ++
  drivers/scsi/scsi_lib.c | 8 +++++---
  2 files changed, 7 insertions(+), 3 deletions(-)
drivers/scsi/hosts.c +2 -0
index 2f162603876f..1c452a1c18fd 100644
@@ -564,6 +564,8 @@ static bool scsi_host_check_in_flight(struct request *rq, void *data,
  	int *count = data;
  	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
 
+	/* This pairs with set_bit() in scsi_host_queue_ready() */
+	smp_mb__before_atomic();
  	if (test_bit(SCMD_STATE_INFLIGHT, &cmd->state))
  		(*count)++;
 
drivers/scsi/scsi_lib.c +5 -3
index b3f14f05340a..0a9a36c349ee 100644
@@ -1353,8 +1353,12 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
  	if (scsi_host_in_recovery(shost))
  		return 0;
 
+	set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
+	/* This pairs with test_bit() in scsi_host_check_in_flight() */
+	smp_mb__after_atomic();
+
  	if (atomic_read(&shost->host_blocked) > 0) {
-		if (scsi_host_busy(shost) > 0)
+		if (scsi_host_busy(shost) > 1)
  			goto starved;
 
  		/*
@@ -1379,8 +1383,6 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
  		spin_unlock_irq(shost->host_lock);
  	}
 
-	__set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
-
  	return 1;
 
  starved:
#2 lore raw
Quoted 37 lines · John Garry
Quoted 13 lines · On 20/01/2021 18:45, [email protected] wrote:

From: Martin Wilck <[email protected]>

Donald: please give this patch a try.

Commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq") contained this hunk:

-       busy = atomic_inc_return(&shost->host_busy) - 1; if (atomic_read(&shost->host_blocked) > 0) { -               if (busy)
+               if (scsi_host_busy(shost) > 0) goto starved;

Quoted 7 lines

The previous code would increase the busy count before checking host_blocked. With 6eb045e092ef, the busy count would be increased (by setting the SCMD_STATE_INFLIGHT bit) after the if clause for host_blocked above.

Users have reported a regression with the smartpqi driver [1] which has been shown to be caused by this commit [2].

Please note these, where potential issue was discussed before:
https://lore.kernel.org/linux-scsi/[email protected]/
https://lore.kernel.org/linux-scsi/20200714104437.GB602708@T590/

Quoted 11 lines

It seems that by moving the increase of the busy counter further down, it could happen that the can_queue limit of the controller could be exceeded if several CPUs were executing this code in parallel on different queues.

This patch attempts to fix it by moving setting the SCMD_STATE_INFLIGHT before the host_blocked test again. It also inserts barriers to make sure scsi_host_busy() on once CPU will notice the increase of the count from another.

[1]: https://marc.info/?l=linux-scsi&m=160271263114829&w=2
[2]: https://marc.info/?l=linux-scsi&m=161116163722099&w=2

Quoted 6 lines
Quoted 1 line
Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")

For failing test, it would be good to know:
- host .can_queue
- host .nr_hw_queues
- number of LUNs in test and their queue dept>> All can be got from lsscsi, apart from nr_hw_queues, which can be got from scsi_host sysfs for kernel >= 5.10

The last test was on 6eb045e092ef + Martins experimental patch, which technically is 5.4.0-rc1

I'm not 100% sure about which data you need and where to find nr_hw_queues exposed. Please ask, if you need more data.

+ lsscsi

[0:0:0:0]    disk    ATA      ST500NM0011      PA09  /dev/sda
[0:0:32:0]   enclosu DP       BP13G+           2.23  -
[12:0:0:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdb
[12:0:1:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdc
[12:0:2:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdd
[12:0:3:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sde
[12:0:4:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdf
[12:0:5:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdg
[12:0:6:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdh
[12:0:7:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdi
[12:0:8:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdj
[12:0:9:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdk
[12:0:10:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdl
[12:0:11:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdm
[12:0:12:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdn
[12:0:13:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdo
[12:0:14:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdp
[12:0:15:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdq
[12:0:16:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdr
[12:0:17:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sds
[12:0:18:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdt
[12:0:19:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdu
[12:0:20:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdv
[12:0:21:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdw
[12:0:22:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdx
[12:0:23:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdy
[12:0:24:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdz
[12:0:25:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaa
[12:0:26:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdab
[12:0:27:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdac
[12:0:28:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdad
[12:0:29:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdae
[12:0:30:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaf
[12:0:31:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdag
[12:0:32:0]  enclosu AIC 12G  3U16SAS3swap     0c01  -
[12:0:33:0]  enclosu AIC 12G  3U16SAS3swap     0c01  -
[12:0:34:0]  enclosu Adaptec  Smart Adapter    3.21  -
[12:2:0:0]   storage Adaptec  1100-8e          3.21  -

+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:2:0:0
[12:2:0:0] storage Adaptec 1100-8e 3.21 -
device_blocked=0
iocounterbits=32
iodone_cnt=0x1e
ioerr_cnt=0x0
iorequest_cnt=0x1e
queue_depth=1013
queue_type=simple
scsi_level=6
state=running
timeout=30
type=12
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:34:0
[12:0:34:0] enclosu Adaptec Smart Adapter 3.21 -
device_blocked=0
iocounterbits=32
iodone_cnt=0x12
ioerr_cnt=0x0
iorequest_cnt=0x12
queue_depth=1
queue_type=none
scsi_level=6
state=running
timeout=30
type=13
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:33:0
[12:0:33:0] enclosu AIC 12G 3U16SAS3swap 0c01 -
device_blocked=0
iocounterbits=32
iodone_cnt=0x12
ioerr_cnt=0x0
iorequest_cnt=0x12
queue_depth=1
queue_type=simple
scsi_level=6
state=running
timeout=30
type=13
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:31:0
[12:0:31:0] disk SEAGATE ST8000NM001A E001 /dev/sdag
device_blocked=0
iocounterbits=32
iodone_cnt=0x19b94
ioerr_cnt=0x0
iorequest_cnt=0x19bba
queue_depth=64
queue_type=simple
scsi_level=8
state=running
timeout=30
type=0
+ cat /sys/bus/scsi/devices/host12/scsi_host/host12/can_queue 1013

Best
Donald

Quoted 53 lines

Cheers,
John

Quoted 48 lines
Cc Ming Lei <[email protected]>
Cc Don Brace <[email protected]>
Cc Kevin Barnett <[email protected]>
Cc Donald Buczek <[email protected]>
Cc John Garry <[email protected]>
Cc Paul Menzel <[email protected]>

---
  drivers/scsi/hosts.c    | 2 ++
  drivers/scsi/scsi_lib.c | 8 +++++---

  2 files changed, 7 insertions(+), 3 deletions(-)
drivers/scsi/hosts.c +2 -0
index 2f162603876f..1c452a1c18fd 100644
@@ -564,6 +564,8 @@ static bool scsi_host_check_in_flight(struct request *rq, void *data,
       int *count = data;
       struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
+    /* This pairs with set_bit() in scsi_host_queue_ready() */
+    smp_mb__before_atomic();
       if (test_bit(SCMD_STATE_INFLIGHT, &cmd->state))
           (*count)++;
drivers/scsi/scsi_lib.c +5 -3
index b3f14f05340a..0a9a36c349ee 100644
@@ -1353,8 +1353,12 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
       if (scsi_host_in_recovery(shost))
           return 0;
+    set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
+    /* This pairs with test_bit() in scsi_host_check_in_flight() */
+    smp_mb__after_atomic();
+
       if (atomic_read(&shost->host_blocked) > 0) {
-        if (scsi_host_busy(shost) > 0)
+        if (scsi_host_busy(shost) > 1)
               goto starved;
           /*
@@ -1379,8 +1383,6 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
           spin_unlock_irq(shost->host_lock);
       }
-    __set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
-
       return 1;

  starved:

Signature · 3 lines
Donald Buczek
[email protected]
Tel: +49 30 8413 1433
#3 lore raw
Quoted 7 lines · Donald Buczek
Quoted 7 lines
Quoted 7 lines

pts to fix it by moving setting the SCMD_STATE_INFLIGHT before
the host_blocked test again. It also inserts barriers to make sure scsi_host_busy() on once CPU will notice the increase of the count from another.

[1]:https://marc.info/?l=linux-scsi&m=160271263114829&w=2 [2]:https://marc.info/?l=linux-scsi&m=161116163722099&w=2

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")

For failing test, it would be good to know:
- host .can_queue
- host .nr_hw_queues
- number of LUNs in test and their queue dept>> All can be got from lsscsi, apart from nr_hw_queues, which can be got from scsi_host sysfs for kernel >= 5.10

The last test was on 6eb045e092ef + Martins experimental patch, which technically is 5.4.0-rc1

I'm not 100% sure about which data you need and where to find nr_hw_queues exposed.

nr_hw_queues is not available on 5.4 kernel via sysfs

it's prob same as count of CPUs in the system

or you can check number of hctxX folders in /sys/kernel/debug/block/sdX (need to be root, and debugfs enabled)

Please ask, if you need more data.

Quoted 40 lines

+ lsscsi

[0:0:0:0]    disk    ATA      ST500NM0011      PA09  /dev/sda
[0:0:32:0]   enclosu DP       BP13G+           2.23  -
[12:0:0:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdb
[12:0:1:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdc
[12:0:2:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdd
[12:0:3:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sde
[12:0:4:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdf
[12:0:5:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdg
[12:0:6:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdh
[12:0:7:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdi
[12:0:8:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdj
[12:0:9:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdk
[12:0:10:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdl
[12:0:11:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdm
[12:0:12:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdn
[12:0:13:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdo
[12:0:14:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdp
[12:0:15:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdq
[12:0:16:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdr
[12:0:17:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sds
[12:0:18:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdt
[12:0:19:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdu
[12:0:20:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdv
[12:0:21:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdw
[12:0:22:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdx
[12:0:23:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdy
[12:0:24:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdz
[12:0:25:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaa
[12:0:26:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdab
[12:0:27:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdac
[12:0:28:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdad
[12:0:29:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdae
[12:0:30:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaf
[12:0:31:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdag
[12:0:32:0]  enclosu AIC 12G  3U16SAS3swap     0c01  -
[12:0:33:0]  enclosu AIC 12G  3U16SAS3swap     0c01  -
[12:0:34:0]  enclosu Adaptec  Smart Adapter    3.21  -
[12:2:0:0]   storage Adaptec  1100-8e          3.21  -
Quoted 1 line

+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0

I figure sdev queue depth is 64 for all disks, like /dev/sdag, below.

lsscsi -l would give that

But, as said here:
https://lore.kernel.org/linux-scsi/20200714104437.GB602708@T590/

Looks like block layer can send too many commands to SCSI host if trying to achieve highest datarates with all those 32x disks. Before 6eb045e092ef, that would be avoided. That's how I see it.

Quoted 55 lines

+ lsscsi -L 12:2:0:0
[12:2:0:0] storage Adaptec 1100-8e 3.21 -

    device_blocked=0
    iocounterbits=32
    iodone_cnt=0x1e
    ioerr_cnt=0x0
    iorequest_cnt=0x1e
    queue_depth=1013
    queue_type=simple
    scsi_level=6
    state=running
    timeout=30
    type=12
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:34:0
[12:0:34:0]  enclosu Adaptec  Smart Adapter    3.21  -
    device_blocked=0
    iocounterbits=32
    iodone_cnt=0x12
    ioerr_cnt=0x0
    iorequest_cnt=0x12
    queue_depth=1
    queue_type=none
    scsi_level=6
    state=running
    timeout=30
    type=13
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:33:0
[12:0:33:0]  enclosu AIC 12G  3U16SAS3swap     0c01  -
    device_blocked=0
    iocounterbits=32
    iodone_cnt=0x12
    ioerr_cnt=0x0
    iorequest_cnt=0x12
    queue_depth=1
    queue_type=simple
    scsi_level=6
    state=running
    timeout=30
    type=13
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:31:0
[12:0:31:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdag
    device_blocked=0
    iocounterbits=32
    iodone_cnt=0x19b94
    ioerr_cnt=0x0
    iorequest_cnt=0x19bba
    queue_depth=64
    queue_type=simple
    scsi_level=8
    state=running
    timeout=30
    type=0
Quoted 3 lines

+ cat /sys/bus/scsi/devices/host12/scsi_host/host12/can_queue 1013

Thanks,
John

#4 lore raw
Quoted 15 lines · John Garry
Quoted 14 lines · Donald Buczek
Quoted 4 lines
Quoted 4 lines

pts to fix it by moving setting the SCMD_STATE_INFLIGHT before
the host_blocked test again. It also inserts barriers to make sure scsi_host_busy() on once CPU will notice the increase of the count from another.

Quoted 8 lines
Quoted 4 lines

[1]:https://marc.info/?l=linux-scsi&m=160271263114829&w=2 [2]:https://marc.info/?l=linux-scsi&m=161116163722099&w=2

Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")

For failing test, it would be good to know:
- host .can_queue
- host .nr_hw_queues
- number of LUNs in test and their queue dept>> All can be got from lsscsi, apart from nr_hw_queues, which can be got from scsi_host sysfs for kernel >= 5.10

The last test was on 6eb045e092ef + Martins experimental patch, which technically is 5.4.0-rc1

Quoted 7 lines
Quoted 1 line

I'm not 100% sure about which data you need and where to find  nr_hw_queues exposed.

nr_hw_queues is not available on 5.4 kernel via sysfs

it's prob same as count of CPUs in the system

or you can check number of hctxX folders in /sys/kernel/debug/block/sdX (need to be root, and debugfs enabled)

root@deadbird:~# ls -d /sys/kernel/debug/block/sdz/hc*
/sys/kernel/debug/block/sdz/hctx0   /sys/kernel/debug/block/sdz/hctx16	/sys/kernel/debug/block/sdz/hctx23  /sys/kernel/debug/block/sdz/hctx30	/sys/kernel/debug/block/sdz/hctx38
/sys/kernel/debug/block/sdz/hctx1   /sys/kernel/debug/block/sdz/hctx17	/sys/kernel/debug/block/sdz/hctx24  /sys/kernel/debug/block/sdz/hctx31	/sys/kernel/debug/block/sdz/hctx39
/sys/kernel/debug/block/sdz/hctx10  /sys/kernel/debug/block/sdz/hctx18	/sys/kernel/debug/block/sdz/hctx25  /sys/kernel/debug/block/sdz/hctx32	/sys/kernel/debug/block/sdz/hctx4
/sys/kernel/debug/block/sdz/hctx11  /sys/kernel/debug/block/sdz/hctx19	/sys/kernel/debug/block/sdz/hctx26  /sys/kernel/debug/block/sdz/hctx33	/sys/kernel/debug/block/sdz/hctx5
/sys/kernel/debug/block/sdz/hctx12  /sys/kernel/debug/block/sdz/hctx2	/sys/kernel/debug/block/sdz/hctx27  /sys/kernel/debug/block/sdz/hctx34	/sys/kernel/debug/block/sdz/hctx6
/sys/kernel/debug/block/sdz/hctx13  /sys/kernel/debug/block/sdz/hctx20	/sys/kernel/debug/block/sdz/hctx28  /sys/kernel/debug/block/sdz/hctx35	/sys/kernel/debug/block/sdz/hctx7
/sys/kernel/debug/block/sdz/hctx14  /sys/kernel/debug/block/sdz/hctx21	/sys/kernel/debug/block/sdz/hctx29  /sys/kernel/debug/block/sdz/hctx36	/sys/kernel/debug/block/sdz/hctx8
/sys/kernel/debug/block/sdz/hctx15  /sys/kernel/debug/block/sdz/hctx22	/sys/kernel/debug/block/sdz/hctx3   /sys/kernel/debug/block/sdz/hctx37	/sys/kernel/debug/block/sdz/hctx9
root@deadbird:~# ls -d /sys/kernel/debug/block/sdz/hc*|wc
     40      40    1390
root@deadbird:~# nproc
40
Quoted 43 lines

Please ask, if you need more data.

Quoted 33 lines

+ lsscsi
[0:0:0:0]    disk    ATA      ST500NM0011      PA09  /dev/sda [0:0:32:0]   enclosu DP       BP13G+           2.23  - [12:0:0:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdb [12:0:1:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdc [12:0:2:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdd [12:0:3:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sde [12:0:4:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdf [12:0:5:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdg [12:0:6:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdh [12:0:7:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdi [12:0:8:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdj [12:0:9:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdk [12:0:10:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdl [12:0:11:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdm [12:0:12:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdn [12:0:13:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdo [12:0:14:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdp [12:0:15:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdq [12:0:16:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdr [12:0:17:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sds [12:0:18:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdt [12:0:19:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdu [12:0:20:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdv [12:0:21:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdw [12:0:22:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdx [12:0:23:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdy [12:0:24:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdz [12:0:25:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaa [12:0:26:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdab [12:0:27:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdac [12:0:28:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdad

Quoted 8 lines

[12:0:29:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdae [12:0:30:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaf [12:0:31:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdag [12:0:32:0]  enclosu AIC 12G  3U16SAS3swap     0c01  - [12:0:33:0]  enclosu AIC 12G  3U16SAS3swap     0c01  - [12:0:34:0]  enclosu Adaptec  Smart Adapter    3.21  - [12:2:0:0]   storage Adaptec  1100-8e          3.21  -
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0

Quoted 1 line

I figure sdev queue depth is 64 for all disks, like /dev/sdag, below.

Yes, I send an example (one of two enclosures, 1 of 32 disks) but verified, that they are the same.

Best
Donald

Quoted 70 lines

lsscsi -l would give that

But, as said here:
https://lore.kernel.org/linux-scsi/20200714104437.GB602708@T590/

Looks like block layer can send too many commands to SCSI host if trying to achieve highest datarates with all those 32x disks. Before 6eb045e092ef, that would be avoided. That's how I see it.

Quoted 49 lines

+ lsscsi -L 12:2:0:0
[12:2:0:0]   storage Adaptec  1100-8e          3.21  - device_blocked=0
    iocounterbits=32
    iodone_cnt=0x1e
    ioerr_cnt=0x0
    iorequest_cnt=0x1e
    queue_depth=1013
    queue_type=simple
    scsi_level=6
    state=running
    timeout=30
    type=12
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:34:0
[12:0:34:0]  enclosu Adaptec  Smart Adapter    3.21  - device_blocked=0
    iocounterbits=32
    iodone_cnt=0x12
    ioerr_cnt=0x0
    iorequest_cnt=0x12
    queue_depth=1
    queue_type=none
    scsi_level=6
    state=running
    timeout=30
    type=13
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:33:0
[12:0:33:0]  enclosu AIC 12G  3U16SAS3swap     0c01  - device_blocked=0
    iocounterbits=32
    iodone_cnt=0x12
    ioerr_cnt=0x0
    iorequest_cnt=0x12
    queue_depth=1
    queue_type=simple
    scsi_level=6
    state=running
    timeout=30
    type=13
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0
+ lsscsi -L 12:0:31:0
[12:0:31:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdag device_blocked=0
    iocounterbits=32
    iodone_cnt=0x19b94
    ioerr_cnt=0x0
    iorequest_cnt=0x19bba

Quoted 9 lines

    queue_depth=64
    queue_type=simple
    scsi_level=8
    state=running
    timeout=30
    type=0
+ cat /sys/bus/scsi/devices/host12/scsi_host/host12/can_queue 1013

Thanks,
John

#5 lore raw
Quoted 66 lines
Quoted 7 lines
Quoted 1 line

I'm not 100% sure about which data you need and where to find nr_hw_queues exposed.

nr_hw_queues is not available on 5.4 kernel via sysfs

it's prob same as count of CPUs in the system

or you can check number of hctxX folders in /sys/kernel/debug/block/sdX (need to be root, and debugfs enabled)

root@deadbird:~# ls -d /sys/kernel/debug/block/sdz/hc*
/sys/kernel/debug/block/sdz/hctx0 /sys/kernel/debug/block/sdz/hctx16 /sys/kernel/debug/block/sdz/hctx23 /sys/kernel/debug/block/sdz/hctx30    /sys/kernel/debug/block/sdz/hctx38
/sys/kernel/debug/block/sdz/hctx1 /sys/kernel/debug/block/sdz/hctx17 /sys/kernel/debug/block/sdz/hctx24 /sys/kernel/debug/block/sdz/hctx31    /sys/kernel/debug/block/sdz/hctx39
/sys/kernel/debug/block/sdz/hctx10 /sys/kernel/debug/block/sdz/hctx18 /sys/kernel/debug/block/sdz/hctx25 /sys/kernel/debug/block/sdz/hctx32    /sys/kernel/debug/block/sdz/hctx4
/sys/kernel/debug/block/sdz/hctx11 /sys/kernel/debug/block/sdz/hctx19 /sys/kernel/debug/block/sdz/hctx26 /sys/kernel/debug/block/sdz/hctx33    /sys/kernel/debug/block/sdz/hctx5
/sys/kernel/debug/block/sdz/hctx12 /sys/kernel/debug/block/sdz/hctx2    /sys/kernel/debug/block/sdz/hctx27 /sys/kernel/debug/block/sdz/hctx34    /sys/kernel/debug/block/sdz/hctx6
/sys/kernel/debug/block/sdz/hctx13 /sys/kernel/debug/block/sdz/hctx20 /sys/kernel/debug/block/sdz/hctx28 /sys/kernel/debug/block/sdz/hctx35    /sys/kernel/debug/block/sdz/hctx7
/sys/kernel/debug/block/sdz/hctx14 /sys/kernel/debug/block/sdz/hctx21 /sys/kernel/debug/block/sdz/hctx29 /sys/kernel/debug/block/sdz/hctx36    /sys/kernel/debug/block/sdz/hctx8
/sys/kernel/debug/block/sdz/hctx15 /sys/kernel/debug/block/sdz/hctx22 /sys/kernel/debug/block/sdz/hctx3 /sys/kernel/debug/block/sdz/hctx37    /sys/kernel/debug/block/sdz/hctx9
root@deadbird:~# ls -d /sys/kernel/debug/block/sdz/hc*|wc
      40      40    1390
root@deadbird:~# nproc
40
Quoted 43 lines

Please ask, if you need more data.

Quoted 41 lines

+ lsscsi
[0:0:0:0]    disk    ATA      ST500NM0011      PA09  /dev/sda [0:0:32:0]   enclosu DP       BP13G+           2.23  - [12:0:0:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdb [12:0:1:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdc [12:0:2:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdd [12:0:3:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sde [12:0:4:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdf [12:0:5:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdg [12:0:6:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdh [12:0:7:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdi [12:0:8:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdj [12:0:9:0]   disk    SEAGATE  ST8000NM001A     E002  /dev/sdk [12:0:10:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdl [12:0:11:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdm [12:0:12:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdn [12:0:13:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdo [12:0:14:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdp [12:0:15:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdq [12:0:16:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdr [12:0:17:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sds [12:0:18:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdt [12:0:19:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdu [12:0:20:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdv [12:0:21:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdw [12:0:22:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdx [12:0:23:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdy [12:0:24:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdz [12:0:25:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaa [12:0:26:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdab [12:0:27:0]  disk    SEAGATE  ST8000NM001A     E002  /dev/sdac [12:0:28:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdad [12:0:29:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdae [12:0:30:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdaf [12:0:31:0]  disk    SEAGATE  ST8000NM001A     E001  /dev/sdag [12:0:32:0]  enclosu AIC 12G  3U16SAS3swap     0c01  - [12:0:33:0]  enclosu AIC 12G  3U16SAS3swap     0c01  - [12:0:34:0]  enclosu Adaptec  Smart Adapter    3.21  - [12:2:0:0]   storage Adaptec  1100-8e          3.21  -
+ for i in 12:2:0:0 12:0:34:0 12:0:33:0 12:0:31:0

Quoted 3 lines
Quoted 1 line

I figure sdev queue depth is 64 for all disks, like /dev/sdag, below.

Yes, I send an example (one of two enclosures, 1 of 32 disks) but verified, that they are the same.

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Thanks,
John

#6 lore raw
Quoted 8 lines · John Garry
Quoted 1 line
Quoted 1 line

Confirmed my suspicions - it looks like the host is sent more commands
than it can handle. We would need many disks to see this issue though,
which you have.

Quoted 4 lines

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I
suppose it could be simply fixed by setting .host_tagset in scsi host template there.

If it's really just that, it should be easy enough to verify.

@Donald, you'd need to test with a 5.10 kernel, and after reproducing the issue, add

.host_tagset = 1,

to the definition of pqi_driver_template in
drivers/scsi/smartpqi/smartpqi_init.c.

You don't need a patch to test that, I believe. Would you able to do this test?

Regards,
Martin

#7 lore raw
Quoted 19 lines · Martin Wilck
Quoted 8 lines · John Garry
Quoted 1 line
Quoted 1 line

Confirmed my suspicions - it looks like the host is sent more commands
than it can handle. We would need many disks to see this issue though,
which you have.

Quoted 4 lines

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I
suppose it could be simply fixed by setting .host_tagset in scsi host template there.

If it's really just that, it should be easy enough to verify.

@Donald, you'd need to test with a 5.10 kernel, and after reproducing the issue, add

Quoted 7 lines

.host_tagset = 1,

to the definition of pqi_driver_template in
drivers/scsi/smartpqi/smartpqi_init.c.

You don't need a patch to test that, I believe. Would you able to do this test?

Sorry, I had overlooked this request. I reviewed this thread now, because I want to switch our production systems to 5.10 LTS.

I could reproduce the problem with Linux 5.10.22. When setting host_tagset = 1, the problem disappeared. Additionally, we have 5.10.22 with the patch running on two previously affected production systems for over 24 hours now. Statistics suggest, that these systems were very likely to trigger the problem in that time frame if the patch didn't work.

So I think this is a working fix which should go to 5.10 stable.

Best
Donald

drivers/scsi/smartpqi/smartpqi_init.c +1 -0
index 9d0229656681f..be429a7cb1512 100644
@@ -6571,6 +6571,7 @@ static struct scsi_host_template pqi_driver_template = {
        .map_queues = pqi_map_queues,
        .sdev_attrs = pqi_sdev_attrs,
        .shost_attrs = pqi_shost_attrs,
+       .host_tagset = 1,
 };
 
 static int pqi_register_scsi(struct pqi_ctrl_info *ctrl_info)
#8 lore raw
In reply to John Garry

-----Original Message-----
From: John Garry [mailto:[email protected]]

Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

I'm not 100% sure about which data you need and where to find nr_hw_queues exposed.

nr_hw_queues is not available on 5.4 kernel via sysfs

it's prob same as count of CPUs in the system

Quoted 7 lines
Quoted 4 lines

or you can check number of hctxX folders in /sys/kernel/debug/block/sdX (need to be root, and debugfs enabled)

I figure sdev queue depth is 64 for all disks, like /dev/sdag, below.

Yes, I send an example (one of two enclosures, 1 of 32 disks) but verified, that they are the same.

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Thanks,
John

John, I tried this for smartpqi, so far, setting host_tagset = 1 seems to be working. The issue normally repros very quickly.

I want to run a few more tests before calling this a good fix. Thanks for your suggestion,
Don Brace

----
[root@cleddyf ~]# lsscsi

[0:0:0:0]    disk    Generic- SD/MMC CRW       1.00  /dev/sdc 
[1:0:0:0]    disk    ASMT     2115             0     /dev/sda 
[2:0:0:0]    disk    ASMT     2115             0     /dev/sdb 
[3:0:0:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdd 
[3:0:1:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sde 
[3:0:2:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdf 
[3:0:3:0]    disk    HP       EH0300FBQDD      HPD5  /dev/sdg 
[3:0:4:0]    disk    HP       EG0900FDJYR      HPD4  /dev/sdh 
[3:0:5:0]    disk    HP       EG0300FCVBF      HPD9  /dev/sdi 
[3:0:6:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdj 
[3:0:7:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdk 
[3:0:8:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdl 
[3:0:9:0]    disk    HP       MO0200FBRWB      HPD9  /dev/sdm 
[3:0:10:0]   disk    HP       MM0500FBFVQ      HPD8  /dev/sdn 
[3:0:11:0]   disk    ATA      MM0500GBKAK      HPGC  /dev/sdo 
[3:0:12:0]   disk    HP       EG0900FBVFQ      HPDC  /dev/sdp 
[3:0:13:0]   disk    HP       VO006400JWZJT    HP00  /dev/sdq 
[3:0:14:0]   disk    HP       VO015360JWZJN    HP00  /dev/sdr 
[3:0:15:0]   enclosu HP       D3700            5.04  -        
[3:0:16:0]   enclosu HP       D3700            5.04  -        
[3:0:17:0]   enclosu HPE      Smart Adapter    3.00  -        
[3:1:0:0]    disk    HPE      LOGICAL VOLUME   3.00  /dev/sds 
[3:2:0:0]    storage HPE      P408e-p SR Gen10 3.00  -      

---
[global]
ioengine=libaio
; rw=randwrite
; percentage_random=40
rw=write
size=100g
bs=4k
direct=1
ramp_time=15
; filename=/mnt/fio_test
; cpus_allowed=0-27
iodepth=1024

[/dev/sdd]
[/dev/sde]
[/dev/sdf]
[/dev/sdg]
[/dev/sdh]
[/dev/sdi]
[/dev/sdj]
[/dev/sdk]
[/dev/sdl]
[/dev/sdm]
[/dev/sdn]
[/dev/sdo]
[/dev/sdp]
[/dev/sdq]
[/dev/sdr]

#9 lore raw
In reply to John Garry

-----Original Message-----
From: John Garry [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

Thanks,
Don Brace

#10 lore raw
Quoted 14 lines · On Tue, 2021-02-02 at 20:04 +0000, [email protected] wrote:

-----Original Message-----
From: John Garry [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Quoted 8 lines

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

How much performance would that cost you?

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Regards
Martin

#11 lore raw
Quoted 30 lines · Martin Wilck
Quoted 14 lines · On Tue, 2021-02-02 at 20:04 +0000, [email protected] wrote:

-----Original Message-----
From: John Garry [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Quoted 8 lines

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

Quoted 4 lines

How much performance would that cost you?

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Aren't they (distros) the only issue? As I mentioned above, for 5.10 mainline stable, I think it's reasonable to backport a patch to set .host_tagset for the driver.

Thanks,
John

#12 lore raw

Dear Linux folks,

Am 03.02.21 um 09:49 schrieb John Garry:

Quoted 30 lines · Martin Wilck
Quoted 29 lines · On Tue, 2021-02-02 at 20:04 +0000, [email protected] wrote:
Quoted 9 lines

-----Original Message-----
From: John Garry [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

Quoted 12 lines

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Thanks,
John

Signature · 5 lines
Don: Even though this works for current kernels, what would chances
of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would
correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

Quoted 6 lines
Quoted 4 lines

How much performance would that cost you?

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Aren't they (distros) the only issue? As I mentioned above, for 5.10 mainline stable, I think it's reasonable to backport a patch to set .host_tagset for the driver.

Indeed. As per the Linux kernel Web site [1]: 5.5 and 5.9 are not maintained anymore (EOL) upstream. So, if distributions decided to go with another Linux kernel release, it’s their job to backport things. If the commit message is well written, and contains the Fixes tag, their tooling should be able to pick it up.

Kind regards,

Paul

[1]: https://www.kernel.org/

#13 lore raw

-----Original Message-----
From: Paul Menzel [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Quoted 23 lines
Quoted 23 lines
Quoted 8 lines

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Quoted 8 lines

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

Quoted 8 lines
Quoted 4 lines

How much performance would that cost you?

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Aren't they (distros) the only issue? As I mentioned above, for 5.10 mainline stable, I think it's reasonable to backport a patch to set .host_tagset for the driver.

Indeed. As per the Linux kernel Web site [1]: 5.5 and 5.9 are not maintained anymore (EOL) upstream. So, if distributions decided to go with another Linux kernel release, it’s their job to backport things. If the commit message is well written, and contains the Fixes tag, their tooling should be able to pick it up.

Kind regards,

Paul

[1]: https://www.kernel.org/

I remove patch smartpqi-fix_host_qdepth_limit and replaced it with a patch that sets host_tagset = 1 in function pqi_register_scsi

Thanks to all for your hard work.
Don

#14 lore raw
In reply to Martin Wilck

-----Original Message-----
From: Martin Wilck [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Quoted 10 lines

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Quoted 8 lines

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

How much performance would that cost you?

Don: For my HBA disk tests...

Dividing can_queue / nr_hw_queues is about a 40% drop. ~380K - 400K IOPS
Setting nr_hw_queues = 1 results in a 1.5 X gain in performance. ~980K IOPS
Setting host_tagset = 1
~640K IOPS

So, it seem that setting nr_hw_queues = 1 results in the best performance.

Is this expected? Would this also be true for the future?

Thanks,
Don Brace

Below is my setup.
---

[3:0:0:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdd 
[3:0:1:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sde 
[3:0:2:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdf 
[3:0:3:0]    disk    HP       EH0300FBQDD      HPD5  /dev/sdg 
[3:0:4:0]    disk    HP       EG0900FDJYR      HPD4  /dev/sdh 
[3:0:5:0]    disk    HP       EG0300FCVBF      HPD9  /dev/sdi 
[3:0:6:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdj 
[3:0:7:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdk 
[3:0:8:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdl 
[3:0:9:0]    disk    HP       MO0200FBRWB      HPD9  /dev/sdm 
[3:0:10:0]   disk    HP       MM0500FBFVQ      HPD8  /dev/sdn 
[3:0:11:0]   disk    ATA      MM0500GBKAK      HPGC  /dev/sdo 
[3:0:12:0]   disk    HP       EG0900FBVFQ      HPDC  /dev/sdp 
[3:0:13:0]   disk    HP       VO006400JWZJT    HP00  /dev/sdq 
[3:0:14:0]   disk    HP       VO015360JWZJN    HP00  /dev/sdr 
[3:0:15:0]   enclosu HP       D3700            5.04  -        
[3:0:16:0]   enclosu HP       D3700            5.04  -        
[3:0:17:0]   enclosu HPE      Smart Adapter    3.00  -        
[3:1:0:0]    disk    HPE      LOGICAL VOLUME   3.00  /dev/sds 
[3:2:0:0]    storage HPE      P408e-p SR Gen10 3.00  -        

-----
[global]
ioengine=libaio
; rw=randwrite
; percentage_random=40
rw=write
size=100g
bs=4k
direct=1
ramp_time=15
; filename=/mnt/fio_test
; cpus_allowed=0-27
iodepth=4096

[/dev/sdd]
[/dev/sde]
[/dev/sdf]
[/dev/sdg]
[/dev/sdh]
[/dev/sdi]
[/dev/sdj]
[/dev/sdk]
[/dev/sdl]
[/dev/sdm]
[/dev/sdn]
[/dev/sdo]
[/dev/sdp]
[/dev/sdq]
[/dev/sdr]

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Regards
Martin

#15 lore raw
Quoted 11 lines · On 03/02/2021 15:56, [email protected] wrote:

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

How much performance would that cost you?

Don: For my HBA disk tests...

Dividing can_queue / nr_hw_queues is about a 40% drop. ~380K - 400K IOPS
Setting nr_hw_queues = 1 results in a 1.5 X gain in performance. ~980K IOPS

So do you just set shost.nr_hw_queues = 1, yet leave the rest of the driver as is?

Please note that when changing from nr_hw_queues many -> 1, then the default IO scheduler changes from none -> mq-deadline, but I would hope that would not make such a big difference.

Quoted 1 line

Setting host_tagset = 1

For this, v5.11-rc6 has a fix which may affect you (2569063c7140), so please include it

Quoted 5 lines

~640K IOPS

So, it seem that setting nr_hw_queues = 1 results in the best performance.

Is this expected? Would this also be true for the future?

Not expected by me

Quoted 26 lines

Thanks,
Don Brace

Below is my setup.
---

[3:0:0:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdd
[3:0:1:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sde
[3:0:2:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdf
[3:0:3:0]    disk    HP       EH0300FBQDD      HPD5  /dev/sdg
[3:0:4:0]    disk    HP       EG0900FDJYR      HPD4  /dev/sdh
[3:0:5:0]    disk    HP       EG0300FCVBF      HPD9  /dev/sdi
[3:0:6:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdj
[3:0:7:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdk
[3:0:8:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdl
[3:0:9:0]    disk    HP       MO0200FBRWB      HPD9  /dev/sdm
[3:0:10:0]   disk    HP       MM0500FBFVQ      HPD8  /dev/sdn
[3:0:11:0]   disk    ATA      MM0500GBKAK      HPGC  /dev/sdo
[3:0:12:0]   disk    HP       EG0900FBVFQ      HPDC  /dev/sdp
[3:0:13:0]   disk    HP       VO006400JWZJT    HP00  /dev/sdq
[3:0:14:0]   disk    HP       VO015360JWZJN    HP00  /dev/sdr
[3:0:15:0]   enclosu HP       D3700            5.04  -
[3:0:16:0]   enclosu HP       D3700            5.04  -
[3:0:17:0]   enclosu HPE      Smart Adapter    3.00  -
[3:1:0:0]    disk    HPE      LOGICAL VOLUME   3.00  /dev/sds
[3:2:0:0]    storage HPE      P408e-p SR Gen10 3.00  -
Quoted 13 lines

-----
[global]
ioengine=libaio
; rw=randwrite
; percentage_random=40
rw=write
size=100g
bs=4k
direct=1
ramp_time=15
; filename=/mnt/fio_test
; cpus_allowed=0-27
iodepth=4096

I normally use iodepth circa 40 to 128, but then I normally just do rw=read for performance testing

Quoted 16 lines

[/dev/sdd]
[/dev/sde]
[/dev/sdf]
[/dev/sdg]
[/dev/sdh]
[/dev/sdi]
[/dev/sdj]
[/dev/sdk]
[/dev/sdl]
[/dev/sdm]
[/dev/sdn]
[/dev/sdo]
[/dev/sdp]
[/dev/sdq]
[/dev/sdr]

#16 lore raw

-----Original Message-----
From: John Garry [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe

Quoted 11 lines · On 03/02/2021 15:56, [email protected] wrote:

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

How much performance would that cost you?

Don: For my HBA disk tests...

Dividing can_queue / nr_hw_queues is about a 40% drop. ~380K - 400K IOPS
Setting nr_hw_queues = 1 results in a 1.5 X gain in performance. ~980K IOPS

So do you just set shost.nr_hw_queues = 1, yet leave the rest of the driver as is?

Please note that when changing from nr_hw_queues many -> 1, then the default IO scheduler changes from none -> mq-deadline, but I would hope that would not make such a big difference.

Quoted 1 line

Setting host_tagset = 1

For this, v5.11-rc6 has a fix which may affect you (2569063c7140), so please include it

Quoted 5 lines

~640K IOPS

So, it seem that setting nr_hw_queues = 1 results in the best performance.

Is this expected? Would this also be true for the future?

Not expected by me

Don: Ok, setting both host_tagset = 1 and nr_hw_queues = 1 yields the same better performance, about 940K IOPS.

Thanks,
Don Brace

Quoted 26 lines

Thanks,
Don Brace

Below is my setup.
---

[3:0:0:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdd
[3:0:1:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sde
[3:0:2:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdf
[3:0:3:0]    disk    HP       EH0300FBQDD      HPD5  /dev/sdg
[3:0:4:0]    disk    HP       EG0900FDJYR      HPD4  /dev/sdh
[3:0:5:0]    disk    HP       EG0300FCVBF      HPD9  /dev/sdi
[3:0:6:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdj
[3:0:7:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdk
[3:0:8:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdl
[3:0:9:0]    disk    HP       MO0200FBRWB      HPD9  /dev/sdm
[3:0:10:0]   disk    HP       MM0500FBFVQ      HPD8  /dev/sdn
[3:0:11:0]   disk    ATA      MM0500GBKAK      HPGC  /dev/sdo
[3:0:12:0]   disk    HP       EG0900FBVFQ      HPDC  /dev/sdp
[3:0:13:0]   disk    HP       VO006400JWZJT    HP00  /dev/sdq
[3:0:14:0]   disk    HP       VO015360JWZJN    HP00  /dev/sdr
[3:0:15:0]   enclosu HP       D3700            5.04  -
[3:0:16:0]   enclosu HP       D3700            5.04  -
[3:0:17:0]   enclosu HPE      Smart Adapter    3.00  -
[3:1:0:0]    disk    HPE      LOGICAL VOLUME   3.00  /dev/sds
[3:2:0:0]    storage HPE      P408e-p SR Gen10 3.00  -
Quoted 13 lines

-----
[global]
ioengine=libaio
; rw=randwrite
; percentage_random=40
rw=write
size=100g
bs=4k
direct=1
ramp_time=15
; filename=/mnt/fio_test
; cpus_allowed=0-27
iodepth=4096

I normally use iodepth circa 40 to 128, but then I normally just do rw=read for performance testing

Quoted 16 lines

[/dev/sdd]
[/dev/sde]
[/dev/sdf]
[/dev/sdg]
[/dev/sdh]
[/dev/sdi]
[/dev/sdj]
[/dev/sdk]
[/dev/sdl]
[/dev/sdm]
[/dev/sdn]
[/dev/sdo]
[/dev/sdp]
[/dev/sdq]
[/dev/sdr]

#17 lore raw
In reply to [email protected]

FYI we have exactly this issue on a machine here running CentOS 8.3 (kernel 4.18.0-240.1.1) (so presumably this happens in RHEL 8 too.)

Controller is MSCC / Adaptec 3154-8i16e driving 60 x 12TB HGST drives configured as five x twelve-drive raid-6, software striped using md, and formatted with xfs.

Test software writes to the array using multiple threads in parallel.

The smartpqi driver would report controller offline within ten minutes or so, with status code 0x6100c

Changed the driver to set 'nr_hw_queues = 1’ and then tested by filling the array with random files (which took a couple of days), which completed fine, so it looks like that one-line change fixes it.

Would, of course, be helpful if this was back-ported.

—
Roger

Quoted 113 lines · On 3 Feb 2021, at 15:56, [email protected] wrote:

-----Original Message-----
From: Martin Wilck [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Quoted 10 lines

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Quoted 8 lines

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

How much performance would that cost you?

Don: For my HBA disk tests...

Dividing can_queue / nr_hw_queues is about a 40% drop. ~380K - 400K IOPS
Setting nr_hw_queues = 1 results in a 1.5 X gain in performance. ~980K IOPS
Setting host_tagset = 1
~640K IOPS

So, it seem that setting nr_hw_queues = 1 results in the best performance.

Is this expected? Would this also be true for the future?

Thanks,
Don Brace

Below is my setup.
---

[3:0:0:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdd 
[3:0:1:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sde 
[3:0:2:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdf 
[3:0:3:0]    disk    HP       EH0300FBQDD      HPD5  /dev/sdg 
[3:0:4:0]    disk    HP       EG0900FDJYR      HPD4  /dev/sdh 
[3:0:5:0]    disk    HP       EG0300FCVBF      HPD9  /dev/sdi 
[3:0:6:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdj 
[3:0:7:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdk 
[3:0:8:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdl 
[3:0:9:0]    disk    HP       MO0200FBRWB      HPD9  /dev/sdm 
[3:0:10:0]   disk    HP       MM0500FBFVQ      HPD8  /dev/sdn 
[3:0:11:0]   disk    ATA      MM0500GBKAK      HPGC  /dev/sdo 
[3:0:12:0]   disk    HP       EG0900FBVFQ      HPDC  /dev/sdp 
[3:0:13:0]   disk    HP       VO006400JWZJT    HP00  /dev/sdq 
[3:0:14:0]   disk    HP       VO015360JWZJN    HP00  /dev/sdr 
[3:0:15:0]   enclosu HP       D3700            5.04  -        
[3:0:16:0]   enclosu HP       D3700            5.04  -        
[3:0:17:0]   enclosu HPE      Smart Adapter    3.00  -        
[3:1:0:0]    disk    HPE      LOGICAL VOLUME   3.00  /dev/sds 
[3:2:0:0]    storage HPE      P408e-p SR Gen10 3.00  -        

-----
[global]
ioengine=libaio
; rw=randwrite
; percentage_random=40
rw=write
size=100g
bs=4k
direct=1
ramp_time=15
; filename=/mnt/fio_test
; cpus_allowed=0-27
iodepth=4096

[/dev/sdd]
[/dev/sde]
[/dev/sdf]
[/dev/sdg]
[/dev/sdh]
[/dev/sdi]
[/dev/sdj]
[/dev/sdk]
[/dev/sdl]
[/dev/sdm]
[/dev/sdn]
[/dev/sdo]
[/dev/sdp]
[/dev/sdq]
[/dev/sdr]

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Regards
Martin

#18 lore raw
Quoted 10 lines · Roger Willcocks

FYI we have exactly this issue on a machine here running CentOS 8.3 (kernel 4.18.0-240.1.1) (so presumably this happens in RHEL 8 too.)

Controller is MSCC / Adaptec 3154-8i16e driving 60 x 12TB HGST drives configured as five x twelve-drive raid-6, software striped using md, and formatted with xfs.

Test software writes to the array using multiple threads in parallel.

The smartpqi driver would report controller offline within ten minutes or so, with status code 0x6100c

Changed the driver to set 'nr_hw_queues = 1’ and then tested by filling the array with random files (which took a couple of days), which completed fine, so it looks like that one-line change fixes it.

That just makes the driver single-queue.

As such, since the driver uses blk_mq_unique_tag_to_hwq(), only hw queue #0 will ever be used in the driver.

And then, since the driver still spreads MSI-X interrupt vectors over all CPUs [from pci_alloc_vectors(PCI_IRQ_AFFINITY)], if CPUs associated with HW queue #0 are offlined (probably just cpu0), there is no CPUs available to service queue #0 interrupt. That's what I think would happen, from a quick glance at the code.

Quoted 123 lines

Would, of course, be helpful if this was back-ported.

—
Roger

Quoted 99 lines · On 3 Feb 2021, at 15:56, [email protected] wrote:

-----Original Message-----
From: Martin Wilck [mailto:[email protected]]
Subject: Re: [PATCH] scsi: scsi_host_queue_ready: increase busy count early

Quoted 10 lines

Confirmed my suspicions - it looks like the host is sent more commands than it can handle. We would need many disks to see this issue though, which you have.

So for stable kernels, 6eb045e092ef is not in 5.4 . Next is 5.10, and I suppose it could be simply fixed by setting .host_tagset in scsi host template there.

Quoted 8 lines

Thanks,
John
--
Don: Even though this works for current kernels, what would chances of this getting back-ported to 5.9 or even further?

Otherwise the original patch smartpqi_fix_host_qdepth_limit would correct this issue for older kernels.

True. However this is 5.12 material, so we shouldn't be bothered by that here. For 5.5 up to 5.9, you need a workaround. But I'm unsure whether smartpqi_fix_host_qdepth_limit would be the solution. You could simply divide can_queue by nr_hw_queues, as suggested before, or even simpler, set nr_hw_queues = 1.

How much performance would that cost you?

Don: For my HBA disk tests...

Dividing can_queue / nr_hw_queues is about a 40% drop. ~380K - 400K IOPS
Setting nr_hw_queues = 1 results in a 1.5 X gain in performance. ~980K IOPS
Setting host_tagset = 1
~640K IOPS

So, it seem that setting nr_hw_queues = 1 results in the best performance.

Is this expected? Would this also be true for the future?

Thanks,
Don Brace

Below is my setup.
---

[3:0:0:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdd
[3:0:1:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sde
[3:0:2:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdf
[3:0:3:0]    disk    HP       EH0300FBQDD      HPD5  /dev/sdg
[3:0:4:0]    disk    HP       EG0900FDJYR      HPD4  /dev/sdh
[3:0:5:0]    disk    HP       EG0300FCVBF      HPD9  /dev/sdi
[3:0:6:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdj
[3:0:7:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdk
[3:0:8:0]    disk    HP       EG0900FBLSK      HPD7  /dev/sdl
[3:0:9:0]    disk    HP       MO0200FBRWB      HPD9  /dev/sdm
[3:0:10:0]   disk    HP       MM0500FBFVQ      HPD8  /dev/sdn
[3:0:11:0]   disk    ATA      MM0500GBKAK      HPGC  /dev/sdo
[3:0:12:0]   disk    HP       EG0900FBVFQ      HPDC  /dev/sdp
[3:0:13:0]   disk    HP       VO006400JWZJT    HP00  /dev/sdq
[3:0:14:0]   disk    HP       VO015360JWZJN    HP00  /dev/sdr
[3:0:15:0]   enclosu HP       D3700            5.04  -
[3:0:16:0]   enclosu HP       D3700            5.04  -
[3:0:17:0]   enclosu HPE      Smart Adapter    3.00  -
[3:1:0:0]    disk    HPE      LOGICAL VOLUME   3.00  /dev/sds
[3:2:0:0]    storage HPE      P408e-p SR Gen10 3.00  -

-----
[global]
ioengine=libaio
; rw=randwrite
; percentage_random=40
rw=write
size=100g
bs=4k
direct=1
ramp_time=15
; filename=/mnt/fio_test
; cpus_allowed=0-27
iodepth=4096

[/dev/sdd]
[/dev/sde]
[/dev/sdf]
[/dev/sdg]
[/dev/sdh]
[/dev/sdi]
[/dev/sdj]
[/dev/sdk]
[/dev/sdl]
[/dev/sdm]
[/dev/sdn]
[/dev/sdo]
[/dev/sdp]
[/dev/sdq]
[/dev/sdr]

Quoted 14 lines

Distribution kernels would be yet another issue, distros can backport host_tagset and get rid of the issue.

Regards
Martin

.

#19 lore raw
Quoted 13 lines · John Garry
Quoted 5 lines · Roger Willcocks

FYI we have exactly this issue on a machine here running CentOS 8.3 (kernel 4.18.0-240.1.1) (so presumably this happens in RHEL 8 too.)
Controller is MSCC / Adaptec 3154-8i16e driving 60 x 12TB HGST drives configured as five x twelve-drive raid-6, software striped using md, and formatted with xfs.
Test software writes to the array using multiple threads in parallel.
The smartpqi driver would report controller offline within ten minutes or so, with status code 0x6100c
Changed the driver to set 'nr_hw_queues = 1’ and then tested by filling the array with random files (which took a couple of days), which completed fine, so it looks like that one-line change fixes it.

That just makes the driver single-queue.

All I can say is it fixes the problem. Write performance is two or three percent faster than CentOS 6.5 on the same hardware.

Quoted 4 lines

As such, since the driver uses blk_mq_unique_tag_to_hwq(), only hw queue #0 will ever be used in the driver.

And then, since the driver still spreads MSI-X interrupt vectors over all CPUs [from pci_alloc_vectors(PCI_IRQ_AFFINITY)], if CPUs associated with HW queue #0 are offlined (probably just cpu0), there is no CPUs available to service queue #0 interrupt. That's what I think would happen, from a quick glance at the code.

Surely that would be an issue even if it used multiple queues (one of which would be queue #0) ?

Quoted 4 lines
Quoted 4 lines

Would, of course, be helpful if this was back-ported. —
Roger

#20 lore raw
Quoted 17 lines · Roger Willcocks
Quoted 13 lines · John Garry
Quoted 5 lines · Roger Willcocks

FYI we have exactly this issue on a machine here running CentOS 8.3 (kernel 4.18.0-240.1.1) (so presumably this happens in RHEL 8 too.)
Controller is MSCC / Adaptec 3154-8i16e driving 60 x 12TB HGST drives configured as five x twelve-drive raid-6, software striped using md, and formatted with xfs.
Test software writes to the array using multiple threads in parallel.
The smartpqi driver would report controller offline within ten minutes or so, with status code 0x6100c
Changed the driver to set 'nr_hw_queues = 1’ and then tested by filling the array with random files (which took a couple of days), which completed fine, so it looks like that one-line change fixes it.

That just makes the driver single-queue.

All I can say is it fixes the problem. Write performance is two or three percent faster than CentOS 6.5 on the same hardware.

Quoted 7 lines
Quoted 4 lines

As such, since the driver uses blk_mq_unique_tag_to_hwq(), only hw queue #0 will ever be used in the driver.

And then, since the driver still spreads MSI-X interrupt vectors over all CPUs [from pci_alloc_vectors(PCI_IRQ_AFFINITY)], if CPUs associated with HW queue #0 are offlined (probably just cpu0), there is no CPUs available to service queue #0 interrupt. That's what I think would happen, from a quick glance at the code.

Surely that would be an issue even if it used multiple queues (one of which would be queue #0) ?

Well, no. Because there is currently a symmetry between HW queue context in the block layer and the HW queues in the LLDD. So if hwq0 were mapped to cpu0 only, if cpu0 is offline, block layer will not send commands on hwq0. By setting nr_hw_queues=1, that symmetry breaks - every cpu tries to send on hwq0, but irq core code will disable hwq0 interrupt when cpu0 is offline - that's because it is managed.

That's how it looks to me - I did not check the LLDD code too closely. Please discuss with Don.

Thanks,
John

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Would, of course, be helpful if this was back-ported. —
Roger

.

#21 lore raw
In reply to Roger Willcocks

Dear Roger,

Am 22.02.21 um 15:23 schrieb Roger Willcocks:

Quoted 2 lines

FYI we have exactly this issue on a machine here running CentOS 8.3 (kernel 4.18.0-240.1.1) (so presumably this happens in RHEL 8 too.)

What driver version do you use?

Quoted 10 lines

Controller is MSCC / Adaptec 3154-8i16e driving 60 x 12TB HGST drives configured as five x twelve-drive raid-6, software striped using md, and formatted with xfs.

Test software writes to the array using multiple threads in parallel.

The smartpqi driver would report controller offline within ten minutes or so, with status code 0x6100c

Quoted 6 lines

Changed the driver to set 'nr_hw_queues = 1’ and then tested by filling the array with random files (which took a couple of days), which completed fine, so it looks like that one-line change fixes it.

Would, of course, be helpful if this was back-ported.

We only noticed the issue starting with Linux 5.5 (commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq").

So I am curious how this problem can be in CentOS 8.3 (Linux kernel 4.18.x).

Quoted 2 lines

—
Roger

Apple Mail mangled your signature delimiter (-- ).

Kind regards,

Paul

#22 lore raw
In reply to [email protected]
Quoted 6 lines · On 20.01.21 19:45, [email protected] wrote:

From: Martin Wilck <[email protected]>

Donald: please give this patch a try.

Commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq") [...]

Unfortunately, this patch (on top of 6eb045e092ef) did not fix the problem. Same error (""controller is offline: status code 0x6100c"") after about 20 minutes of the test load.

Best
Donald

#23 lore raw
Quoted 5 lines · Donald Buczek
Quoted 4 lines · On 20.01.21 19:45, [email protected] wrote:

From: Martin Wilck <[email protected]>

Donald: please give this patch a try.

Quoted 8 lines
Quoted 3 lines

Commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")
[...]

Unfortunately, this patch (on top of 6eb045e092ef) did not fix the problem. Same error (""controller is offline: status code 0x6100c"") after about 20 minutes of the test load.

Too bad. Well, I believe it was worth a try. Thanks for testing it.

Martin

#24 lore raw
Quoted 5 lines · Martin Wilck
Quoted 4 lines · Donald Buczek

Unfortunately, this patch (on top of 6eb045e092ef) did not fix the problem. Same error (""controller is offline: status code 0x6100c"")

Rethinking, this patch couldn't have fixed your problem. I apologize for the dumb suggestion.

However, I still believe I had a point in that patch, and would like to ask the experts for an opinion.

Assume no commands in flight (busy == 0), host_blocked == 2, and that two CPUs enter scsi_host_queue_ready() at the same time. Before 6eb045e092ef, it was impossible that the function returned 1 on either CPU in this situation.

CPU 1 CPU 2

scsi_host_queue_ready scsi_host_queue_ready
host_busy = 1 host_busy = 2
(busy = 0) (busy = 1)
host_blocked = 1 goto starved
goto out_dec add sdev to starved list
host_busy = 1   host_busy = 0
return 0 return 0

With 6eb045e092ef (and without my patch), the result could be:

CPU 1 CPU 2

scsi_host_queue_ready scsi_host_queue_ready

  read scsi_host_busy() = 0          read scsi_host_busy() = 0
  host_blocked = 1                   host_blocked = 0
  goto out_dec                       remove sdev from starved list
  return 0                           set SCMD_STATE_INFLIGHT 
                                       (host_busy = 1)
                                     return 1

So now, one command can be sent, and the host is unblocked. A very different outcome than before. Was this intentional?

Thanks
Martin

#25 lore raw
In reply to [email protected]
Quoted 17 lines · On Wed, Jan 20, 2021 at 07:45:48PM +0100, [email protected] wrote:

From: Martin Wilck <[email protected]>

Donald: please give this patch a try.

Commit 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq") contained this hunk:

(quoted diff) +1 -2
-       busy = atomic_inc_return(&shost->host_busy) - 1;
        if (atomic_read(&shost->host_blocked) > 0) {
-               if (busy)
+               if (scsi_host_busy(shost) > 0)
                        goto starved;

The previous code would increase the busy count before checking host_blocked. With 6eb045e092ef, the busy count would be increased (by setting the SCMD_STATE_INFLIGHT bit) after the if clause for host_blocked above.

Quoted 6 lines

Users have reported a regression with the smartpqi driver [1] which has been shown to be caused by this commit [2].

It seems that by moving the increase of the busy counter further down, it could happen that the can_queue limit of the controller could be exceeded if several CPUs were executing this code in parallel on different queues.

can_queue limit should never be exceeded because it is respected by blk-mq since each hw queue's queue depth is .can_queue.

smartpqi's issue is that its .can_queue does not represent each hw queue's depth, instead the .can_queue represents queue depth of the whole HBA.

As John mentioned, smartpqi should have switched to hosttags.

BTW, looks the following code has soft lockup risk:

pqi_alloc_io_request():

        while (1) {
                io_request = &ctrl_info->io_request_pool[i];
                if (atomic_inc_return(&io_request->refcount) == 1)
                        break;
                atomic_dec(&io_request->refcount);
                i = (i + 1) % ctrl_info->max_io_slots;
        }
Quoted 7 lines

This patch attempts to fix it by moving setting the SCMD_STATE_INFLIGHT before the host_blocked test again. It also inserts barriers to make sure scsi_host_busy() on once CPU will notice the increase of the count from another.

[1]: https://marc.info/?l=linux-scsi&m=160271263114829&w=2
[2]: https://marc.info/?l=linux-scsi&m=161116163722099&w=2

If the above is true wrt. smartpqi's can_queue usage, your patch may not fix the issue completely in which you think '.can_queue is exceeded'.

Quoted 20 lines
Fixes 6eb045e092ef ("scsi: core: avoid host-wide host_busy counter for scsi_mq")
Cc Ming Lei <[email protected]>
Cc Don Brace <[email protected]>
Cc Kevin Barnett <[email protected]>
Cc Donald Buczek <[email protected]>
Cc John Garry <[email protected]>
Cc Paul Menzel <[email protected]>
---
 drivers/scsi/hosts.c    | 2 ++
 drivers/scsi/scsi_lib.c | 8 +++++---
 2 files changed, 7 insertions(+), 3 deletions(-)
drivers/scsi/hosts.c +0 -0
index 2f162603876f..1c452a1c18fd 100644

drivers/scsi/hosts.c static bool scsi_host_check_in_flight(struct request *rq, void *data, · line 568

Quoted 5 lines
drivers/scsi/hosts.c +2 -0
@@ -564,6 +564,8 @@ static bool scsi_host_check_in_flight(struct request *rq, void *data,
 	int *count = data;
 	struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
 
+	/* This pairs with set_bit() in scsi_host_queue_ready() */
+	smp_mb__before_atomic();

So the above barrier orders atomic_read(&shost->host_blocked) and test_bit()?

Quoted 22 lines
drivers/scsi/hosts.c +0 -0
 	if (test_bit(SCMD_STATE_INFLIGHT, &cmd->state))
 		(*count)++;
 
drivers/scsi/scsi_lib.c +5 -1
index b3f14f05340a..0a9a36c349ee 100644
@@ -1353,8 +1353,12 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
 	if (scsi_host_in_recovery(shost))
 		return 0;
 
+	set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
+	/* This pairs with test_bit() in scsi_host_check_in_flight() */
+	smp_mb__after_atomic();
+
 	if (atomic_read(&shost->host_blocked) > 0) {
-		if (scsi_host_busy(shost) > 0)
+		if (scsi_host_busy(shost) > 1)
 			goto starved;
 
 		/*

drivers/scsi/scsi_lib.c static inline int scsi_host_queue_ready(struct request_queue *q, · line 1383

Quoted 5 lines
drivers/scsi/scsi_lib.c +0 -2
@@ -1379,8 +1383,6 @@ static inline int scsi_host_queue_ready(struct request_queue *q,
 		spin_unlock_irq(shost->host_lock);
 	}
 
-	__set_bit(SCMD_STATE_INFLIGHT, &cmd->state);
-

Looks this patch fine.

However, I'd suggest to confirm smartpqi's .can_queue usage first, which looks one big issue.

Signature · 1 line
Ming
#26 lore raw
Quoted 15 lines · Ming Lei
Quoted 9 lines
drivers/scsi/hosts.c +0 -0
index 2f162603876f..1c452a1c18fd 100644
@@ -564,6 +564,8 @@ static bool scsi_host_check_in_flight(struct
 request *rq, void *data,
         int *count = data;
         struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
Quoted 3 lines


+       /* This pairs with set_bit() in scsi_host_queue_ready() */ +       smp_mb__before_atomic();

So the above barrier orders atomic_read(&shost->host_blocked) and test_bit()?

Yes, I believe the change to SCMD_STATE_INFLIGHT should be visible before host_blocked is tested. For that, I would need to insert a full smb_mb() instead of smp_mb_after_atomic() below, though ... right? Which means that the smp_mb__before_atomic() above could be removed. The important thing is that if two threads execute scsi_host_queue_ready() simultaneously, one of them must see the SCMD_STATE_INFLIGHT bit of the other.

Btw, I realized that calling scsi_host_busy() in scsi_host_queue_ready() is inefficient, because all we need to know there is whether the value is > 1; we don't need to iterate through the entire tag space.

Thanks,
Martin

Quoted 41 lines
Quoted 28 lines

        if (test_bit(SCMD_STATE_INFLIGHT, &cmd->state)) (*count)++;

drivers/scsi/scsi_lib.c +0 -0
index b3f14f05340a..0a9a36c349ee 100644
@@ -1353,8 +1353,12 @@ static inline int
 scsi_host_queue_ready(struct request_queue *q,
         if (scsi_host_in_recovery(shost))
                 return 0;


+       set_bit(SCMD_STATE_INFLIGHT, &cmd->state); +       /* This pairs with test_bit() in scsi_host_check_in_flight() */
+       smp_mb__after_atomic();
+
        if (atomic_read(&shost->host_blocked) > 0) { -               if (scsi_host_busy(shost) > 0) +               if (scsi_host_busy(shost) > 1) goto starved;

                /*

drivers/scsi/scsi_lib.c +0 -0
@@ -1379,8 +1383,6 @@ static inline int
 scsi_host_queue_ready(struct request_queue *q,
                 spin_unlock_irq(shost->host_lock);
         }
Quoted 3 lines


-       __set_bit(SCMD_STATE_INFLIGHT, &cmd->state); -

Looks this patch fine.

However, I'd suggest to confirm smartpqi's .can_queue usage first, which
looks one big issue.