Skip to contentSkip to conversation

block: I/O error occurs during SATA disk stress test

PATCH Superseded Gu Mi 2 participants 3 messages Aug 25, 2022 – Aug 26, 2022 Block layer

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Gu Mi

author
#0 lore raw

PATCH v2 1/1 block: I/O error occurs during SATA disk stress test

The problem occurs in two async processes, One is when a new IO calls the blk_mq_start_request() interface to start sending,The other is that the block layer timer process calls the blk_mq_req_expired interface to check whether there is an IO timeout.

When an instruction out of sequence occurs between blk_add_timer and WRITE_ONCE(rq->state,MQ_RQ_IN_FLIGHT) in the interface blk_mq_start_request,at this time, the block timer is checking the new IO timeout, Since the req status has been set to MQ_RQ_IN_FLIGHT and req->deadline is 0 at this time, the new IO will be misjudged as a timeout.

Our repair plan is for the deadline to be 0, and we do not think that a timeout occurs. At the same time, because the jiffies of the 32-bit system will be reversed shortly after the system is turned on, we will add 1 jiffies to the deadline at this time.

---
v1->v2:

time_after_eq() can handle the overflow, so remove the change on 32-bit in blk_add_timer().

 block/blk-mq.c | 2 ++
 1 file changed, 2 insertions(+)
block/blk-mq.c +2 -0
index 4b90d2d..6defaa1 100644
@@ -1451,6 +1451,8 @@ static bool blk_mq_req_expired(struct request *rq, unsigned long *next)
 		return false;
 
 	deadline = READ_ONCE(rq->deadline);
+	if (unlikely(deadline == 0))
+		return false;
 	if (time_after_eq(jiffies, deadline))
 		return true;
 
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 32 lines · Gu Mi

The problem occurs in two async processes, One is when a new IO calls the blk_mq_start_request() interface to start sending,The other is that the block layer timer process calls the blk_mq_req_expired interface to check whether there is an IO timeout.

When an instruction out of sequence occurs between blk_add_timer and WRITE_ONCE(rq->state,MQ_RQ_IN_FLIGHT) in the interface blk_mq_start_request,at this time, the block timer is checking the new IO timeout, Since the req status has been set to MQ_RQ_IN_FLIGHT and req->deadline is 0 at this time, the new IO will be misjudged as a timeout.

Our repair plan is for the deadline to be 0, and we do not think that a timeout occurs. At the same time, because the jiffies of the 32-bit system will be reversed shortly after the system is turned on, we will add 1 jiffies to the deadline at this time.

---
v1->v2:

time_after_eq() can handle the overflow, so remove the change on 32-bit in blk_add_timer().

  block/blk-mq.c | 2 ++
  1 file changed, 2 insertions(+)
block/blk-mq.c +0 -0
index 4b90d2d..6defaa1 100644

block/blk-mq.c static bool blk_mq_req_expired(struct request *rq, unsigned long *next) · line 1458

Quoted 8 lines
block/blk-mq.c +2 -0
@@ -1451,6 +1451,8 @@ static bool blk_mq_req_expired(struct request *rq, unsigned long *next)
  		return false;
 
  	deadline = READ_ONCE(rq->deadline);
+	if (unlikely(deadline == 0))
+		return false;
  	if (time_after_eq(jiffies, deadline))
  		return true;
 

rq->deadline == 0 can be a valid deadline value so the above patch doesn't look right to me.

Thanks,

Bart.

#2 lore raw
In reply to Gu Mi
Quoted 12 lines · Gu Mi

The problem occurs in two async processes, One is when a new IO calls the blk_mq_start_request() interface to start sending,The other is that the block layer timer process calls the blk_mq_req_expired interface to check whether there is an IO timeout.

When an instruction out of sequence occurs between blk_add_timer and WRITE_ONCE(rq->state,MQ_RQ_IN_FLIGHT) in the interface blk_mq_start_request,at this time, the block timer is checking the new IO timeout, Since the req status has been set to MQ_RQ_IN_FLIGHT and req->deadline is 0 at this time, the new IO will be misjudged as a timeout.

Quoted 4 lines

Our repair plan is for the deadline to be 0, and we do not think that a timeout occurs. At the same time, because the jiffies of the 32-bit system will be reversed shortly after the system is turned on, we will add 1 jiffies to the deadline at this time.

Hi Gu,

With which kernel version has this race been observed? Since commit 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter") the request reference count is increased before the timeout handler (blk_mq_check_expired()) is called. Do you agree that since then it is no longer possible that blk_mq_start_request() is called while blk_mq_check_expired() is in progress?

Thanks,

Bart.