block: I/O error occurs during SATA disk stress test
PATCH v2 1/1 block: I/O error occurs during SATA disk stress test
The problem occurs in two async processes, One is when a new IO calls the blk_mq_start_request() interface to start sending,The other is that the block layer timer process calls the blk_mq_req_expired interface to check whether there is an IO timeout.
When an instruction out of sequence occurs between blk_add_timer and WRITE_ONCE(rq->state,MQ_RQ_IN_FLIGHT) in the interface blk_mq_start_request,at this time, the block timer is checking the new IO timeout, Since the req status has been set to MQ_RQ_IN_FLIGHT and req->deadline is 0 at this time, the new IO will be misjudged as a timeout.
Our repair plan is for the deadline to be 0, and we do not think that a timeout occurs. At the same time, because the jiffies of the 32-bit system will be reversed shortly after the system is turned on, we will add 1 jiffies to the deadline at this time.
---v1->v2:
time_after_eq() can handle the overflow, so remove the change on 32-bit in blk_add_timer().
block/blk-mq.c | 2 ++ 1 file changed, 2 insertions(+)
block/blk-mq.c +2 -0
@@ -1451,6 +1451,8 @@ static bool blk_mq_req_expired(struct request *rq, unsigned long *next)return false; deadline =READ_ONCE (rq->deadline); +if (unlikely (deadline ==0 )) +return false;if (time_after_eq (jiffies, deadline))return true;
Signature · 1 line
1.8.3.1
Quoted 32 lines · Gu Mi
The problem occurs in two async processes, One is when a new IO calls the
blk_mq_start_request()interface to start sending,The other is that the block layer timer process calls theblk_mq_req_expiredinterface to check whether there is an IO timeout.When an instruction out of sequence occurs between
blk_add_timerandWRITE_ONCE(rq->state,MQ_RQ_IN_FLIGHT)in the interface blk_mq_start_request,at this time, the block timer is checking the new IO timeout, Since the req status has been set toMQ_RQ_IN_FLIGHTandreq->deadlineis 0 at this time, the new IO will be misjudged as a timeout.Our repair plan is for the deadline to be 0, and we do not think that a timeout occurs. At the same time, because the jiffies of the 32-bit system will be reversed shortly after the system is turned on, we will add 1 jiffies to the deadline at this time.
Signed-off-by Gu Mi <[email protected]>---
v1->v2:
time_after_eq()can handle the overflow, so remove the change on 32-bit inblk_add_timer().block/blk-mq.c | 2 ++ 1 file changed, 2 insertions(+)block/blk-mq.c +0
-0
block/blk-mq.c static bool blk_mq_req_expired(struct request *rq, unsigned long *next) · line 1458
Quoted 8 lines
block/blk-mq.c +2
-0@@ -1451,6 +1451,8 @@ static bool blk_mq_req_expired(struct request *rq, unsigned long *next)return false; deadline =READ_ONCE (rq->deadline); +if (unlikely (deadline ==0 )) +return false;if (time_after_eq (jiffies, deadline))return true;
rq->deadline == 0 can be a valid deadline value so the above patch doesn't look right to me.
Thanks,
Bart.
Quoted 12 lines · Gu Mi
The problem occurs in two async processes, One is when a new IO calls the
blk_mq_start_request()interface to start sending,The other is that the block layer timer process calls theblk_mq_req_expiredinterface to check whether there is an IO timeout.When an instruction out of sequence occurs between
blk_add_timerandWRITE_ONCE(rq->state,MQ_RQ_IN_FLIGHT)in the interface blk_mq_start_request,at this time, the block timer is checking the new IO timeout, Since the req status has been set toMQ_RQ_IN_FLIGHTandreq->deadlineis 0 at this time, the new IO will be misjudged as a timeout.
Quoted 4 lines
Our repair plan is for the deadline to be 0, and we do not think that a timeout occurs. At the same time, because the jiffies of the 32-bit system will be reversed shortly after the system is turned on, we will add 1 jiffies to the deadline at this time.
Hi Gu,
With which kernel version has this race been observed? Since commit 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter") the request reference count is increased before the timeout handler (blk_mq_check_expired()) is called. Do you agree that since then it is no longer possible that blk_mq_start_request() is called while blk_mq_check_expired() is in progress?
Thanks,
Bart.