usb: dwc3: gadget: Replace list_for_each_entry_safe() if using giveback

PATCH Superseded Wesley Cheng 2 participants 4 messages May 7, 2021 – May 10, 2021

Subsystems (2)

USB DesignWare USB3 DRD IP driver

lore

Share & tools

lore

Wesley Cheng

author

2021-05-07 18:42 UTC #0 lore raw

PATCH 1/1 usb: dwc3: gadget: Replace list_for_each_entry_safe() if using giveback

The list_for_each_entry_safe() macro saves the current item (n) and the item after (n+1), so that n can be safely removed without corrupting the list. However, when traversing the list and removing items using gadget giveback, the DWC3 lock is briefly released, allowing other routines to execute. There is a situation where while items are being removed from the cancelled_list using dwc3_gadget_ep_cleanup_cancelled_requests(), the pullup disable routine is running in parallel (due to UDC unbind). As the cleanup routine removes n, and the pullup disable removes n+1, once the cleanup retakes the DWC3 lock, it references a request who was already removed/handled. With list debug enabled, this leads to a panic. Ensure all instances of the macro are replaced where gadget giveback is used.

Fixes d4f1afe5e896 ("usb: dwc3: gadget: move requests to cancelled_list")

Signed-off-by Wesley Cheng <[email protected]>

---
 drivers/usb/dwc3/gadget.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

drivers/usb/dwc3/gadget.c +4 -4

index dd80e5c..efa939b 100644
@@ -1737,10 +1737,10 @@ static void dwc3_gadget_ep_skip_trbs(struct dwc3_ep *dep, struct dwc3_request *r
 static void dwc3_gadget_ep_cleanup_cancelled_requests(struct dwc3_ep *dep)
 {
 	struct dwc3_request		*req;
-	struct dwc3_request		*tmp;
 	struct dwc3			*dwc = dep->dwc;
 
-	list_for_each_entry_safe(req, tmp, &dep->cancelled_list, list) {
+	while (!list_empty(&dep->cancelled_list)) {
+		req = next_request(&dep->cancelled_list);
 		dwc3_gadget_ep_skip_trbs(dep, req);
 		switch (req->status) {
 		case DWC3_REQUEST_STATUS_DISCONNECTED:

@@ -2935,11 +2935,11 @@ static void dwc3_gadget_ep_cleanup_completed_requests(struct dwc3_ep *dep,
 		const struct dwc3_event_depevt *event, int status)
 {
 	struct dwc3_request	*req;
-	struct dwc3_request	*tmp;
 
-	list_for_each_entry_safe(req, tmp, &dep->started_list, list) {
+	while (!list_empty(&dep->started_list)) {
 		int ret;
 
+		req = next_request(&dep->started_list);
 		ret = dwc3_gadget_ep_cleanup_completed_request(dep, event,
 				req, status);
 		if (ret)

Signature · 2 lines

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Peter Chen

2021-05-08 03:45 UTC #1 lore raw

Quoted 4 lines · Wesley Cheng

The list_for_each_entry_safe() macro saves the current item (n) and the item after (n+1), so that n can be safely removed without corrupting the list. However, when traversing the list and removing items using gadget giveback, the DWC3 lock is briefly released,

I see dwc3_gadget_del_and_unmap_request remove the list, the lock is still held there. Am I something wrong?

Peter

Quoted 51 lines

allowing other routines to execute. There is a situation where while items are being removed from the cancelled_list using dwc3_gadget_ep_cleanup_cancelled_requests(), the pullup disable routine is running in parallel (due to UDC unbind). As the cleanup routine removes n, and the pullup disable removes n+1, once the cleanup retakes the DWC3 lock, it references a request who was already removed/handled. With list debug enabled, this leads to a panic. Ensure all instances of the macro are replaced where gadget giveback is used.
Fixes d4f1afe5e896 ("usb: dwc3: gadget: move requests to cancelled_list")
Signed-off-by Wesley Cheng <[email protected]>
---
 drivers/usb/dwc3/gadget.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/usb/dwc3/gadget.c +4 -4
index dd80e5c..efa939b 100644
@@ -1737,10 +1737,10 @@ static void dwc3_gadget_ep_skip_trbs(struct dwc3_ep *dep, struct dwc3_request *r
 static void dwc3_gadget_ep_cleanup_cancelled_requests(struct dwc3_ep *dep)
 {
 	struct dwc3_request		*req;
-	struct dwc3_request		*tmp;
 	struct dwc3			*dwc = dep->dwc;
 
-	list_for_each_entry_safe(req, tmp, &dep->cancelled_list, list) {
+	while (!list_empty(&dep->cancelled_list)) {
+		req = next_request(&dep->cancelled_list);
 		dwc3_gadget_ep_skip_trbs(dep, req);
 		switch (req->status) {
 		case DWC3_REQUEST_STATUS_DISCONNECTED:
@@ -2935,11 +2935,11 @@ static void dwc3_gadget_ep_cleanup_completed_requests(struct dwc3_ep *dep,
 		const struct dwc3_event_depevt *event, int status)
 {
 	struct dwc3_request	*req;
-	struct dwc3_request	*tmp;
 
-	list_for_each_entry_safe(req, tmp, &dep->started_list, list) {
+	while (!list_empty(&dep->started_list)) {
 		int ret;
 
+		req = next_request(&dep->started_list);
 		ret = dwc3_gadget_ep_cleanup_completed_request(dep, event,
 				req, status);
 		if (ret)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Signature · 2 lines

Thanks,
Peter Chen

Wesley Cheng

author

2021-05-08 08:15 UTC #2 lore raw

Quoted 5 lines · Peter Chen

Quoted 4 lines · Wesley Cheng
The list_for_each_entry_safe() macro saves the current item (n) and the item after (n+1), so that n can be safely removed without corrupting the list. However, when traversing the list and removing items using gadget giveback, the DWC3 lock is briefly released,

Hi Peter,

Thanks for the review.

Quoted 4 lines

I see dwc3_gadget_del_and_unmap_request remove the list, the lock is still held there. Am I something wrong?

The scenario the issue happens in is say the follow thread is running the sequence below:

Thread#1:
__dwc3_gadget_ep_set_halt() - CLEAR HALT
-> dwc3_gadget_ep_cleanup_cancelled_requests()

    ->list_for_each_entry_safe()
    ->dwc3_gadget_giveback()
      ->dwc3_gadget_del_and_unmap_request()-n deleted cancelled_list
      ->spin_unlock

Thread#2:
dwc3_gadget_pullup()
->waiting for dwc3 spin_lock
->Thread#1 released lock
->dwc3_stop_active_transfers()

    ->dwc3_remove_requests()
      ->fetches n+1 item from cancelled_list (n removed by thread#1)
      ->dwc3_gadget_giveback()
        ->dwc3_gadget_del_and_unmap_request()-n+1 deleted cancelled_list
        ->spin_unlock

So now, if thread#1 takes the DWC3 lock again, it will continue to item n+1, which was already removed by thread#2, leading to a double list removal. We saw this issue on our platform after enabling list debug.

Thanks
Wesley Cheng

Quoted 54 lines

Peter
Quoted 51 lines
allowing other routines to execute. There is a situation where while items are being removed from the cancelled_list using dwc3_gadget_ep_cleanup_cancelled_requests(), the pullup disable routine is running in parallel (due to UDC unbind). As the cleanup routine removes n, and the pullup disable removes n+1, once the cleanup retakes the DWC3 lock, it references a request who was already removed/handled. With list debug enabled, this leads to a panic. Ensure all instances of the macro are replaced where gadget giveback is used.
Fixes d4f1afe5e896 ("usb: dwc3: gadget: move requests to cancelled_list")
Signed-off-by Wesley Cheng <[email protected]>
---
 drivers/usb/dwc3/gadget.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/usb/dwc3/gadget.c +4 -4
index dd80e5c..efa939b 100644
@@ -1737,10 +1737,10 @@ static void dwc3_gadget_ep_skip_trbs(struct dwc3_ep *dep, struct dwc3_request *r
 static void dwc3_gadget_ep_cleanup_cancelled_requests(struct dwc3_ep *dep)
 {
 	struct dwc3_request		*req;
-	struct dwc3_request		*tmp;
 	struct dwc3			*dwc = dep->dwc;
 
-	list_for_each_entry_safe(req, tmp, &dep->cancelled_list, list) {
+	while (!list_empty(&dep->cancelled_list)) {
+		req = next_request(&dep->cancelled_list);
 		dwc3_gadget_ep_skip_trbs(dep, req);
 		switch (req->status) {
 		case DWC3_REQUEST_STATUS_DISCONNECTED:
@@ -2935,11 +2935,11 @@ static void dwc3_gadget_ep_cleanup_completed_requests(struct dwc3_ep *dep,
 		const struct dwc3_event_depevt *event, int status)
 {
 	struct dwc3_request	*req;
-	struct dwc3_request	*tmp;
 
-	list_for_each_entry_safe(req, tmp, &dep->started_list, list) {
+	while (!list_empty(&dep->started_list)) {
 		int ret;
 
+		req = next_request(&dep->started_list);
 		ret = dwc3_gadget_ep_cleanup_completed_request(dep, event,
 				req, status);
 		if (ret)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Signature · 2 lines

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Peter Chen

Reviewed-by

2021-05-10 00:55 UTC #3 lore raw

Quoted 40 lines · Wesley Cheng

Quoted 5 lines · Peter Chen
Quoted 4 lines · Wesley Cheng
The list_for_each_entry_safe() macro saves the current item (n) and the item after (n+1), so that n can be safely removed without corrupting the list. However, when traversing the list and removing items using gadget giveback, the DWC3 lock is briefly released,
Hi Peter,
Thanks for the review.
Quoted 4 lines
I see dwc3_gadget_del_and_unmap_request remove the list, the lock is still held there. Am I something wrong?
The scenario the issue happens in is say the follow thread is running the sequence below:
Thread#1:
__dwc3_gadget_ep_set_halt() - CLEAR HALT
-> dwc3_gadget_ep_cleanup_cancelled_requests()
    ->list_for_each_entry_safe()
    ->dwc3_gadget_giveback()
      ->dwc3_gadget_del_and_unmap_request()-n deleted cancelled_list
      ->spin_unlock
Thread#2:
dwc3_gadget_pullup()
->waiting for dwc3 spin_lock
->Thread#1 released lock
->dwc3_stop_active_transfers()
    ->dwc3_remove_requests()
      ->fetches n+1 item from cancelled_list (n removed by thread#1)
      ->dwc3_gadget_giveback()
        ->dwc3_gadget_del_and_unmap_request()-n+1 deleted cancelled_list
        ->spin_unlock

Quoted 3 lines

So now, if thread#1 takes the DWC3 lock again, it will continue to item n+1, which was already removed by thread#2, leading to a double list removal. We saw this issue on our platform after enabling list debug.

It is cleared now. Would you please update commit log a little by appending your call stack analysis?

Reviewed-by Peter Chen <[email protected]>

Peter

Quoted 62 lines

Thanks
Wesley Cheng
Quoted 54 lines
Peter
Quoted 51 lines
allowing other routines to execute. There is a situation where while items are being removed from the cancelled_list using dwc3_gadget_ep_cleanup_cancelled_requests(), the pullup disable routine is running in parallel (due to UDC unbind). As the cleanup routine removes n, and the pullup disable removes n+1, once the cleanup retakes the DWC3 lock, it references a request who was already removed/handled. With list debug enabled, this leads to a panic. Ensure all instances of the macro are replaced where gadget giveback is used.
Fixes d4f1afe5e896 ("usb: dwc3: gadget: move requests to cancelled_list")
Signed-off-by Wesley Cheng <[email protected]>
---
 drivers/usb/dwc3/gadget.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/usb/dwc3/gadget.c +4 -4
index dd80e5c..efa939b 100644
@@ -1737,10 +1737,10 @@ static void dwc3_gadget_ep_skip_trbs(struct dwc3_ep *dep, struct dwc3_request *r
 static void dwc3_gadget_ep_cleanup_cancelled_requests(struct dwc3_ep *dep)
 {
 	struct dwc3_request		*req;
-	struct dwc3_request		*tmp;
 	struct dwc3			*dwc = dep->dwc;
 
-	list_for_each_entry_safe(req, tmp, &dep->cancelled_list, list) {
+	while (!list_empty(&dep->cancelled_list)) {
+		req = next_request(&dep->cancelled_list);
 		dwc3_gadget_ep_skip_trbs(dep, req);
 		switch (req->status) {
 		case DWC3_REQUEST_STATUS_DISCONNECTED:
@@ -2935,11 +2935,11 @@ static void dwc3_gadget_ep_cleanup_completed_requests(struct dwc3_ep *dep,
 		const struct dwc3_event_depevt *event, int status)
 {
 	struct dwc3_request	*req;
-	struct dwc3_request	*tmp;
 
-	list_for_each_entry_safe(req, tmp, &dep->started_list, list) {
+	while (!list_empty(&dep->started_list)) {
 		int ret;
 
+		req = next_request(&dep->started_list);
 		ret = dwc3_gadget_ep_cleanup_completed_request(dep, event,
 				req, status);
 		if (ret)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Signature · 2 lines

Thanks,
Peter Chen