Skip to contentSkip to conversation

fix races in CDC-WDM

RFC In review Oliver Neukum 2 participants 22 messages Aug 12, 2020 – Sep 22, 2020 USB

lore
Share & tools
lore
#0 lore raw

cover fix races in CDC-WDM

CDC-WDM was not written with multithreaded users or uncooperative devices in mind.
This leads to race conditions and hangs in flush().

#1 lore raw

RFC 1/5 CDC-WDM: fix hangs in flush()

In a multithreaded scenario a flush() and a write() may be waiting for the same IO to complete. Hence completion of output must use wake_up_all(), even in error handling, while a flush may be waiting for an intended but not started Io.

---
 drivers/usb/class/cdc-wdm.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/usb/class/cdc-wdm.c +7 -4
index 7f5de956a2fc..a834294ce4e5 100644
@@ -151,7 +151,7 @@ static void wdm_out_callback(struct urb *urb)
 	kfree(desc->outbuf);
 	desc->outbuf = NULL;
 	clear_bit(WDM_IN_USE, &desc->flags);
-	wake_up(&desc->wait);
+	wake_up_all(&desc->wait);
 }
 
 static void wdm_in_callback(struct urb *urb)
@@ -424,6 +424,7 @@ static ssize_t wdm_write
 	if (rv < 0) {
 		desc->outbuf = NULL;
 		clear_bit(WDM_IN_USE, &desc->flags);
+		wake_up_all(&desc->wait); /* for flush() */
 		dev_err(&desc->intf->dev, "Tx URB error: %d\n", rv);
 		rv = usb_translate_errors(rv);
 		goto out_free_mem_pm;
@@ -586,6 +587,7 @@ static ssize_t wdm_read
 static int wdm_flush(struct file *file, fl_owner_t id)
 {
 	struct wdm_device *desc = file->private_data;
+	int rv;
 
 	wait_event(desc->wait,
 			/*
@@ -600,11 +602,12 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 	/* cannot dereference desc->intf if WDM_DISCONNECTING */
 	if (test_bit(WDM_DISCONNECTING, &desc->flags))
 		return -ENODEV;
-	if (desc->werr < 0)
+	rv = desc->werr;
+	if (rv < 0)
 		dev_err(&desc->intf->dev, "Error in flush path: %d\n",
-			desc->werr);
+			rv);
 
-	return usb_translate_errors(desc->werr);
+	return usb_translate_errors(rv);
 }
 
 static __poll_t wdm_poll(struct file *file, struct poll_table_struct *wait)
Signature · 1 line
2.16.4
#2 lore raw

RFC 2/5 CDC-WDM: introduce a timeout in flush()

In reply to Oliver Neukum

Malicious or defective hardware may cease communication while flush() is running. In last consequence we need a timeout, as hardware that remains silent must be ignored.

The 30 seconds are coming out of thin air.

---
 drivers/usb/class/cdc-wdm.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/usb/class/cdc-wdm.c +16 -2
index a834294ce4e5..329f8d5f7c8d 100644
@@ -58,6 +58,9 @@ MODULE_DEVICE_TABLE (usb, wdm_ids);
 
 #define WDM_MAX			16
 
+/* flush() is uninterruptible, but we cannot wait forever */
+#define WDM_FLUSH_TIMEOUT	(30 * HZ)
+
 /* CDC-WMC r1.1 requires wMaxCommand to be "at least 256 decimal (0x100)" */
 #define WDM_DEFAULT_BUFSIZE	256
 
@@ -589,7 +592,7 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 	struct wdm_device *desc = file->private_data;
 	int rv;
 
-	wait_event(desc->wait,
+	rv = wait_event_timeout(desc->wait,
 			/*
 			 * needs both flags. We cannot do with one
 			 * because resetting it would cause a race
@@ -597,11 +600,14 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 			 * a disconnect
 			 */
 			!test_bit(WDM_IN_USE, &desc->flags) ||
-			test_bit(WDM_DISCONNECTING, &desc->flags));
+			test_bit(WDM_DISCONNECTING, &desc->flags),
+			WDM_FLUSH_TIMEOUT);
 
 	/* cannot dereference desc->intf if WDM_DISCONNECTING */
 	if (test_bit(WDM_DISCONNECTING, &desc->flags))
 		return -ENODEV;
+	if (!rv)
+		return -EIO;
 	rv = desc->werr;
 	if (rv < 0)
 		dev_err(&desc->intf->dev, "Error in flush path: %d\n",
@@ -659,6 +665,14 @@ static int wdm_open(struct inode *inode, struct file *file)
 		goto out;
 	}
 
+	/*
+	 * in case flush() had timed out
+	 */
+	usb_kill_urb(desc->command);
+	spin_lock_irq(&desc->iuspin);
+	desc->werr = 0;
+	spin_unlock_irq(&desc->iuspin);
+
 	/* using write lock to protect desc->count */
 	mutex_lock(&desc->wlock);
 	if (!desc->count++) {
Signature · 1 line
2.16.4
#3 lore raw

RFC 3/5 CDC-WDM: making flush() interruptible

In reply to Oliver Neukum

There is no need for flush() to be uninterruptible. close(2) is allowed to return -EAGAIN. Change it.

---
 drivers/usb/class/cdc-wdm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/usb/class/cdc-wdm.c +3 -1
index 329f8d5f7c8d..f442f34f38da 100644
@@ -592,7 +592,7 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 	struct wdm_device *desc = file->private_data;
 	int rv;
 
-	rv = wait_event_timeout(desc->wait,
+	rv = wait_event_interruptible_timeout(desc->wait,
 			/*
 			 * needs both flags. We cannot do with one
 			 * because resetting it would cause a race
@@ -608,6 +608,8 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 		return -ENODEV;
 	if (!rv)
 		return -EIO;
+	if (rv < 0)
+		return -EINTR;
 	rv = desc->werr;
 	if (rv < 0)
 		dev_err(&desc->intf->dev, "Error in flush path: %d\n",
Signature · 1 line
2.16.4
#4 lore raw

RFC 4/5 CDC-WDM: fix race reporting errors in flush

In reply to Oliver Neukum

In case a race was lost and multiple fds used,
an error could be reported multiple times. To fix
this a spinlock must be taken.

---
 drivers/usb/class/cdc-wdm.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/usb/class/cdc-wdm.c +6 -2
index f442f34f38da..6071c9044574 100644
@@ -610,10 +610,14 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 		return -EIO;
 	if (rv < 0)
 		return -EINTR;
+
+	spin_lock_irq(&desc->iuspin);
 	rv = desc->werr;
+	desc->werr = 0;
+	spin_unlock_irq(&desc->iuspin);
+
 	if (rv < 0)
-		dev_err(&desc->intf->dev, "Error in flush path: %d\n",
-			rv);
+		dev_err(&desc->intf->dev, "Error in flush path: %d\n", rv);
 
 	return usb_translate_errors(rv);
 }
Signature · 1 line
2.16.4
#5 lore raw

RFC 5/5 CDC-WDM: remove use of intf->dev after potential disconnect

In reply to Oliver Neukum

After a disconnect intf->dev is not a valid pointer any longer as flush() uses it only for logging purposes logging is not worth it. Remove the dev_err()

---
 drivers/usb/class/cdc-wdm.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/usb/class/cdc-wdm.c +0 -3
index 6071c9044574..2dcf1fb16f2b 100644
@@ -616,9 +616,6 @@ static int wdm_flush(struct file *file, fl_owner_t id)
 	desc->werr = 0;
 	spin_unlock_irq(&desc->iuspin);
 
-	if (rv < 0)
-		dev_err(&desc->intf->dev, "Error in flush path: %d\n", rv);
-
 	return usb_translate_errors(rv);
 }
 
Signature · 1 line
2.16.4
#6 lore raw
In reply to Oliver Neukum
Quoted 4 lines · Oliver Neukum

CDC-WDM was not written with multithreaded users or uncooperative devices in mind.
This leads to race conditions and hangs in flush().

In patch "[RFC 2/5] CDC-WDM: introduce a timeout in flush()", since multiple users can share "desc", wouldn't someone's usb_submit_urb() from wdm_write() gets unexpectedly interfered by someone else's usb_kill_urb(desc->command) from wdm_open() ?

In patch "[RFC 2/5] CDC-WDM: introduce a timeout in flush()", don't we need to similarly apply timeout to wait_event_interruptible() in wdm_write(), for the same problem will happen if hardware remained silent forever?

In patch "[RFC 3/5] CDC-WDM: making flush() interruptible", it is legal to return -EINTR
from close(). But I think that returning -EINTR from close() is not recommended because
it can confuse multithreaded application (retrying close() upon -EINTR is not safe).

In patch "[RFC 5/5] CDC-WDM: remove use of intf->dev after potential disconnect",

        /* cannot dereference desc->intf if WDM_DISCONNECTING */
        if (test_bit(WDM_DISCONNECTING, &desc->flags))
                return -ENODEV;

can be also removed, for this check is meant for not to dereference desc->intf after disconnect ?

Guessing from symmetry, do we need to check WDM_DISCONNECTING or WDM_RESETTING in wait_event_interruptible_timeout() from wdm_flush() when wait_event_interruptible() in wdm_write() is not checking WDM_DISCONNECTING nor WDM_RESETTING ?

Does it make sense to wait for response of someone else's usb_submit_urb() when someone is calling close(), for there is no guarantee that failure notice received via wdm_flush() via some file descriptor corresponds to usb_submit_urb() request from wdm_write() call from that file descriptor?

#7 lore raw

Am Mittwoch, den 12.08.2020, 23:29 +0900 schrieb Tetsuo Handa:

Quoted 4 lines · Oliver Neukum
Quoted 3 lines

CDC-WDM was not written with multithreaded users or uncooperative devices in mind.
This leads to race conditions and hangs in flush().

Hi,

thank you for waiting. Family emergency.

Quoted 3 lines

In patch "[RFC 2/5] CDC-WDM: introduce a timeout in flush()", since multiple users can share "desc", wouldn't someone's usb_submit_urb() from wdm_write() gets unexpectedly interfered by someone else's usb_kill_urb(desc->command) from wdm_open() ?

Yes. Fixed.

Quoted 3 lines

In patch "[RFC 2/5] CDC-WDM: introduce a timeout in flush()", don't we need to similarly apply timeout to wait_event_interruptible() in wdm_write(), for the same problem will happen if hardware remained silent forever?

Technically a device may take forever. Interrupting a wait in write() is not problematic.

Quoted 3 lines

In patch "[RFC 3/5] CDC-WDM: making flush() interruptible", it is legal to return -EINTR
from close(). But I think that returning -EINTR from close() is not recommended because
it can confuse multithreaded application (retrying close() upon -EINTR is not safe).

Well, but what is the alternative? Should we ignore signals?

Quoted 8 lines

In patch "[RFC 5/5] CDC-WDM: remove use of intf->dev after potential disconnect",

        /* cannot dereference desc->intf if WDM_DISCONNECTING */
        if (test_bit(WDM_DISCONNECTING, &desc->flags))
                return -ENODEV;

can be also removed, for this check is meant for not to dereference desc->intf after disconnect ?

It can be removed, but that would make error reporting worse.

Quoted 3 lines

Guessing from symmetry, do we need to check WDM_DISCONNECTING or WDM_RESETTING in wait_event_interruptible_timeout() from wdm_flush() when wait_event_interruptible() in wdm_write() is not checking WDM_DISCONNECTING nor WDM_RESETTING ?

Added

Quoted 4 lines

Does it make sense to wait for response of someone else's usb_submit_urb() when someone is calling close(), for there is no guarantee that failure notice received via wdm_flush() via some file descriptor corresponds to usb_submit_urb() request from wdm_write() call from that file descriptor?

Well, user space may do multithreading. Whether it makes sense is another question. We just need to return results confirming to the standards. You noticed bugs. I think the next version will fix them.

What else would we do?

Regards
Oliver

#8 lore raw
Quoted 8 lines · Oliver Neukum
Quoted 4 lines

Does it make sense to wait for response of someone else's usb_submit_urb() when someone is calling close(), for there is no guarantee that failure notice received via wdm_flush() via some file descriptor corresponds to usb_submit_urb() request from wdm_write() call from that file descriptor?

Well, user space may do multithreading. Whether it makes sense is another question. We just need to return results confirming to the standards. You noticed bugs. I think the next version will fix them.

My question is how do you handle if App1 and App2 (not multithreading but multiprocessing) shares the "desc" ? Unless

Quoted 6 lines
Quoted 3 lines

In patch "[RFC 3/5] CDC-WDM: making flush() interruptible", it is legal to return -EINTR
from close(). But I think that returning -EINTR from close() is not recommended because
it can confuse multithreaded application (retrying close() upon -EINTR is not safe).

Well, but what is the alternative? Should we ignore signals?

we return the error from write() request (i.e. give up trying to report errors from close() event), we can't pass results to the intended recipients.

#9 lore raw

Am Donnerstag, den 10.09.2020, 19:01 +0900 schrieb Tetsuo Handa:

Quoted 6 lines · Oliver Neukum
Quoted 5 lines
Quoted 4 lines

Does it make sense to wait for response of someone else's usb_submit_urb() when someone is calling close(), for there is no guarantee that failure notice received via wdm_flush() via some file descriptor corresponds to usb_submit_urb() request from wdm_write() call from that file descriptor?

Quoted 6 lines
Quoted 3 lines

Well, user space may do multithreading. Whether it makes sense is another question. We just need to return results confirming to the standards. You noticed bugs. I think the next version will fix them.

My question is how do you handle if App1 and App2 (not multithreading but multiprocessing) shares the "desc" ? Unless

Well, device locking is a function of user space. This is Unix. If you have two uncoordinated writers to a device, you cannot expect sensible results. You can expect that the driver does not hang, of course.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

In patch "[RFC 3/5] CDC-WDM: making flush() interruptible", it is legal to return -EINTR
from close(). But I think that returning -EINTR from close() is not recommended because
it can confuse multithreaded application (retrying close() upon -EINTR is not safe).

Well, but what is the alternative? Should we ignore signals?

we return the error from write() request (i.e. give up trying to report errors from close() event), we can't pass results to the intended recipients.

That means

* harming the single threaded for the sake of the few multithreaded
* it would not work for O_NONBLOCK
* if you use a device from multiple threads or tasks, locking is your problem

Is there something we can do in flush()?

Regards
Oliver

#10 lore raw
Quoted 7 lines · Oliver Neukum
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

In patch "[RFC 3/5] CDC-WDM: making flush() interruptible", it is legal to return -EINTR
from close(). But I think that returning -EINTR from close() is not recommended because
it can confuse multithreaded application (retrying close() upon -EINTR is not safe).

Well, but what is the alternative? Should we ignore signals?

Quoted 6 lines
Quoted 2 lines

we return the error from write() request (i.e. give up trying to report errors from close() event), we can't pass results to the intended recipients.

That means

* harming the single threaded for the sake of the few multithreaded

What is wrong with the single threaded user? As I describe below, there is no guarantee that wdm_write() can complete immediately (even if O_NONBLOCK flag is set).

Quoted 1 line

* it would not work for O_NONBLOCK

It does work for O_NONBLOCK. Please see my proposal at
https://lkml.kernel.org/r/[email protected] . Since my proposal calls mutex_unlock() before start waiting for response, my proposal does not block mutex_lock_interruptible() from O_NONBLOCK write().

O_NONBLOCK cannot guarantee that "we do not wait for memory allocation request by memdup_user()/copy_from_user()/usb_submit_urb(GFP_KERNEL)". It is possible that O_NONBLOCK write() is blocked for many _minutes_ at (at least) these three locations.

O_NONBLOCK only guarantees that "it does not wait when we can't start immediately".

Quoted 2 lines

* if you use a device from multiple threads or tasks, locking is your problem

You mean "a device" as "struct wdm_device *desc" ? Then, it does not matter because the mutex, buffer, status etc. are per "struct wdm_device *desc" objects. Nobody will be disturbed by somebody else using different "struct wdm_device *desc".

Quoted 2 lines

Is there something we can do in flush()?

I consider that wdm_flush() is a wrong place to return an error. It is possible that a userspace process reaches wdm_flush() due to being killed by SIGKILL (rather than via calling close() syscall). Then, that userspace process will never receive the error fetched from wdm_flush(). Also, if that userspace process is killed by the OOM killer, being able to terminate and release resources as soon as possible is more preferable than try to wait for response.

#11 lore raw

Am Dienstag, den 15.09.2020, 19:30 +0900 schrieb Tetsuo Handa:

Quoted 5 lines

On 2020/09/15 18:14, Oliver Neukum wrote

Quoted 1 line

Is there something we can do in flush()?

I consider that wdm_flush() is a wrong place to return an error. It is possible that

I am afraid that is a basic problem we need to resolve. As I understand
it, flush() as a method exists precisely to report errors. Otherwise
you could implement it in release(). But this is not called for every close().

Hence a driver is supposed to start IO upon write() and report the result to the next call, which can be either write() or close(), the latter corresponding to flush().

Quoted 2 lines

a userspace process reaches wdm_flush() due to being killed by SIGKILL (rather than via calling close() syscall). Then, that userspace process will never receive the error

If you are killed by a signal you are in a race condition anyway. It cannot be handled.

Quoted 3 lines

fetched from wdm_flush(). Also, if that userspace process is killed by the OOM killer, being able to terminate and release resources as soon as possible is more preferable than try to wait for response.

Right, so should we just proceed in case of a dieing task? How do we do that?

Regards
Oliver

#12 lore raw
Quoted 11 lines · Oliver Neukum

Am Dienstag, den 15.09.2020, 19:30 +0900 schrieb Tetsuo Handa:

Quoted 5 lines

On 2020/09/15 18:14, Oliver Neukum wrote

Quoted 1 line

Is there something we can do in flush()?

I consider that wdm_flush() is a wrong place to return an error. It is possible that

I am afraid that is a basic problem we need to resolve. As I understand
it, flush() as a method exists precisely to report errors. Otherwise
you could implement it in release(). But this is not called for every close().

I think fsync() or ioctl() is a better method for reporting errors.

But anyway I consider that any error from N'th write() request should be returned by N'th write() request.

Quoted 4 lines

Hence a driver is supposed to start IO upon write() and report the result to the next call, which can be either write() or close(), the latter corresponding to flush().

Whether N'th write() request succeeded remains unknown until (N+1)'th write() request or close() request is issued? That sounds a strange design.

If there is nothing more to write(), how userspace process knows whether N'th write() request succeeded? Wait for writability using poll() ?

How do we guarantee that N'th write() request already set desc->werr before (N+1)'th next write() request is issued? If (N+1)'th write() request reached memdup_user() before desc->werr is set by callback of N'th write() request, (N+1)'th write() request will fail to report the error from N'th write() request. Yes, that error would be reported by (N+2)'th write() request, but the userspace process might have already discarded data needed for taking some actions (e.g. print error messages, retry the write() request with same argument).

Quoted 6 lines
Quoted 3 lines

a userspace process reaches wdm_flush() due to being killed by SIGKILL (rather than via calling close() syscall). Then, that userspace process will never receive the error

If you are killed by a signal you are in a race condition anyway. It cannot be handled.

What is the purpose of sending the error to the userspace process via write() or close()? Isn't the purpose to allow userspace process to do something (e.g. print error messages, retry the write() request with same argument) ? If close() returned an error, it might be too late to retry the write() request with same argument.

Quoted 7 lines
Quoted 4 lines

fetched from wdm_flush(). Also, if that userspace process is killed by the OOM killer, being able to terminate and release resources as soon as possible is more preferable than try to wait for response.

Right, so should we just proceed in case of a dieing task? How do we do that?

We need to avoid waiting at wdm_flush() if killed.

And I think that wdm_flush() is a strange interface for reporting the error.

#13 lore raw

Am Mittwoch, den 16.09.2020, 20:14 +0900 schrieb Tetsuo Handa:

Quoted 5 lines · Oliver Neukum
Quoted 4 lines

Am Dienstag, den 15.09.2020, 19:30 +0900 schrieb Tetsuo Handa:

Quoted 3 lines

On 2020/09/15 18:14, Oliver Neukum wrote

Quoted 1 line

Is there something we can do in flush()?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I consider that wdm_flush() is a wrong place to return an error. It is possible that

I am afraid that is a basic problem we need to resolve. As I understand
it, flush() as a method exists precisely to report errors. Otherwise
you could implement it in release(). But this is not called for every close().

I think fsync() or ioctl() is a better method for reporting errors.

Very well, I am implementing fsync(). However, I must say that the very existance of fsync() tells us that write() is not expected to push data out to devices and hence report results.

Quoted 2 lines

Whether N'th write() request succeeded remains unknown until (N+1)'th write() request or close() request is issued? That sounds a strange design.

Welcome to the world of Unix. This is necessary if you want to block as rarely as possible.

Quoted 2 lines

If there is nothing more to write(), how userspace process knows whether N'th write() request succeeded? Wait for writability using poll() ?

Technically fsync(). poll() will tell you that you can write again. That is a related but different concept.

Quoted 1 line

What is the purpose of sending the error to the userspace process via write() or close()?

Yes. However to do so, user space must be running. That the death of a process interferes with error handling is independent from that.

Quoted 1 line

And I think that wdm_flush() is a strange interface for reporting the error.

Well, POSIX is what it is.The close() syscall is supposed to return errors. Hence flush() must report them.

Regards
Oliver

#14 lore raw
Quoted 13 lines · Oliver Neukum

Am Mittwoch, den 16.09.2020, 20:14 +0900 schrieb Tetsuo Handa:

Quoted 12 lines · Oliver Neukum
Quoted 10 lines

Am Dienstag, den 15.09.2020, 19:30 +0900 schrieb Tetsuo Handa:

Quoted 4 lines

On 2020/09/15 18:14, Oliver Neukum wrote

Quoted 1 line

Is there something we can do in flush()?

I consider that wdm_flush() is a wrong place to return an error. It is possible that

I am afraid that is a basic problem we need to resolve. As I understand
it, flush() as a method exists precisely to report errors. Otherwise
you could implement it in release(). But this is not called for every close().

Quoted 5 lines
Quoted 1 line

I think fsync() or ioctl() is a better method for reporting errors.

Very well, I am implementing fsync(). However, I must say that the very existance of fsync() tells us that write() is not expected to push data out to devices and hence report results.

If you ask userspace programs to be updated to call fsync(), we can ask userspace programs be updated to call ioctl().

I expected you to implement wdm_ioctl() for fetching last error code. Then,

Quoted 6 lines
Quoted 3 lines

Whether N'th write() request succeeded remains unknown until (N+1)'th write() request or close() request is issued? That sounds a strange design.

Welcome to the world of Unix. This is necessary if you want to block as rarely as possible.

we can apply my proposal (wait for response at wdm_write() by default) as a baseline for not to break existing userspace programs (except latency), followed by a patch which allows userspace programs to use that wdm_ioctl() in order not to wait for response at wdm_write(), which is enabled by calling wdm_ioctl() (in order to recover latency caused by waiting for response at wdm_write()).

Quoted 4 lines
Quoted 1 line

What is the purpose of sending the error to the userspace process via write() or close()?

Yes. However to do so, user space must be running. That the death of a process interferes with error handling is independent from that.

Why need to send the error to the userspace process when that process was killed? My question

Isn't the purpose to allow userspace process to do something (e.g. print error messages, retry the write() request with same argument) ? If close() returned an error, it might be too late to retry the write() request with same argument.

which is unrelated to the userspace process being killed. My suggestion is that we can send the error from wdm_write() (for synchronous mode) or wdm_ioctl() (for asynchronous mode).

Quoted 5 lines
Quoted 2 lines

And I think that wdm_flush() is a strange interface for reporting the error.

Well, POSIX is what it is.The close() syscall is supposed to return errors. Hence flush() must report them.

If we check the error at wdm_write() or wdm_ioctl(), there is no error to report at wdm_flush(). Therefore, we can remove wdm_flush() completely.

I can't read this series without squashing into single patch. Making changes which will be after all removed in [RFC 5/7] is sad. Please do [RFC 5/7] before [RFC 4/7]. Then, you won't need to make unneeded modifications. I'd like to see one cleanup patch, one possible unsafe dereference fix patch, and one deadlock avoidance patch.

You did not answer to

How do we guarantee that N'th write() request already set desc->werr before (N+1)'th next write() request is issued? If (N+1)'th write() request reached memdup_user() before desc->werr is set by callback of N'th write() request, (N+1)'th write() request will fail to report the error from N'th write() request. Yes, that error would be reported by (N+2)'th write() request, but the userspace process might have already discarded data needed for taking some actions (e.g. print error messages, retry the write() request with same argument).

. At least I think that

        spin_lock_irq(&desc->iuspin);
        we = desc->werr;
        desc->werr = 0;
        spin_unlock_irq(&desc->iuspin);
        if (we < 0)
                return usb_translate_errors(we);

in wdm_write() should be moved to after !test_bit(WDM_IN_USE, &desc->flags).

In [RFC 2/7], I think that

                /* in case flush() had timed out */
                usb_kill_urb(desc->command);

which is called only when desc->count == 0 in wdm_open() is pointless, for since desc->count is incremented/decremented with wdm_mutex held, kill_urbs(desc) which is called when desc->count == 0 in wdm_release() must have already called usb_kill_urb(desc->command) before allowing wdm_open() to reach there.

In addition, is

        /* using write lock to protect desc->count */
        mutex_lock(&desc->wlock);

required? Isn't wdm_mutex that is actually protecting desc->count from modification? If it is desc->wlock that is actually protecting desc->count, the !desc->count check in wdm_release() and the desc->count == 1 check in wdm_open() have to be done with desc->wlock held.

In [RFC 3/7], patch description says

There is no need for flush() to be uninterruptible. close(2) is allowed to return -EAGAIN. Change it.

but the code does

	if (rv < 0)
		return -EINTR;

. Which error code do you want to use? (I still prefer removing wdm_flush() completely...)

#15 lore raw

Am Donnerstag, den 17.09.2020, 20:24 +0900 schrieb Tetsuo Handa:

Quoted 7 lines · Oliver Neukum
Quoted 1 line

Am Mittwoch, den 16.09.2020, 20:14 +0900 schrieb Tetsuo Handa:

If you ask userspace programs to be updated to call fsync(), we can ask userspace programs be updated to call ioctl().

I expected you to implement wdm_ioctl() for fetching last error code. Then,

Again, we are not redefining APIs. The APIs for character devices are well defined by POSIX. Please see the man pages. Introducing a whole new ioctl() is out of question.

The API and its semantics are clear. Write schedules a write:

A successful return from write() does not make any guarantee that data has been committed to disk. On some filesystems, including NFS, it does not even guarantee that space has successfully been reserved for the data. In this case, some errors might be delayed until a future write(2), fsync(2), or even close(2). The only way to be sure is to call fsync(2) after you are done writing all your data.

Fsync flushes data:

fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) so that all changed information can be retrieved even if the system crashes or is rebooted. This includes writing through or flushing a disk cache if present. The call blocks until the device reports that the transfer has completed.

If user space does not call fsync(), the error is supposed to be reported by the next write() and if there is no next write(), close() shall report it.

Quoted 5 lines

we can apply my proposal (wait for response at wdm_write() by default) as a baseline for not to break existing userspace programs (except latency), followed by a patch which allows userspace programs to use that wdm_ioctl() in order not to wait for response at wdm_write(), which is enabled by calling wdm_ioctl() (in order to recover latency caused by waiting for response at wdm_write()).

I am sorry, but the API is defined by POSIX.

Quoted 11 lines
Quoted 4 lines
Quoted 1 line

What is the purpose of sending the error to the userspace process via write() or close()?

Yes. However to do so, user space must be running. That the death of a process interferes with error handling is independent from that.

Why need to send the error to the userspace process when that process was killed? My question

Isn't the purpose to allow userspace process to do something (e.g. print error messages, retry the write() request with same argument) ? If close() returned an error, it might be too late to retry the write() request with same argument.

Yes. Technically you need to use fsync(). Hence I implemented it.

Quoted 2 lines

If we check the error at wdm_write() or wdm_ioctl(), there is no error to report at wdm_flush(). Therefore, we can remove wdm_flush() completely.

Again, the API is defined by POSIX. If user space calls write() and then close(), close() must report the error.

Quoted 2 lines

I can't read this series without squashing into single patch. Making changes which will be after all removed in [RFC 5/7] is sad. Please do [RFC 5/7] before [RFC 4/7].

Done.

Quoted 2 lines

Then, you won't need to make unneeded modifications. I'd like to see one cleanup patch, one possible unsafe dereference fix patch, and one deadlock avoidance patch.

This needs to partially go into stable. Hence fixes must come first.

Quoted 9 lines

You did not answer to

How do we guarantee that N'th write() request already set desc->werr before (N+1)'th next write() request is issued? If (N+1)'th write() request reached memdup_user() before desc->werr is set by callback of N'th write() request, (N+1)'th write() request will fail to report the error from N'th write() request. Yes, that error would be reported by (N+2)'th write() request, but the userspace process might have already discarded data needed for taking some actions (e.g. print error messages, retry the write() request with same argument).

We don't, nor do we have to. You are right, error reporting can be improved. I implemented fsync() to do so.

Quoted 10 lines

. At least I think that

        spin_lock_irq(&desc->iuspin);
        we = desc->werr;
        desc->werr = 0;
        spin_unlock_irq(&desc->iuspin);
        if (we < 0)
                return usb_translate_errors(we);

in wdm_write() should be moved to after !test_bit(WDM_IN_USE, &desc->flags).

Why?

Quoted 9 lines

In [RFC 2/7], I think that

                /* in case flush() had timed out */
                usb_kill_urb(desc->command);

which is called only when desc->count == 0 in wdm_open() is pointless, for since desc->count is incremented/decremented with wdm_mutex held, kill_urbs(desc) which is called when desc->count == 0 in wdm_release() must have already called usb_kill_urb(desc->command) before allowing wdm_open() to reach there.

You are right. I am going to remove it.

Quoted 9 lines

In addition, is

        /* using write lock to protect desc->count */
        mutex_lock(&desc->wlock);

required? Isn't wdm_mutex that is actually protecting desc->count from modification? If it is desc->wlock that is actually protecting desc->count, the !desc->count check in wdm_release() and the desc->count == 1 check in wdm_open() have to be done with desc->wlock held.

Correct. So should wdm_mutex be dropped earlier?

Quoted 11 lines

In [RFC 3/7], patch description says

There is no need for flush() to be uninterruptible. close(2) is allowed to return -EAGAIN. Change it.

but the code does

	if (rv < 0)
		return -EINTR;

. Which error code do you want to use? (I still prefer removing wdm_flush() completely...)

-EINTR. Sorry. I shall change the description.

Regards
Oliver

#16 lore raw
Quoted 4 lines · Oliver Neukum

The API and its semantics are clear. Write schedules a write:

A successful return from write() does not make any guarantee that data has been committed to disk. On some filesystems, including NFS, it does not even guarantee that space has successfully been reserved for the data. In this case, some errors might be delayed until a future write(2), fsync(2), or even close(2). The only way to be sure is to call fsync(2) after you are done writing all your data.

But I think that this leaves a room for allowing write() to imply fflush() (i.e. write() is allowed to wait for data to be committed to disk).

Quoted 8 lines

Fsync flushes data:

fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) so that all changed information can be retrieved even if the system crashes or is rebooted. This includes writing through or flushing a disk cache if present. The call blocks until the device reports that the transfer has completed.

If user space does not call fsync(), the error is supposed to be reported by the next write() and if there is no next write(), close() shall report it.

Where does "the next" (and not "the next after the next") write() come from?

Quoted 10 lines
Quoted 9 lines

You did not answer to

How do we guarantee that N'th write() request already set desc->werr before (N+1)'th next write() request is issued? If (N+1)'th write() request reached memdup_user() before desc->werr is set by callback of N'th write() request, (N+1)'th write() request will fail to report the error from N'th write() request. Yes, that error would be reported by (N+2)'th write() request, but the userspace process might have already discarded data needed for taking some actions (e.g. print error messages, retry the write() request with same argument).

Quoted 2 lines

We don't, nor do we have to. You are right, error reporting can be improved. I implemented fsync() to do so.

You are saying that if user space does not call fsync(), the error is allowed to be reported by the next after the next (in other words, (N+2)'th) write() ?

Quoted 10 lines
Quoted 10 lines

. At least I think that

        spin_lock_irq(&desc->iuspin);
        we = desc->werr;
        desc->werr = 0;
        spin_unlock_irq(&desc->iuspin);
        if (we < 0)
                return usb_translate_errors(we);
Quoted 3 lines
Quoted 1 line

in wdm_write() should be moved to after !test_bit(WDM_IN_USE, &desc->flags).

Why?

Otherwise, we can't make sure (N+1)'th write() will report error from N'th write().

Since I don't know the characteristics of data passed via wdm_write() (I guess that the data is some stateful controlling commands rather than meaningless byte stream), I guess that (N+1)'th wdm_write() attempt should be made only after confirming that N'th wdm_write() attempt received wdm_callback() response. To preserve state / data used by N'th wdm_write() attempt, reporting the error from too late write() attempt would be useless.

Quoted 11 lines
Quoted 9 lines

In addition, is

        /* using write lock to protect desc->count */
        mutex_lock(&desc->wlock);

required? Isn't wdm_mutex that is actually protecting desc->count from modification? If it is desc->wlock that is actually protecting desc->count, the !desc->count check in wdm_release() and the desc->count == 1 check in wdm_open() have to be done with desc->wlock held.

Correct. So should wdm_mutex be dropped earlier?

If recover_from_urb_loss() can tolerate stale desc->count value, wdm_mutex already protects desc->count. I don't know how this module works. I don't know whether wdm_mutex and/or desc->wlock is held when recover_from_urb_loss() is called from wdm_resume(). It seems that desc->wlock is held but wdm_mutex is not held when recover_from_urb_loss() is called from wdm_post_reset().

By the way, after the fixes, we could replace

  spin_lock_irq(&desc->iuspin);
  rv = desc->werr;
  desc->werr = 0;
  spin_unlock_irq(&desc->iuspin);

with

  rv = xchg(&desc->werr, 0);

and avoid spin_lock_irq()/spin_unlock_irq() because there are many locations which needs to check and clear the error...

#17 lore raw

Am Freitag, den 18.09.2020, 01:17 +0900 schrieb Tetsuo Handa:

Quoted 8 lines · Oliver Neukum
Quoted 4 lines

The API and its semantics are clear. Write schedules a write:

A successful return from write() does not make any guarantee that data has been committed to disk. On some filesystems, including NFS, it does not even guarantee that space has successfully been reserved for the data. In this case, some errors might be delayed until a future write(2), fsync(2), or even close(2). The only way to be sure is to call fsync(2) after you are done writing all your data.

But I think that this leaves a room for allowing write() to imply fflush() (i.e. write() is allowed to wait for data to be committed to disk).

That would be inferior and very bad for the non-blocking case.

Quoted 4 lines
Quoted 2 lines

If user space does not call fsync(), the error is supposed to be reported by the next write() and if there is no next write(), close() shall report it.

Where does "the next" (and not "the next after the next") write() come from?

We would indeed by on spec. However, we perform best if we return an error as soon as possible.

Quoted 2 lines

You are saying that if user space does not call fsync(), the error is allowed to be reported by the next after the next (in other words, (N+2)'th) write() ?

Yes. The man page is clear on that.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

. At least I think that

        spin_lock_irq(&desc->iuspin);
        we = desc->werr;
        desc->werr = 0;
        spin_unlock_irq(&desc->iuspin);
        if (we < 0)
                return usb_translate_errors(we);
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

in wdm_write() should be moved to after !test_bit(WDM_IN_USE, &desc->flags).

Why?

Otherwise, we can't make sure (N+1)'th write() will report error from N'th write().

We should move the test for reporting errors later, so that it is sure to be carried out? I am afraid I cannot follow that logic.

Quoted 6 lines

Since I don't know the characteristics of data passed via wdm_write() (I guess that the data is some stateful controlling commands rather than meaningless byte stream), I guess that (N+1)'th wdm_write() attempt should be made only after confirming that N'th wdm_write() attempt received wdm_callback() response. To preserve state / data used by N'th wdm_write() attempt, reporting the error from too late write() attempt would be useless.

We cannot make assumptions on how user space uses the driver. Somebody manually connecting and typing in commands letter by letter must also work.

We can optimize for the common case, but we must operate according to the specs.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

In addition, is

        /* using write lock to protect desc->count */
        mutex_lock(&desc->wlock);
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

required? Isn't wdm_mutex that is actually protecting desc->count from modification? If it is desc->wlock that is actually protecting desc->count, the !desc->count check in wdm_release() and the desc->count == 1 check in wdm_open() have to be done with desc->wlock held.

Correct. So should wdm_mutex be dropped earlier?

If recover_from_urb_loss() can tolerate stale desc->count value, wdm_mutex already

It cannot.

Quoted 4 lines

protects desc->count. I don't know how this module works. I don't know whether wdm_mutex and/or desc->wlock is held when recover_from_urb_loss() is called from wdm_resume(). It seems that desc->wlock is held but wdm_mutex is not held when recover_from_urb_loss() is called from wdm_post_reset().

Indeed.

Quoted 10 lines

By the way, after the fixes, we could replace

  spin_lock_irq(&desc->iuspin);
  rv = desc->werr;
  desc->werr = 0;
  spin_unlock_irq(&desc->iuspin);
Quoted 6 lines

with

  rv = xchg(&desc->werr, 0);

and avoid spin_lock_irq()/spin_unlock_irq() because there are many locations which needs to check and clear the error...

Have you checked whether this has implications on memory ordering?

Regards
Oliver

#18 lore raw
Quoted 5 lines · Oliver Neukum
Quoted 4 lines
Quoted 2 lines

If user space does not call fsync(), the error is supposed to be reported by the next write() and if there is no next write(), close() shall report it.

Where does "the next" (and not "the next after the next") write() come from?

Quoted 7 lines

We would indeed by on spec. However, we perform best if we return an error as soon as possible.

Quoted 2 lines

You are saying that if user space does not call fsync(), the error is allowed to be reported by the next after the next (in other words, (N+2)'th) write() ?

Yes. The man page is clear on that.

To understand it, I must understand why it is safe to defer error reporting.

Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

. At least I think that

        spin_lock_irq(&desc->iuspin);
        we = desc->werr;
        desc->werr = 0;
        spin_unlock_irq(&desc->iuspin);
        if (we < 0)
                return usb_translate_errors(we);
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

in wdm_write() should be moved to after !test_bit(WDM_IN_USE, &desc->flags).

Why?

Otherwise, we can't make sure (N+1)'th write() will report error from N'th write().

We should move the test for reporting errors later, so that it is sure to be carried out? I am afraid I cannot follow that logic.

Quoted 6 lines

Since I don't know the characteristics of data passed via wdm_write() (I guess that the data is some stateful controlling commands rather than meaningless byte stream), I guess that (N+1)'th wdm_write() attempt should be made only after confirming that N'th wdm_write() attempt received wdm_callback() response. To preserve state / data used by N'th wdm_write() attempt, reporting the error from too late write() attempt would be useless.

Quoted 3 lines

We cannot make assumptions on how user space uses the driver. Somebody manually connecting and typing in commands letter by letter must also work.

I'm not making any assumptions on how user space uses the driver. Up to now, your explanation makes me think that you are making assumptions on how user space uses cdc-wdm driver.

I'm querying you about characteristics of data passed to wdm_write(). Without knowing the difference between writing to cdc-wdm driver and normal file on some filesystem, I can't judge whether it is acceptable to defer reporting errors. When an error occurred when writing to normal file on some filesystem, the userspace would simply treat that file as broken (e.g. delete such file). The userspace of this cdc-wdm driver might want that any error is immediately reported, for I'm thinking that each data passed to wdm_write() is stateful, for

  /* CDC-WMC r1.1 requires wMaxCommand to be "at least 256 decimal (0x100)" */
  #define WDM_DEFAULT_BUFSIZE     256

and

        if (count > desc->wMaxCommand)
                count = desc->wMaxCommand;

makes me think that wdm_write() has to be able to handle chunk of data in atomic manner.

Quoted 20 lines

We can optimize for the common case, but we must operate according to the specs.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

In addition, is

        /* using write lock to protect desc->count */
        mutex_lock(&desc->wlock);
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

required? Isn't wdm_mutex that is actually protecting desc->count from modification? If it is desc->wlock that is actually protecting desc->count, the !desc->count check in wdm_release() and the desc->count == 1 check in wdm_open() have to be done with desc->wlock held.

Correct. So should wdm_mutex be dropped earlier?

If recover_from_urb_loss() can tolerate stale desc->count value, wdm_mutex already

Quoted 8 lines

It cannot.

Quoted 4 lines

protects desc->count. I don't know how this module works. I don't know whether wdm_mutex and/or desc->wlock is held when recover_from_urb_loss() is called from wdm_resume(). It seems that desc->wlock is held but wdm_mutex is not held when recover_from_urb_loss() is called from wdm_post_reset().

Indeed.

I don't like [RFC 8/8]. Please drop [RFC 8/8] because not only it is unrelated to hang up problem syzbot is reporting but also lock dependency is still unclear. The /* using write lock to protect desc->count */ comment is hardly helpful because wdm_disconnect() accesses desc->count with only wdm_mutex held. Detailed explanation about why releasing wdm_mutex early in wdm_open() is safe with regards to e.g. wdm_disconnect()/recover_from_urb_loss()/wdm_release() will be required for [RFC 8/8].

Also, pointless

+ /* in case flush() had timed out */
+ usb_kill_urb(desc->command);

is still in [RFC 2/8].

Please do squash into one patch, and add detailed/polite patch description like
https://lkml.kernel.org/r/1597188375-4787-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp . Current series is too fragmented to understand, and nobody can review such series.

#19 lore raw

Am Dienstag, den 22.09.2020, 10:56 +0900 schrieb Tetsuo Handa:

Quoted 4 lines · Oliver Neukum

To understand it, I must understand why it is safe to defer error reporting.

It is not. There is nothing to understand here. If user space needs a guarantee that data has been pushed out without an error, it will have to call fsync()

Let me explain.
POSIX, as described in the man page, sets some rules about what a driver must do and must not do. In other areas it makes recommendations a good driver should follow. One of them is that we block as little as possible, at the risk of delaying error reporting. A good driver will report errors as soon as possible while being compatible with doing IO asynchronously.

Quoted 3 lines

I'm querying you about characteristics of data passed to wdm_write(). Without knowing the difference between writing to cdc-wdm driver and normal file on some filesystem, I can't judge whether it is acceptable to defer reporting errors.

That is simply not a decision you or I make. The man page clearly says that it is acceptable. If user space does not like that, it must call fsync() after write().

Quoted 8 lines

When an error occurred when writing to normal file on some filesystem, the userspace would simply treat that file as broken (e.g. delete such file). The userspace of this cdc-wdm driver might want that any error is immediately reported, for I'm thinking that each data passed to wdm_write() is stateful, for

  /* CDC-WMC r1.1 requires wMaxCommand to be "at least 256 decimal (0x100)" */
  #define WDM_DEFAULT_BUFSIZE     256
Quoted 7 lines

and

        if (count > desc->wMaxCommand)
                count = desc->wMaxCommand;

makes me think that wdm_write() has to be able to handle chunk of data in atomic manner.

No. the WDM spec says the clear opposite and the man page of write says so, too.

Quoted 2 lines

I don't like [RFC 8/8]. Please drop [RFC 8/8] because not only it is unrelated to hang up problem syzbot is reporting but also lock dependency is still unclear.

Very well. I will make a patch set that clearly fixes bugs and a secondary one with optimizations.

Regards
Oliver

#20 lore raw
Quoted 8 lines · Oliver Neukum

Am Dienstag, den 22.09.2020, 10:56 +0900 schrieb Tetsuo Handa:

Quoted 2 lines · Oliver Neukum

To understand it, I must understand why it is safe to defer error reporting.

It is not. There is nothing to understand here. If user space needs a guarantee that data has been pushed out without an error, it will have to call fsync()

Quoted 7 lines
Quoted 3 lines

I'm querying you about characteristics of data passed to wdm_write(). Without knowing the difference between writing to cdc-wdm driver and normal file on some filesystem, I can't judge whether it is acceptable to defer reporting errors.

That is simply not a decision you or I make. The man page clearly says that it is acceptable. If user space does not like that, it must call fsync() after write().

Then, cdc-wdm driver did not implement fsync() was a big problem. Userspace needs to be careful not to give up upon -EINVAL when running on older kernels which did not implement wdm_fsync().

The remaining concern would be how to handle unresponding hardware, for blocking wdm_write()/wdm_read() and wdm_fsync() are using wait_event_interruptible(). If the caller do not have a mean to send a signal, the caller might hung up forever when the hardware stopped responding. Please add a comment that userspace needs to be careful when calling these functions.

#21 lore raw

Am Dienstag, den 22.09.2020, 17:34 +0900 schrieb Tetsuo Handa:

Quoted 12 lines · Oliver Neukum
Quoted 11 lines

Am Dienstag, den 22.09.2020, 10:56 +0900 schrieb Tetsuo Handa:

Quoted 2 lines · Oliver Neukum

To understand it, I must understand why it is safe to defer error reporting.

It is not. There is nothing to understand here. If user space needs a guarantee that data has been pushed out without an error, it will have to call fsync()

Quoted 3 lines

I'm querying you about characteristics of data passed to wdm_write(). Without knowing the difference between writing to cdc-wdm driver and normal file on some filesystem, I can't judge whether it is acceptable to defer reporting errors.

Quoted 7 lines
Quoted 3 lines

That is simply not a decision you or I make. The man page clearly says that it is acceptable. If user space does not like that, it must call fsync() after write().

Then, cdc-wdm driver did not implement fsync() was a big problem. Userspace needs to be careful not to give up upon -EINVAL when running on older kernels which did not implement wdm_fsync().

Very well. So I'll call the lack of fsync() a bug, which should be fixed in stable.

Quoted 5 lines

The remaining concern would be how to handle unresponding hardware, for blocking wdm_write()/wdm_read() and wdm_fsync() are using wait_event_interruptible(). If the caller do not have a mean to send a signal, the caller might hung up forever when the hardware stopped responding. Please add a comment that userspace needs to be careful when calling these functions.

wdm_flush() has such a comment. Yet no driver can make a guarantee that a device will make progress in IO. The driver must, however, provide a means of dealing with such cases. Usually that means handling signals. That is the normal semantics of a write() syscall.

I believe we are covered on that.

Regards
Oliver