lockless n_tty receive path

PATCH Superseded Peter Hurley 3 participants 237 messages Mar 19, 2013 – Jul 25, 2013 Page 2 of 2

Subsystems (3)

Network drivers Staging TTY layer and serial drivers

lore

Share & tools

lore

Peter Hurley

author

2013-04-15 15:19 UTC #119 lore raw

PATCH v3 23/24 n_tty: Special case pty flow control

In reply to Peter Hurley

The pty driver forces ldisc flow control on, regardless of available receive buffer space, so the writer can be woken whenever unthrottle is called. However, this 'forced throttle' has performance consequences, as multiple atomic operations are necessary to unthrottle and perform the write wakeup for every input line (in canonical mode).

Instead, short-circuit the unthrottle if the tty is a pty and perform the write wakeup directly.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

drivers/tty/n_tty.c +13 -0

index 00debd0..5211de2 100644
@@ -231,6 +231,8 @@ static void n_tty_write_wakeup(struct tty_struct *tty)
 
 static inline void n_tty_check_throttle(struct tty_struct *tty)
 {
+	if (tty->driver->type == TTY_DRIVER_TYPE_PTY)
+		return;
 	/*
 	 * Check the remaining room for the input canonicalization
 	 * mode.  We don't want to throttle the driver if we're in

@@ -250,6 +252,17 @@ static inline void n_tty_check_throttle(struct tty_struct *tty)
 
 static inline void n_tty_check_unthrottle(struct tty_struct *tty)
 {
+	if (tty->driver->type == TTY_DRIVER_TYPE_PTY) {
+		if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
+			return;
+		if (!tty->count)
+			return;
+		n_tty_set_room(tty);
+		n_tty_write_wakeup(tty->link);
+		wake_up_interruptible_poll(&tty->link->write_wait, POLLOUT);
+		return;
+	}
+
 	/* If there is enough space in the read buffer now, let the
 	 * low-level driver know. We use chars_in_buffer() to
 	 * check the buffer, as it now knows about canonical mode.

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-04-15 15:32 UTC #120 lore raw

PATCH v3 24/24 n_tty: Queue buffer work on any available cpu

In reply to Peter Hurley

Scheduling buffer work on the same cpu as the read() thread limits the parallelism now possible between the receive_buf path and the n_tty_read() path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/tty/n_tty.c +1 -1

index 5211de2..4dbb558 100644
@@ -198,7 +198,7 @@ static void n_tty_set_room(struct tty_struct *tty)
 		 */
 		WARN_RATELIMIT(test_bit(TTY_LDISC_HALTED, &tty->flags),
 			       "scheduling buffer work for halted ldisc\n");
-		schedule_work(&tty->port->buf.work);
+		queue_work(system_unbound_wq, &tty->port->buf.work);
 	}
 }

Signature · 1 line

1.8.1.2

Greg Kroah-Hartman

maintainer

2013-04-15 20:14 UTC #121 lore raw

In reply to Peter Hurley

Quoted 7 lines · Peter Hurley

Greg,
Unfortunately, this series is dependent on the 'ldsem patchset'. The reason is that this series abandons tty->receive_room as a flow control mechanism (because that requires locking), and the TIOCSETD ioctl _without ldsem_ uses tty->receive_room to shutoff i/o.

Can you also resend the ldsem patch series, as I no longer have it around anymore, and I don't think it was part of this big resend, right?

thanks,

greg k-h

Peter Hurley

author

2013-04-16 10:16 UTC #122 lore raw

PATCH 1/7 tty: Add timed, writer-prioritized rw semaphore

The semantics of a rw semaphore are almost ideally suited for tty line discipline lifetime management; multiple active threads obtain "references" (read locks) while performing i/o to prevent the loss or change of the current line discipline (write lock).

Unfortunately, the existing rw_semaphore is ill-suited in other ways;
1) TIOCSETD ioctl (change line discipline) expects to return an error if the line discipline cannot be exclusively locked within 5 secs. Lock wait timeouts are not supported by rwsem.
2) A tty hangup is expected to halt and scrap pending i/o, so exclusive locking must be prioritized.
Writer priority is not supported by rwsem.

Add ld_semaphore which implements these requirements in a semantically similar way to rw_semaphore.

Writer priority is handled by separate wait lists for readers and writers. Pending write waits are priortized before existing read waits and prevent further read locks.

Wait timeouts are trivially added, but obviously change the lock semantics as lock attempts can fail (but only due to timeout).

This implementation incorporates the write-lock stealing work of Michel Lespinasse <[email protected]>.

Cc Michel Lespinasse <[email protected]>

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/Makefile      |   2 +-
 drivers/tty/tty_ldsem.c   | 453 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/tty_ldisc.h |  46 +++++
 3 files changed, 500 insertions(+), 1 deletion(-)
 create mode 100644 drivers/tty/tty_ldsem.c

drivers/tty/Makefile +1 -1

index 6b78399..58ad1c0 100644
@@ -1,5 +1,5 @@
 obj-$(CONFIG_TTY)		+= tty_io.o n_tty.o tty_ioctl.o tty_ldisc.o \
-				   tty_buffer.o tty_port.o tty_mutex.o
+				   tty_buffer.o tty_port.o tty_mutex.o tty_ldsem.o
 obj-$(CONFIG_LEGACY_PTYS)	+= pty.o
 obj-$(CONFIG_UNIX98_PTYS)	+= pty.o
 obj-$(CONFIG_AUDIT)		+= tty_audit.o

drivers/tty/tty_ldsem.c new file +453 -0

453 lines not on this page · raw message

include/linux/tty_ldisc.h +46 -0

52 lines not on this page · raw message

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-04-16 10:15 UTC #123 lore raw

PATCH 2/7 tty: Add lock/unlock ldisc pair functions

Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

drivers/tty/tty_ldisc.c +87 -0

index 1afe192..ae0287f 100644
@@ -31,6 +31,13 @@
 #define tty_ldisc_debug(tty, f, args...)
 #endif
 
+/* lockdep nested classes for tty->ldisc_sem */
+enum {
+	LDISC_SEM_NORMAL,
+	LDISC_SEM_OTHER,
+};
+
+
 /*
  *	This guards the refcounted line discipline lists. The lock
  *	must be taken with irqs off because there are hangup path

@@ -351,6 +358,86 @@ void tty_ldisc_deref(struct tty_ldisc *ld)
 }
 EXPORT_SYMBOL_GPL(tty_ldisc_deref);
 
+
+static inline int __lockfunc
+tty_ldisc_lock(struct tty_struct *tty, unsigned long timeout)
+{
+	return ldsem_down_write(&tty->ldisc_sem, timeout);
+}
+
+static inline int __lockfunc
+tty_ldisc_lock_nested(struct tty_struct *tty, unsigned long timeout)
+{
+	return ldsem_down_write_nested(&tty->ldisc_sem,
+				       LDISC_SEM_OTHER, timeout);
+}
+
+static inline void tty_ldisc_unlock(struct tty_struct *tty)
+{
+	return ldsem_up_write(&tty->ldisc_sem);
+}
+
+static int __lockfunc
+tty_ldisc_lock_pair_timeout(struct tty_struct *tty, struct tty_struct *tty2,
+			    unsigned long timeout)
+{
+	int ret;
+
+	if (tty < tty2) {
+		ret = tty_ldisc_lock(tty, timeout);
+		if (ret) {
+			ret = tty_ldisc_lock_nested(tty2, timeout);
+			if (!ret)
+				tty_ldisc_unlock(tty);
+		}
+	} else {
+		/* if this is possible, it has lots of implications */
+		WARN_ON_ONCE(tty == tty2);
+		if (tty2 && tty != tty2) {
+			ret = tty_ldisc_lock(tty2, timeout);
+			if (ret) {
+				ret = tty_ldisc_lock_nested(tty, timeout);
+				if (!ret)
+					tty_ldisc_unlock(tty2);
+			}
+		} else
+			ret = tty_ldisc_lock(tty, timeout);
+	}
+
+	if (!ret)
+		return -EBUSY;
+
+	set_bit(TTY_LDISC_HALTED, &tty->flags);
+	if (tty2)
+		set_bit(TTY_LDISC_HALTED, &tty2->flags);
+	return 0;
+}
+
+static void __lockfunc
+tty_ldisc_lock_pair(struct tty_struct *tty, struct tty_struct *tty2)
+{
+	tty_ldisc_lock_pair_timeout(tty, tty2, MAX_SCHEDULE_TIMEOUT);
+}
+
+static void __lockfunc tty_ldisc_unlock_pair(struct tty_struct *tty,
+					     struct tty_struct *tty2)
+{
+	tty_ldisc_unlock(tty);
+	if (tty2)
+		tty_ldisc_unlock(tty2);
+}
+
+static void __lockfunc tty_ldisc_enable_pair(struct tty_struct *tty,
+					     struct tty_struct *tty2)
+{
+	clear_bit(TTY_LDISC_HALTED, &tty->flags);
+	if (tty2)
+		clear_bit(TTY_LDISC_HALTED, &tty2->flags);
+
+	tty_ldisc_unlock_pair(tty, tty2);
+}
+
+
 /**
  *	tty_ldisc_enable	-	allow ldisc use
  *	@tty: terminal to activate ldisc on

Signature · 1 line

1.8.1.2

Greg Kroah-Hartman

maintainer

2013-05-20 19:34 UTC #124 lore raw

Re: [PATCH 2/7] tty: Add lock/unlock ldisc pair functions

Quoted 8 lines · Peter Hurley

Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
 drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

This patch breaks the build :(

I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?

thanks,

greg k-h

Peter Hurley

author

2013-05-20 21:44 UTC #125 lore raw

Re: [PATCH 2/7] tty: Add lock/unlock ldisc pair functions

Quoted 11 lines · Greg Kroah-Hartman

Quoted 8 lines · Peter Hurley
Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
  drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 87 insertions(+)
This patch breaks the build :(

Sorry about that.

Quoted 3 lines

I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?

Maybe it would be better to reorder 7/7 to be 1/6, if that's ok? More work for me but the history will be cleaner.

Regards,
Peter

Greg Kroah-Hartman

maintainer

2013-05-20 23:06 UTC #126 lore raw

Re: [PATCH 2/7] tty: Add lock/unlock ldisc pair functions

Quoted 13 lines · Peter Hurley

Quoted 11 lines · Greg Kroah-Hartman
Quoted 8 lines · Peter Hurley
Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
 drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
This patch breaks the build :(

Quoted 8 lines

Sorry about that.
Quoted 3 lines
I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?
Maybe it would be better to reorder 7/7 to be 1/6, if that's ok? More work for me but the history will be cleaner.

Sure, that works for me as well, care to just resend them? Or, I can dig them out of my archive if needed.

thanks,

greg k-h

Peter Hurley

author

2013-05-20 23:39 UTC #127 lore raw

Re: [PATCH 2/7] tty: Add lock/unlock ldisc pair functions

Quoted 20 lines · Greg Kroah-Hartman

Quoted 19 lines · Peter Hurley
Quoted 11 lines · Greg Kroah-Hartman
Quoted 8 lines · Peter Hurley
Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
  drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 87 insertions(+)
This patch breaks the build :(
Sorry about that.
Quoted 3 lines
I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?

Quoted 5 lines

Quoted 2 lines
Maybe it would be better to reorder 7/7 to be 1/6, if that's ok? More work for me but the history will be cleaner.
Sure, that works for me as well, care to just resend them? Or, I can dig them out of my archive if needed.

I'll resend them because the rebase will require some re-editing.

Thanks,
Peter

Greg Kroah-Hartman

maintainer

2013-06-03 19:24 UTC #128 lore raw

Re: [PATCH 2/7] tty: Add lock/unlock ldisc pair functions

Quoted 21 lines · Peter Hurley

Quoted 20 lines · Greg Kroah-Hartman
Quoted 19 lines · Peter Hurley
Quoted 11 lines · Greg Kroah-Hartman
Quoted 8 lines · Peter Hurley
Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
 drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
This patch breaks the build :(
Sorry about that.
Quoted 3 lines
I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?

Quoted 7 lines

Quoted 5 lines
Quoted 2 lines
Maybe it would be better to reorder 7/7 to be 1/6, if that's ok? More work for me but the history will be cleaner.
Sure, that works for me as well, care to just resend them? Or, I can dig them out of my archive if needed.
I'll resend them because the rebase will require some re-editing.

Did you ever resend these? If so, I didn't see them :(

thanks,

greg k-h

Peter Hurley

author

2013-06-15 11:05 UTC #129 lore raw

cover ldsem patchset, reordered and rebased

Quoted 24 lines · Peter Hurley

Quoted 24 lines · Greg Kroah-Hartman
Quoted 23 lines · Peter Hurley
Quoted 21 lines · Greg Kroah-Hartman
Quoted 11 lines · Peter Hurley
Quoted 8 lines
Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
  drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 87 insertions(+)
This patch breaks the build :(
Sorry about that.
Quoted 3 lines
I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?
Maybe it would be better to reorder 7/7 to be 1/6, if that's ok? More work for me but the history will be cleaner.

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
Sure, that works for me as well, care to just resend them? Or, I can dig them out of my archive if needed.
I'll resend them because the rebase will require some re-editing.
Did you ever resend these? If so, I didn't see them :(

Greg,

Sorry for the delay here. In addition to the re-editing required, I merged in a couple of fixes and these needed re-testing.

These apply cleanly to tty-next.

Changes from previous 7-patch series:
1. In tty_ldisc_ref(), snapshot tty->ldisc holding ldsem. The conditions for this to be a fix cannot actually occur but this is better anyway.
2. In tty_set_ldisc(), the new ldisc needs to be released if tty_ldisc_lock_pair_timeout() fails.

Regards,
Peter Hurley

Peter Hurley (6):
tty: Fix tty_ldisc_lock name collision
tty: Add lock/unlock ldisc pair functions
tty: Replace ldisc locking with ldisc_sem
tty: Clarify ldisc variable
tty: Fix hangup race with TIOCSETD ioctl
tty: Clarify multiple-references comment in TIOCSETD ioctl

 drivers/tty/tty_buffer.c  |   2 +-
 drivers/tty/tty_io.c      |  11 +-
 drivers/tty/tty_ldisc.c   | 450 ++++++++++++++++------------------------------
 include/linux/tty.h       |   4 +-
 include/linux/tty_ldisc.h |   3 +-
 5 files changed, 158 insertions(+), 312 deletions(-)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 11:04 UTC #130 lore raw

PATCH 1/6 tty: Fix tty_ldisc_lock name collision

The file scope spinlock identifier, tty_ldisc_lock, will collide with the file scope lock function tty_ldisc_lock() so rename it.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_io.c    |  4 ++--
 drivers/tty/tty_ldisc.c | 46 +++++++++++++++++++++++-----------------------
 2 files changed, 25 insertions(+), 25 deletions(-)

drivers/tty/tty_io.c +2 -2

index 6464029..e7be3a5 100644
@@ -603,7 +603,7 @@ static int tty_signal_session_leader(struct tty_struct *tty, int exit_session)
  *		BTM
  *		  redirect lock for undoing redirection
  *		  file list lock for manipulating list of ttys
- *		  tty_ldisc_lock from called functions
+ *		  tty_ldiscs_lock from called functions
  *		  termios_mutex resetting termios data
  *		  tasklist_lock to walk task list for hangup event
  *		    ->siglock to protect ->signal/->sighand

@@ -2199,7 +2199,7 @@ static int tty_fasync(int fd, struct file *filp, int on)
  *	FIXME: does not honour flow control ??
  *
  *	Locking:
- *		Called functions take tty_ldisc_lock
+ *		Called functions take tty_ldiscs_lock
  *		current->signal->tty check is safe without locks
  *
  *	FIXME: may race normal receive processing

drivers/tty/tty_ldisc.c +23 ~~-23~~

173 lines not on this page · raw message

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 11:05 UTC #131 lore raw

PATCH 2/6 tty: Add lock/unlock ldisc pair functions

In reply to Peter Hurley

Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

drivers/tty/tty_ldisc.c +87 -0

index 8166260..418c9f6 100644
@@ -31,6 +31,13 @@
 #define tty_ldisc_debug(tty, f, args...)
 #endif
 
+/* lockdep nested classes for tty->ldisc_sem */
+enum {
+	LDISC_SEM_NORMAL,
+	LDISC_SEM_OTHER,
+};
+
+
 /*
  *	This guards the refcounted line discipline lists. The lock
  *	must be taken with irqs off because there are hangup path

@@ -351,6 +358,86 @@ void tty_ldisc_deref(struct tty_ldisc *ld)
 }
 EXPORT_SYMBOL_GPL(tty_ldisc_deref);
 
+
+static inline int __lockfunc
+tty_ldisc_lock(struct tty_struct *tty, unsigned long timeout)
+{
+	return ldsem_down_write(&tty->ldisc_sem, timeout);
+}
+
+static inline int __lockfunc
+tty_ldisc_lock_nested(struct tty_struct *tty, unsigned long timeout)
+{
+	return ldsem_down_write_nested(&tty->ldisc_sem,
+				       LDISC_SEM_OTHER, timeout);
+}
+
+static inline void tty_ldisc_unlock(struct tty_struct *tty)
+{
+	return ldsem_up_write(&tty->ldisc_sem);
+}
+
+static int __lockfunc
+tty_ldisc_lock_pair_timeout(struct tty_struct *tty, struct tty_struct *tty2,
+			    unsigned long timeout)
+{
+	int ret;
+
+	if (tty < tty2) {
+		ret = tty_ldisc_lock(tty, timeout);
+		if (ret) {
+			ret = tty_ldisc_lock_nested(tty2, timeout);
+			if (!ret)
+				tty_ldisc_unlock(tty);
+		}
+	} else {
+		/* if this is possible, it has lots of implications */
+		WARN_ON_ONCE(tty == tty2);
+		if (tty2 && tty != tty2) {
+			ret = tty_ldisc_lock(tty2, timeout);
+			if (ret) {
+				ret = tty_ldisc_lock_nested(tty, timeout);
+				if (!ret)
+					tty_ldisc_unlock(tty2);
+			}
+		} else
+			ret = tty_ldisc_lock(tty, timeout);
+	}
+
+	if (!ret)
+		return -EBUSY;
+
+	set_bit(TTY_LDISC_HALTED, &tty->flags);
+	if (tty2)
+		set_bit(TTY_LDISC_HALTED, &tty2->flags);
+	return 0;
+}
+
+static void __lockfunc
+tty_ldisc_lock_pair(struct tty_struct *tty, struct tty_struct *tty2)
+{
+	tty_ldisc_lock_pair_timeout(tty, tty2, MAX_SCHEDULE_TIMEOUT);
+}
+
+static void __lockfunc tty_ldisc_unlock_pair(struct tty_struct *tty,
+					     struct tty_struct *tty2)
+{
+	tty_ldisc_unlock(tty);
+	if (tty2)
+		tty_ldisc_unlock(tty2);
+}
+
+static void __lockfunc tty_ldisc_enable_pair(struct tty_struct *tty,
+					     struct tty_struct *tty2)
+{
+	clear_bit(TTY_LDISC_HALTED, &tty->flags);
+	if (tty2)
+		clear_bit(TTY_LDISC_HALTED, &tty2->flags);
+
+	tty_ldisc_unlock_pair(tty, tty2);
+}
+
+
 /**
  *	tty_ldisc_enable	-	allow ldisc use
  *	@tty: terminal to activate ldisc on

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 11:04 UTC #132 lore raw

PATCH 3/6 tty: Replace ldisc locking with ldisc_sem

In reply to Peter Hurley

Line discipline locking was performed with a combination of a mutex, a status bit, a count, and a waitqueue -- basically, a rw semaphore.

Replace the existing combination with an ld_semaphore.

Fixes:
1) the 'reference acquire after ldisc locked' bug
2) the over-complicated halt mechanism
3) lock order wrt. tty_lock()
4) dropping locks while changing ldisc
5) previously unidentified deadlock while locking ldisc from both linked ttys concurrently
6) previously unidentified recursive deadlocks

Adds much-needed lockdep diagnostics.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c  |   2 +-
 drivers/tty/tty_io.c      |   7 +-
 drivers/tty/tty_ldisc.c   | 329 +++++++---------------------------------------
 include/linux/tty.h       |   4 +-
 include/linux/tty_ldisc.h |   3 +-
 5 files changed, 52 insertions(+), 293 deletions(-)

drivers/tty/tty_buffer.c +1 -1

index 9121c1f..a42a028 100644
@@ -429,7 +429,7 @@ static void flush_to_ldisc(struct work_struct *work)
 		return;
 
 	disc = tty_ldisc_ref(tty);
-	if (disc == NULL)	/*  !TTY_LDISC */
+	if (disc == NULL)
 		return;
 
 	spin_lock_irqsave(&buf->lock, flags);

drivers/tty/tty_io.c +3 -4

index e7be3a5..5ee51ba 100644
@@ -1388,8 +1388,7 @@ static int tty_reopen(struct tty_struct *tty)
 	struct tty_driver *driver = tty->driver;
 
 	if (test_bit(TTY_CLOSING, &tty->flags) ||
-			test_bit(TTY_HUPPING, &tty->flags) ||
-			test_bit(TTY_LDISC_CHANGING, &tty->flags))
+			test_bit(TTY_HUPPING, &tty->flags))
 		return -EIO;
 
 	if (driver->type == TTY_DRIVER_TYPE_PTY &&

@@ -1405,7 +1404,7 @@ static int tty_reopen(struct tty_struct *tty)
 	}
 	tty->count++;
 
-	WARN_ON(!test_bit(TTY_LDISC, &tty->flags));
+	WARN_ON(!tty->ldisc);
 
 	return 0;
 }

@@ -3014,7 +3013,7 @@ void initialize_tty_struct(struct tty_struct *tty,
 	tty->pgrp = NULL;
 	mutex_init(&tty->legacy_mutex);
 	mutex_init(&tty->termios_mutex);
-	mutex_init(&tty->ldisc_mutex);
+	init_ldsem(&tty->ldisc_sem);
 	init_waitqueue_head(&tty->write_wait);
 	init_waitqueue_head(&tty->read_wait);
 	INIT_WORK(&tty->hangup_work, do_tty_hangup);

drivers/tty/tty_ldisc.c +46 ~~-283~~

539 lines not on this page · raw message

include/linux/tty.h +1 -3

index 82ab69b..a665c7e 100644
@@ -238,7 +238,7 @@ struct tty_struct {
 	int index;
 
 	/* Protects ldisc changes: Lock tty not pty */
-	struct mutex ldisc_mutex;
+	struct ld_semaphore ldisc_sem;
 	struct tty_ldisc *ldisc;
 
 	struct mutex atomic_write_lock;

@@ -306,8 +306,6 @@ struct tty_file_private {
 #define TTY_DO_WRITE_WAKEUP 	5	/* Call write_wakeup after queuing new */
 #define TTY_PUSH 		6	/* n_tty private */
 #define TTY_CLOSING 		7	/* ->close() in progress */
-#define TTY_LDISC 		9	/* Line discipline attached */
-#define TTY_LDISC_CHANGING 	10	/* Line discipline changing */
 #define TTY_LDISC_OPEN	 	11	/* Line discipline is open */
 #define TTY_PTY_LOCK 		16	/* pty private */
 #define TTY_NO_WRITE_SPLIT 	17	/* Preserve write boundaries to driver */

include/linux/tty_ldisc.h +1 -2

index 7b24bbd..22ee107 100644
@@ -197,8 +197,7 @@ struct tty_ldisc_ops {
 
 struct tty_ldisc {
 	struct tty_ldisc_ops *ops;
-	atomic_t users;
-	wait_queue_head_t wq_idle;
+	struct tty_struct *tty;
 };
 
 #define TTY_LDISC_MAGIC	0x5403

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 11:04 UTC #133 lore raw

PATCH 4/6 tty: Clarify ldisc variable

In reply to Peter Hurley

Rename o_ldisc to avoid confusion with the ldisc of the 'other' tty.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

drivers/tty/tty_ldisc.c +7 -7

index a4fd3a1..6c63b73 100644
@@ -517,7 +517,7 @@ static void tty_ldisc_restore(struct tty_struct *tty, struct tty_ldisc *old)
 int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 {
 	int retval;
-	struct tty_ldisc *o_ldisc, *new_ldisc;
+	struct tty_ldisc *old_ldisc, *new_ldisc;
 	struct tty_struct *o_tty = tty->link;
 
 	new_ldisc = tty_ldisc_get(tty, ldisc);

@@ -543,7 +543,7 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 	/* FIXME: why 'shutoff' input if the ldisc is locked? */
 	tty->receive_room = 0;
 
-	o_ldisc = tty->ldisc;
+	old_ldisc = tty->ldisc;
 	tty_lock(tty);
 
 	/* FIXME: for testing only */

@@ -558,8 +558,8 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 		return -EIO;
 	}
 
-	/* Shutdown the current discipline. */
-	tty_ldisc_close(tty, o_ldisc);
+	/* Shutdown the old discipline. */
+	tty_ldisc_close(tty, old_ldisc);
 
 	/* Now set up the new line discipline. */
 	tty->ldisc = new_ldisc;

@@ -569,17 +569,17 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 	if (retval < 0) {
 		/* Back to the old one or N_TTY if we can't */
 		tty_ldisc_put(new_ldisc);
-		tty_ldisc_restore(tty, o_ldisc);
+		tty_ldisc_restore(tty, old_ldisc);
 	}
 
 	/* At this point we hold a reference to the new ldisc and a
 	   a reference to the old ldisc. If we ended up flipping back
 	   to the existing ldisc we have two references to it */
 
-	if (tty->ldisc->ops->num != o_ldisc->ops->num && tty->ops->set_ldisc)
+	if (tty->ldisc->ops->num != old_ldisc->ops->num && tty->ops->set_ldisc)
 		tty->ops->set_ldisc(tty);
 
-	tty_ldisc_put(o_ldisc);
+	tty_ldisc_put(old_ldisc);
 
 	/*
 	 *	Allow ldisc referencing to occur again

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 11:04 UTC #134 lore raw

PATCH 5/6 tty: Fix hangup race with TIOCSETD ioctl

In reply to Peter Hurley

The hangup may already have happened; check for that state also.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

drivers/tty/tty_ldisc.c +2 -4

index 6c63b73..08b8a1b 100644
@@ -546,10 +546,8 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 	old_ldisc = tty->ldisc;
 	tty_lock(tty);
 
-	/* FIXME: for testing only */
-	WARN_ON(test_bit(TTY_HUPPED, &tty->flags));
-
-	if (test_bit(TTY_HUPPING, &tty->flags)) {
+	if (test_bit(TTY_HUPPING, &tty->flags) ||
+	    test_bit(TTY_HUPPED, &tty->flags)) {
 		/* We were raced by the hangup method. It will have stomped
 		   the ldisc data and closed the ldisc down */
 		tty_ldisc_enable_pair(tty, o_tty);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 11:04 UTC #135 lore raw

PATCH 6/6 tty: Clarify multiple-references comment in TIOCSETD ioctl

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

drivers/tty/tty_ldisc.c +6 -4

index 08b8a1b..edeb1dd 100644
@@ -570,13 +570,15 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 		tty_ldisc_restore(tty, old_ldisc);
 	}
 
-	/* At this point we hold a reference to the new ldisc and a
-	   a reference to the old ldisc. If we ended up flipping back
-	   to the existing ldisc we have two references to it */
-
 	if (tty->ldisc->ops->num != old_ldisc->ops->num && tty->ops->set_ldisc)
 		tty->ops->set_ldisc(tty);
 
+	/* At this point we hold a reference to the new ldisc and a
+	   reference to the old ldisc, or we hold two references to
+	   the old ldisc (if it was restored as part of error cleanup
+	   above). In either case, releasing a single reference from
+	   the old ldisc is correct. */
+
 	tty_ldisc_put(old_ldisc);
 
 	/*

Signature · 1 line

1.8.1.2

Greg Kroah-Hartman

maintainer

2013-07-23 23:44 UTC #136 lore raw

Re: [PATCH 0/6] ldsem patchset, reordered and rebased

In reply to Peter Hurley

Quoted 37 lines · Peter Hurley

Quoted 24 lines · Peter Hurley
Quoted 24 lines · Greg Kroah-Hartman
Quoted 23 lines · Peter Hurley
Quoted 21 lines · Greg Kroah-Hartman
Quoted 11 lines · Peter Hurley
Quoted 8 lines
Just as the tty pair must be locked in a stable sequence (ie, independent of which is consider the 'other' tty), so must the ldisc pair be locked in a stable sequence as well.
Signed-off-by Peter Hurley <[email protected]>
---
  drivers/tty/tty_ldisc.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 87 insertions(+)
This patch breaks the build :(
Sorry about that.
Quoted 3 lines
I've taken the first one, care to mush this one with the 7/7 patch, so there are no build breaks, and resend the 5 resulting patches so I can apply these?
Maybe it would be better to reorder 7/7 to be 1/6, if that's ok? More work for me but the history will be cleaner.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Sure, that works for me as well, care to just resend them? Or, I can dig them out of my archive if needed.
I'll resend them because the rebase will require some re-editing.
Did you ever resend these? If so, I didn't see them :(
Greg,
Sorry for the delay here. In addition to the re-editing required, I merged in a couple of fixes and these needed re-testing.

Quoted 8 lines

These apply cleanly to tty-next.
Changes from previous 7-patch series:
1. In tty_ldisc_ref(), snapshot tty->ldisc holding ldsem. The conditions for this to be a fix cannot actually occur but this is better anyway.
2. In tty_set_ldisc(), the new ldisc needs to be released if tty_ldisc_lock_pair_timeout() fails.

Now applied, thanks.

greg k-h

Peter Hurley

author

2013-04-16 10:16 UTC #137 lore raw

PATCH 3/7 tty: Replace ldisc locking with ldisc_sem

In reply to Peter Hurley

Line discipline locking was performed with a combination of a mutex, a status bit, a count, and a waitqueue -- basically, a rw semaphore.

Replace the existing combination with an ld_semaphore.

Adds much-needed lockdep diagnostics.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c  |   2 +-
 drivers/tty/tty_io.c      |   7 +-
 drivers/tty/tty_ldisc.c   | 324 ++++++----------------------------------------
 include/linux/tty.h       |   4 +-
 include/linux/tty_ldisc.h |   3 +-
 5 files changed, 48 insertions(+), 292 deletions(-)

drivers/tty/tty_buffer.c +1 -1

index 578aa75..8e8d730 100644
@@ -429,7 +429,7 @@ static void flush_to_ldisc(struct work_struct *work)
 		return;
 
 	disc = tty_ldisc_ref(tty);
-	if (disc == NULL)	/*  !TTY_LDISC */
+	if (disc == NULL)
 		return;
 
 	spin_lock_irqsave(&buf->lock, flags);

drivers/tty/tty_io.c +3 -4

index 2f77af4..06f0e25 100644
@@ -1326,8 +1326,7 @@ static int tty_reopen(struct tty_struct *tty)
 	struct tty_driver *driver = tty->driver;
 
 	if (test_bit(TTY_CLOSING, &tty->flags) ||
-			test_bit(TTY_HUPPING, &tty->flags) ||
-			test_bit(TTY_LDISC_CHANGING, &tty->flags))
+			test_bit(TTY_HUPPING, &tty->flags))
 		return -EIO;
 
 	if (driver->type == TTY_DRIVER_TYPE_PTY &&

@@ -1343,7 +1342,7 @@ static int tty_reopen(struct tty_struct *tty)
 	}
 	tty->count++;
 
-	WARN_ON(!test_bit(TTY_LDISC, &tty->flags));
+	WARN_ON(!tty->ldisc);
 
 	return 0;
 }

@@ -2952,7 +2951,7 @@ void initialize_tty_struct(struct tty_struct *tty,
 	tty->pgrp = NULL;
 	mutex_init(&tty->legacy_mutex);
 	mutex_init(&tty->termios_mutex);
-	mutex_init(&tty->ldisc_mutex);
+	init_ldsem(&tty->ldisc_sem);
 	init_waitqueue_head(&tty->write_wait);
 	init_waitqueue_head(&tty->read_wait);
 	INIT_WORK(&tty->hangup_work, do_tty_hangup);

drivers/tty/tty_ldisc.c +42 ~~-282~~

534 lines not on this page · raw message

include/linux/tty.h +1 -3

index bfa6fca..2c109a3 100644
@@ -238,7 +238,7 @@ struct tty_struct {
 	int index;
 
 	/* Protects ldisc changes: Lock tty not pty */
-	struct mutex ldisc_mutex;
+	struct ld_semaphore ldisc_sem;
 	struct tty_ldisc *ldisc;
 
 	struct mutex atomic_write_lock;

@@ -306,8 +306,6 @@ struct tty_file_private {
 #define TTY_DO_WRITE_WAKEUP 	5	/* Call write_wakeup after queuing new */
 #define TTY_PUSH 		6	/* n_tty private */
 #define TTY_CLOSING 		7	/* ->close() in progress */
-#define TTY_LDISC 		9	/* Line discipline attached */
-#define TTY_LDISC_CHANGING 	10	/* Line discipline changing */
 #define TTY_LDISC_OPEN	 	11	/* Line discipline is open */
 #define TTY_HW_COOK_OUT 	14	/* Hardware can do output cooking */
 #define TTY_HW_COOK_IN 		15	/* Hardware can do input cooking */

include/linux/tty_ldisc.h +1 -2

index ca000fc..272075e 100644
@@ -197,8 +197,7 @@ struct tty_ldisc_ops {
 
 struct tty_ldisc {
 	struct tty_ldisc_ops *ops;
-	atomic_t users;
-	wait_queue_head_t wq_idle;
+	struct tty_struct *tty;
 };
 
 #define TTY_LDISC_MAGIC	0x5403

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-04-16 10:15 UTC #138 lore raw

PATCH 4/7 tty: Clarify ldisc variable

In reply to Peter Hurley

Rename o_ldisc to avoid confusion with the ldisc of the 'other' tty.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

drivers/tty/tty_ldisc.c +7 -7

index a150f95..9ace119 100644
@@ -516,7 +516,7 @@ static void tty_ldisc_restore(struct tty_struct *tty, struct tty_ldisc *old)
 int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 {
 	int retval;
-	struct tty_ldisc *o_ldisc, *new_ldisc;
+	struct tty_ldisc *old_ldisc, *new_ldisc;
 	struct tty_struct *o_tty = tty->link;
 
 	new_ldisc = tty_ldisc_get(tty, ldisc);

@@ -540,7 +540,7 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 	/* FIXME: why 'shutoff' input if the ldisc is locked? */
 	tty->receive_room = 0;
 
-	o_ldisc = tty->ldisc;
+	old_ldisc = tty->ldisc;
 	tty_lock(tty);
 
 	/* FIXME: for testing only */

@@ -555,8 +555,8 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 		return -EIO;
 	}
 
-	/* Shutdown the current discipline. */
-	tty_ldisc_close(tty, o_ldisc);
+	/* Shutdown the old discipline. */
+	tty_ldisc_close(tty, old_ldisc);
 
 	/* Now set up the new line discipline. */
 	tty->ldisc = new_ldisc;

@@ -566,17 +566,17 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 	if (retval < 0) {
 		/* Back to the old one or N_TTY if we can't */
 		tty_ldisc_put(new_ldisc);
-		tty_ldisc_restore(tty, o_ldisc);
+		tty_ldisc_restore(tty, old_ldisc);
 	}
 
 	/* At this point we hold a reference to the new ldisc and a
 	   a reference to the old ldisc. If we ended up flipping back
 	   to the existing ldisc we have two references to it */
 
-	if (tty->ldisc->ops->num != o_ldisc->ops->num && tty->ops->set_ldisc)
+	if (tty->ldisc->ops->num != old_ldisc->ops->num && tty->ops->set_ldisc)
 		tty->ops->set_ldisc(tty);
 
-	tty_ldisc_put(o_ldisc);
+	tty_ldisc_put(old_ldisc);
 
 	/*
 	 *	Allow ldisc referencing to occur again

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-04-16 10:15 UTC #139 lore raw

PATCH 5/7 tty: Fix hangup race with TIOCSETD ioctl

In reply to Peter Hurley

The hangup may already have happened; check for that state also.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

drivers/tty/tty_ldisc.c +2 -4

index 9ace119..84ba790 100644
@@ -543,10 +543,8 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 	old_ldisc = tty->ldisc;
 	tty_lock(tty);
 
-	/* FIXME: for testing only */
-	WARN_ON(test_bit(TTY_HUPPED, &tty->flags));
-
-	if (test_bit(TTY_HUPPING, &tty->flags)) {
+	if (test_bit(TTY_HUPPING, &tty->flags) ||
+	    test_bit(TTY_HUPPED, &tty->flags)) {
 		/* We were raced by the hangup method. It will have stomped
 		   the ldisc data and closed the ldisc down */
 		tty_ldisc_enable_pair(tty, o_tty);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-04-16 10:15 UTC #140 lore raw

PATCH 6/7 tty: Clarify multiple-references comment in TIOCSETD ioctl

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

drivers/tty/tty_ldisc.c +6 -4

index 84ba790..9725c94 100644
@@ -567,13 +567,15 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 		tty_ldisc_restore(tty, old_ldisc);
 	}
 
-	/* At this point we hold a reference to the new ldisc and a
-	   a reference to the old ldisc. If we ended up flipping back
-	   to the existing ldisc we have two references to it */
-
 	if (tty->ldisc->ops->num != old_ldisc->ops->num && tty->ops->set_ldisc)
 		tty->ops->set_ldisc(tty);
 
+	/* At this point we hold a reference to the new ldisc and a
+	   reference to the old ldisc, or we hold two references to
+	   the old ldisc (if it was restored as part of error cleanup
+	   above). In either case, releasing a single reference from
+	   the old ldisc is correct. */
+
 	tty_ldisc_put(old_ldisc);
 
 	/*

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-04-16 10:15 UTC #141 lore raw

PATCH 7/7 tty: Fix tty_ldisc_lock name collision

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

drivers/tty/tty_ldisc.c +12 ~~-12~~

index 9725c94..ba49c0e 100644
@@ -44,7 +44,7 @@ enum {
  *	callers who will do ldisc lookups and cannot sleep.
  */
 
-static DEFINE_RAW_SPINLOCK(tty_ldisc_lock);
+static DEFINE_RAW_SPINLOCK(tty_ldiscs_lock);
 /* Line disc dispatch table */
 static struct tty_ldisc_ops *tty_ldiscs[NR_LDISCS];

@@ -58,7 +58,7 @@ static struct tty_ldisc_ops *tty_ldiscs[NR_LDISCS];
  *	from this point onwards.
  *
  *	Locking:
- *		takes tty_ldisc_lock to guard against ldisc races
+ *		takes tty_ldiscs_lock to guard against ldisc races
  */
 
 int tty_register_ldisc(int disc, struct tty_ldisc_ops *new_ldisc)

@@ -69,11 +69,11 @@ int tty_register_ldisc(int disc, struct tty_ldisc_ops *new_ldisc)
 	if (disc < N_TTY || disc >= NR_LDISCS)
 		return -EINVAL;
 
-	raw_spin_lock_irqsave(&tty_ldisc_lock, flags);
+	raw_spin_lock_irqsave(&tty_ldiscs_lock, flags);
 	tty_ldiscs[disc] = new_ldisc;
 	new_ldisc->num = disc;
 	new_ldisc->refcount = 0;
-	raw_spin_unlock_irqrestore(&tty_ldisc_lock, flags);
+	raw_spin_unlock_irqrestore(&tty_ldiscs_lock, flags);
 
 	return ret;
 }

@@ -88,7 +88,7 @@ EXPORT_SYMBOL(tty_register_ldisc);
  *	currently in use.
  *
  *	Locking:
- *		takes tty_ldisc_lock to guard against ldisc races
+ *		takes tty_ldiscs_lock to guard against ldisc races
  */
 
 int tty_unregister_ldisc(int disc)

@@ -99,12 +99,12 @@ int tty_unregister_ldisc(int disc)
 	if (disc < N_TTY || disc >= NR_LDISCS)
 		return -EINVAL;
 
-	raw_spin_lock_irqsave(&tty_ldisc_lock, flags);
+	raw_spin_lock_irqsave(&tty_ldiscs_lock, flags);
 	if (tty_ldiscs[disc]->refcount)
 		ret = -EBUSY;
 	else
 		tty_ldiscs[disc] = NULL;
-	raw_spin_unlock_irqrestore(&tty_ldisc_lock, flags);
+	raw_spin_unlock_irqrestore(&tty_ldiscs_lock, flags);
 
 	return ret;
 }

@@ -115,7 +115,7 @@ static struct tty_ldisc_ops *get_ldops(int disc)
 	unsigned long flags;
 	struct tty_ldisc_ops *ldops, *ret;
 
-	raw_spin_lock_irqsave(&tty_ldisc_lock, flags);
+	raw_spin_lock_irqsave(&tty_ldiscs_lock, flags);
 	ret = ERR_PTR(-EINVAL);
 	ldops = tty_ldiscs[disc];
 	if (ldops) {

@@ -125,7 +125,7 @@ static struct tty_ldisc_ops *get_ldops(int disc)
 			ret = ldops;
 		}
 	}
-	raw_spin_unlock_irqrestore(&tty_ldisc_lock, flags);
+	raw_spin_unlock_irqrestore(&tty_ldiscs_lock, flags);
 	return ret;
 }

@@ -133,10 +133,10 @@ static void put_ldops(struct tty_ldisc_ops *ldops)
 {
 	unsigned long flags;
 
-	raw_spin_lock_irqsave(&tty_ldisc_lock, flags);
+	raw_spin_lock_irqsave(&tty_ldiscs_lock, flags);
 	ldops->refcount--;
 	module_put(ldops->owner);
-	raw_spin_unlock_irqrestore(&tty_ldisc_lock, flags);
+	raw_spin_unlock_irqrestore(&tty_ldiscs_lock, flags);
 }
 
 /**

@@ -149,7 +149,7 @@ static void put_ldops(struct tty_ldisc_ops *ldops)
  *	available
  *
  *	Locking:
- *		takes tty_ldisc_lock to guard against ldisc races
+ *		takes tty_ldiscs_lock to guard against ldisc races
  */
 
 static struct tty_ldisc *tty_ldisc_get(struct tty_struct *tty, int disc)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #142 lore raw

v4 cover lockless n_tty receive path

In reply to Peter Hurley

Greg,

As before, this patchset is dependent on the 'ldsem patchset'. The reason is that this series abandons tty->receive_room as a flow control mechanism (because that requires locking), and the TIOCSETD ioctl _without ldsem_ uses tty->receive_room to shutoff i/o.

It is also dependent on 'n_tty fixes' which I recently resent to you.

Regards,
Peter Hurley

*******

This patchset is the 1st of 4 patchsets which implements an almost entirely lockless receive path from driver to user-space.

Non-rigorous performance measurements show a 9~15x speed improvement on SMP in end-to-end copying with all 4 patchsets applied.

** v4 changes **
- Rebased on tty.git/tty-next of 14 Jun

** v3 changes **
- Instead of a new receive_room() ldisc method which requires acquiring the termios_rwsem twice for every flip buffer received, this patchset version adds an alternate receive_buf2() ldisc method for use with flow-controlled line disciplines (like N_TTY). This also fixes a race when termios can be changed between computing the receive space available and the subsequent receive_buf().
- Converts vt paste_selection() to use a helper function for this new ldisc method.
- Protects the n_tty_write() path from termios changes.
- Optimizes the N_TTY throttle/unthrottle by only offering termios read-safety to the driver throttle()/unthrottle() methods.
- Special-casing pty throttle/unthrottle to avoid multiple atomic operations for every read.

** v2 changes **
- Rebased on top of 'tty: Fix race condition if flushing tty flip buffers'
- I forgot to mention; this is ~35% faster on end-to-end tests on SMP.

This patchset implements lockless receive from tty flip buffers to the n_tty read buffer and lockless copy into the user-space read buffer.

By lockless, I'm referring to the fine-grained read_lock formerly used to serialize access to the shared n_tty read buffer (which wasn't being used everywhere it should have been).

In the current n_tty, the read_lock is grabbed a minimum of 3 times per byte!
- ^^^^
- should say 2 times per byte!

The read_lock is unnecessary to serialize access between the flip buffer work and the single reader, as this is a single-producer/single-consumer pattern.

However, other threads may attempt to read or modify the buffer indices, notably for buffer flushing and for setting/resetting termios (there are some others). In addition, termios changes can cause havoc while the tty flip buffer work is pushing more data. Read more about that here: https://lkml.org/lkml/2013/2/22/480

Both hurdles are overcome with the same mechanism: converting the termios_mutex to a r/w semaphore (just a normal one :).

Both the receive_buf() path and the read() path claim a reader lock on the termios_rwsem. This prevents concurrent changes to termios. Also, flush_buffer() and TIOCINQ ioctl obtain a write lock on the termios_rwsem to exclude the flip buffer work and user-space read from accessing the buffer indices while resetting them.

This patchset also implements a block copy from the read_buf into the user-space buffer in canonical mode (rather than the current byte-by-byte method).

Peter Hurley (24):
tty: Don't change receive_room for ioctl(TIOCSETD)
tty: Simplify tty buffer/ldisc interface with helper function
tty: Make ldisc input flow control concurrency-friendly
n_tty: Factor canonical mode copy from n_tty_read()
n_tty: Line copy to user buffer in canonical mode
n_tty: Split n_tty_chars_in_buffer() for reader-only interface
tty: Deprecate ldisc .chars_in_buffer() method
n_tty: Get read_cnt through accessor
n_tty: Don't wrap input buffer indices at buffer size
n_tty: Remove read_cnt
tty: Convert termios_mutex to termios_rwsem
n_tty: Access termios values safely
n_tty: Replace canon_data with index comparison
n_tty: Make N_TTY ldisc receive path lockless
n_tty: Reset lnext if canonical mode changes
n_tty: Fix type mismatches in receive_buf raw copy
n_tty: Don't wait for buffer work in read() loop
n_tty: Separate buffer indices to prevent cache-line sharing
tty: Only guarantee termios read safety for throttle/unthrottle
n_tty: Move chars_in_buffer() to factor throttle/unthrottle
n_tty: Factor throttle/unthrottle into helper functions
n_tty: Move n_tty_write_wakeup() to avoid forward declaration
n_tty: Special case pty flow control
n_tty: Queue buffer work on any available cpu

 drivers/net/irda/irtty-sir.c |   8 +-
 drivers/tty/n_tty.c          | 662 ++++++++++++++++++++++++++-----------------
 drivers/tty/pty.c            |   4 +-
 drivers/tty/tty_buffer.c     |  34 ++-
 drivers/tty/tty_io.c         |  15 +-
 drivers/tty/tty_ioctl.c      |  90 +++---
 drivers/tty/tty_ldisc.c      |  13 +-
 drivers/tty/vt/selection.c   |   4 +-
 drivers/tty/vt/vt.c          |   4 +-
 include/linux/tty.h          |  21 +-
 include/linux/tty_ldisc.h    |  13 +
 11 files changed, 530 insertions(+), 338 deletions(-)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #143 lore raw

PATCH v4 1/24 tty: Don't change receive_room for ioctl(TIOCSETD)

tty_set_ldisc() is guaranteed exclusive use of the line discipline by tty_ldisc_lock_pair_timeout(); shutting off input by resetting receive_room is unnecessary.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_ldisc.c | 3 ---
 1 file changed, 3 deletions(-)

drivers/tty/tty_ldisc.c +0 -3

index edeb1dd..86356e3 100644
@@ -540,9 +540,6 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 		return 0;
 	}
 
-	/* FIXME: why 'shutoff' input if the ldisc is locked? */
-	tty->receive_room = 0;
-
 	old_ldisc = tty->ldisc;
 	tty_lock(tty);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #144 lore raw

PATCH v4 2/24 tty: Simplify tty buffer/ldisc interface with helper function

In reply to Peter Hurley

Ldisc interface functions must be called with interrupts enabled. Separating the ldisc calls into a helper function simplies the eventual removal of the spinlock.

Note that access to the buf->head ptr outside the spinlock is safe here because;
* __tty_buffer_flush() is prevented from running while buffer work performs i/o,
* tty_buffer_find() only assigns buf->head if the flip buffer list is empty (which is never the case in flush_to_ldisc() since at least one buffer is always left in the list after use)
Access to the read index outside the spinlock is safe here for the same reasons.

Update the buffer's read index _after_ the data has been received by the ldisc.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

drivers/tty/tty_buffer.c +17 ~~-12~~

index a42a028..6c7a1d0 100644
@@ -403,6 +403,18 @@ int tty_prepare_flip_string_flags(struct tty_port *port,
 EXPORT_SYMBOL_GPL(tty_prepare_flip_string_flags);
 
 
+static int
+receive_buf(struct tty_struct *tty, struct tty_buffer *head, int count)
+{
+	struct tty_ldisc *disc = tty->ldisc;
+
+	count = min_t(int, count, tty->receive_room);
+	if (count)
+		disc->ops->receive_buf(tty, head->char_buf_ptr + head->read,
+				       head->flag_buf_ptr + head->read, count);
+	head->read += count;
+	return count;
+}
 
 /**
  *	flush_to_ldisc

@@ -438,8 +450,6 @@ static void flush_to_ldisc(struct work_struct *work)
 		struct tty_buffer *head;
 		while ((head = buf->head) != NULL) {
 			int count;
-			char *char_buf;
-			unsigned char *flag_buf;
 
 			count = head->commit - head->read;
 			if (!count) {

@@ -449,16 +459,10 @@ static void flush_to_ldisc(struct work_struct *work)
 				tty_buffer_free(port, head);
 				continue;
 			}
-			if (!tty->receive_room)
-				break;
-			if (count > tty->receive_room)
-				count = tty->receive_room;
-			char_buf = head->char_buf_ptr + head->read;
-			flag_buf = head->flag_buf_ptr + head->read;
-			head->read += count;
 			spin_unlock_irqrestore(&buf->lock, flags);
-			disc->ops->receive_buf(tty, char_buf,
-							flag_buf, count);
+
+			count = receive_buf(tty, head, count);
+
 			spin_lock_irqsave(&buf->lock, flags);
 			/* Ldisc or user is trying to flush the buffers.
 			   We may have a deferred request to flush the

@@ -469,7 +473,8 @@ static void flush_to_ldisc(struct work_struct *work)
 				clear_bit(TTYP_FLUSHPENDING, &port->iflags);
 				wake_up(&tty->read_wait);
 				break;
-			}
+			} else if (!count)
+				break;
 		}
 		clear_bit(TTYP_FLUSHING, &port->iflags);
 	}

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #145 lore raw

PATCH v4 3/24 tty: Make ldisc input flow control concurrency-friendly

In reply to Peter Hurley

Although line discipline receiving is single-producer/single-consumer, using tty->receive_room to manage flow control creates unnecessary critical regions requiring additional lock use.

Instead, introduce the optional .receive_buf2() ldisc method which returns the # of bytes actually received. Serialization is guaranteed by the caller.

In turn, the line discipline should schedule the buffer work item whenever space becomes available; ie., when there is room to receive data and receive_room() previously returned 0 (the buffer work item stops processing if receive_buf2() returns 0). Note the 'no room' state need not be atomic despite concurrent use by two threads because only the buffer work thread can set the state and only the read() thread can clear the state.

Add n_tty_receive_buf2() as the receive_buf2() method for N_TTY. Provide a public helper function, tty_ldisc_receive_buf(), to use when directly accessing the receive_buf() methods.

Line disciplines not using input flow control can continue to set tty->receive_room to a fixed value and only provide the receive_buf() method.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c        | 72 +++++++++++++++++++++++++++++-----------------
 drivers/tty/tty_buffer.c   | 13 ++++++---
 drivers/tty/vt/selection.c |  4 +--
 include/linux/tty.h        | 13 +++++++++
 include/linux/tty_ldisc.h  | 13 +++++++++
 5 files changed, 82 insertions(+), 33 deletions(-)

drivers/tty/n_tty.c +45 ~~-27~~

index 4bf0fc0..eddeb78 100644
@@ -79,6 +79,9 @@ struct n_tty_data {
 	unsigned long overrun_time;
 	int num_overrun;
 
+	/* non-atomic */
+	bool no_room;
+
 	unsigned char lnext:1, erasing:1, raw:1, real_raw:1, icanon:1;
 	unsigned char echo_overrun:1;

@@ -114,25 +117,10 @@ static inline int tty_put_user(struct tty_struct *tty, unsigned char x,
 	return put_user(x, ptr);
 }
 
-/**
- *	n_tty_set_room	-	receive space
- *	@tty: terminal
- *
- *	Updates tty->receive_room to reflect the currently available space
- *	in the input buffer, and re-schedules the flip buffer work if space
- *	just became available.
- *
- *	Locks: Concurrent update is protected with read_lock
- */
-
-static int set_room(struct tty_struct *tty)
+static int receive_room(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int left;
-	int old_left;
-	unsigned long flags;
-
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 
 	if (I_PARMRK(tty)) {
 		/* Multiply read_cnt by 3, since each byte might take up to

@@ -150,18 +138,27 @@ static int set_room(struct tty_struct *tty)
 	 */
 	if (left <= 0)
 		left = ldata->icanon && !ldata->canon_data;
-	old_left = tty->receive_room;
-	tty->receive_room = left;
 
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
-
-	return left && !old_left;
+	return left;
 }
 
+/**
+ *	n_tty_set_room	-	receive space
+ *	@tty: terminal
+ *
+ *	Re-schedules the flip buffer work if space just became available.
+ *
+ *	Locks: Concurrent update is protected with read_lock
+ */
+
 static void n_tty_set_room(struct tty_struct *tty)
 {
+	struct n_tty_data *ldata = tty->disc_data;
+
 	/* Did this open up the receive buffer? We may need to flip */
-	if (set_room(tty)) {
+	if (unlikely(ldata->no_room) && receive_room(tty)) {
+		ldata->no_room = 0;
+
 		WARN_RATELIMIT(tty->port->itty == NULL,
 				"scheduling with invalid itty\n");
 		/* see if ldisc has been killed - if so, this means that

@@ -1408,8 +1405,8 @@ static void n_tty_write_wakeup(struct tty_struct *tty)
  *	calls one at a time and in order (or using flush_to_ldisc)
  */
 
-static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
-			      char *fp, int count)
+static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
+			  char *fp, int count)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	const unsigned char *p;

@@ -1464,8 +1461,6 @@ static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			tty->ops->flush_chars(tty);
 	}
 
-	set_room(tty);
-
 	if ((!ldata->icanon && (ldata->read_cnt >= ldata->minimum_to_wake)) ||
 		L_EXTPROC(tty)) {
 		kill_fasync(&tty->fasync, SIGIO, POLL_IN);

@@ -1480,7 +1475,7 @@ static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	 */
 	while (1) {
 		tty_set_flow_change(tty, TTY_THROTTLE_SAFE);
-		if (tty->receive_room >= TTY_THRESHOLD_THROTTLE)
+		if (receive_room(tty) >= TTY_THRESHOLD_THROTTLE)
 			break;
 		if (!tty_throttle_safe(tty))
 			break;

@@ -1488,6 +1483,28 @@ static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	__tty_set_flow_change(tty, 0);
 }
 
+static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
+			      char *fp, int count)
+{
+	__receive_buf(tty, cp, fp, count);
+}
+
+static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,
+			      char *fp, int count)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	int room;
+
+	tty->receive_room = room = receive_room(tty);
+	if (!room)
+		ldata->no_room = 1;
+	count = min(count, room);
+	if (count)
+		__receive_buf(tty, cp, fp, count);
+
+	return count;
+}
+
 int is_ignored(int sig)
 {
 	return (sigismember(&current->blocked, sig) ||

@@ -2203,6 +2220,7 @@ struct tty_ldisc_ops tty_ldisc_N_TTY = {
 	.receive_buf     = n_tty_receive_buf,
 	.write_wakeup    = n_tty_write_wakeup,
 	.fasync		 = n_tty_fasync,
+	.receive_buf2	 = n_tty_receive_buf2,
 };
 
 /**

drivers/tty/tty_buffer.c +9 -4

index 6c7a1d0..ff1b2e3 100644
@@ -407,11 +407,16 @@ static int
 receive_buf(struct tty_struct *tty, struct tty_buffer *head, int count)
 {
 	struct tty_ldisc *disc = tty->ldisc;
+	char 	      *p = head->char_buf_ptr + head->read;
+	unsigned char *f = head->flag_buf_ptr + head->read;
 
-	count = min_t(int, count, tty->receive_room);
-	if (count)
-		disc->ops->receive_buf(tty, head->char_buf_ptr + head->read,
-				       head->flag_buf_ptr + head->read, count);
+	if (disc->ops->receive_buf2)
+		count = disc->ops->receive_buf2(tty, p, f, count);
+	else {
+		count = min_t(int, count, tty->receive_room);
+		if (count)
+			disc->ops->receive_buf(tty, p, f, count);
+	}
 	head->read += count;
 	return count;
 }

drivers/tty/vt/selection.c +2 -2

index 60b7b69..2ca8d6b 100644
@@ -356,8 +356,8 @@ int paste_selection(struct tty_struct *tty)
 			continue;
 		}
 		count = sel_buffer_lth - pasted;
-		count = min(count, tty->receive_room);
-		ld->ops->receive_buf(tty, sel_buffer + pasted, NULL, count);
+		count = tty_ldisc_receive_buf(ld, sel_buffer + pasted, NULL,
+					      count);
 		pasted += count;
 	}
 	remove_wait_queue(&vc->paste_wait, &wait);

include/linux/tty.h +13 -0

index 7269daf..8323ee4 100644
@@ -557,6 +557,19 @@ extern void tty_ldisc_init(struct tty_struct *tty);
 extern void tty_ldisc_deinit(struct tty_struct *tty);
 extern void tty_ldisc_begin(void);
 
+static inline int tty_ldisc_receive_buf(struct tty_ldisc *ld, unsigned char *p,
+					char *f, int count)
+{
+	if (ld->ops->receive_buf2)
+		count = ld->ops->receive_buf2(ld->tty, p, f, count);
+	else {
+		count = min_t(int, count, ld->tty->receive_room);
+		if (count)
+			ld->ops->receive_buf(ld->tty, p, f, count);
+	}
+	return count;
+}
+
 
 /* n_tty.c */
 extern struct tty_ldisc_ops tty_ldisc_N_TTY;

include/linux/tty_ldisc.h +13 -0

index 23bdd9d..f15c898 100644
@@ -109,6 +109,17 @@
  *
  *	Tells the discipline that the DCD pin has changed its status.
  *	Used exclusively by the N_PPS (Pulse-Per-Second) line discipline.
+ *
+ * int	(*receive_buf2)(struct tty_struct *, const unsigned char *cp,
+ *			char *fp, int count);
+ *
+ *	This function is called by the low-level tty driver to send
+ *	characters received by the hardware to the line discpline for
+ *	processing.  <cp> is a pointer to the buffer of input
+ *	character received by the device.  <fp> is a pointer to a
+ *	pointer of flag bytes which indicate whether a character was
+ *	received with a parity error, etc.
+ *	If assigned, prefer this function for automatic flow control.
  */
 
 #include <linux/fs.h>

@@ -195,6 +206,8 @@ struct tty_ldisc_ops {
 	void	(*write_wakeup)(struct tty_struct *);
 	void	(*dcd_change)(struct tty_struct *, unsigned int);
 	void	(*fasync)(struct tty_struct *tty, int on);
+	int	(*receive_buf2)(struct tty_struct *, const unsigned char *cp,
+				char *fp, int count);
 
 	struct  module *owner;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #146 lore raw

PATCH v4 4/24 n_tty: Factor canonical mode copy from n_tty_read()

In reply to Peter Hurley

Simplify n_tty_read(); extract complex copy algorithm into separate function, canon_copy_to_user().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 95 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 57 insertions(+), 38 deletions(-)

drivers/tty/n_tty.c +57 ~~-38~~

index eddeb78..7fd7774 100644
@@ -1747,6 +1747,62 @@ static int copy_from_read_buf(struct tty_struct *tty,
 	return retval;
 }
 
+/**
+ *	canon_copy_to_user	-	copy read data in canonical mode
+ *	@tty: terminal device
+ *	@b: user data
+ *	@nr: size of data
+ *
+ *	Helper function for n_tty_read.  It is only called when ICANON is on;
+ *	it copies characters one at a time from the read buffer to the user
+ *	space buffer.
+ *
+ *	Called under the atomic_read_lock mutex
+ */
+
+static int canon_copy_to_user(struct tty_struct *tty,
+			      unsigned char __user **b,
+			      size_t *nr)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	unsigned long flags;
+	int eol, c;
+
+	/* N.B. avoid overrun if nr == 0 */
+	raw_spin_lock_irqsave(&ldata->read_lock, flags);
+	while (*nr && ldata->read_cnt) {
+
+		eol = test_and_clear_bit(ldata->read_tail, ldata->read_flags);
+		c = ldata->read_buf[ldata->read_tail];
+		ldata->read_tail = (ldata->read_tail+1) & (N_TTY_BUF_SIZE-1);
+		ldata->read_cnt--;
+		if (eol) {
+			/* this test should be redundant:
+			 * we shouldn't be reading data if
+			 * canon_data is 0
+			 */
+			if (--ldata->canon_data < 0)
+				ldata->canon_data = 0;
+		}
+		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+
+		if (!eol || (c != __DISABLED_CHAR)) {
+			if (tty_put_user(tty, c, *b))
+				return -EFAULT;
+			*b += 1;
+			*nr -= 1;
+		}
+		if (eol) {
+			tty_audit_push(tty);
+			return 0;
+		}
+		raw_spin_lock_irqsave(&ldata->read_lock, flags);
+	}
+	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+
+	return 0;
+}
+
 extern ssize_t redirected_tty_write(struct file *, const char __user *,
 							size_t, loff_t *);

@@ -1916,44 +1972,7 @@ do_it_again:
 		}
 
 		if (ldata->icanon && !L_EXTPROC(tty)) {
-			/* N.B. avoid overrun if nr == 0 */
-			raw_spin_lock_irqsave(&ldata->read_lock, flags);
-			while (nr && ldata->read_cnt) {
-				int eol;
-
-				eol = test_and_clear_bit(ldata->read_tail,
-						ldata->read_flags);
-				c = ldata->read_buf[ldata->read_tail];
-				ldata->read_tail = ((ldata->read_tail+1) &
-						  (N_TTY_BUF_SIZE-1));
-				ldata->read_cnt--;
-				if (eol) {
-					/* this test should be redundant:
-					 * we shouldn't be reading data if
-					 * canon_data is 0
-					 */
-					if (--ldata->canon_data < 0)
-						ldata->canon_data = 0;
-				}
-				raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
-
-				if (!eol || (c != __DISABLED_CHAR)) {
-					if (tty_put_user(tty, c, b++)) {
-						retval = -EFAULT;
-						b--;
-						raw_spin_lock_irqsave(&ldata->read_lock, flags);
-						break;
-					}
-					nr--;
-				}
-				if (eol) {
-					tty_audit_push(tty);
-					raw_spin_lock_irqsave(&ldata->read_lock, flags);
-					break;
-				}
-				raw_spin_lock_irqsave(&ldata->read_lock, flags);
-			}
-			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+			retval = canon_copy_to_user(tty, &b, &nr);
 			if (retval)
 				break;
 		} else {

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #147 lore raw

PATCH v4 5/24 n_tty: Line copy to user buffer in canonical mode

In reply to Peter Hurley

Instead of pushing one char per loop, pre-compute the data length to copy and copy all at once.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 110 ++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 77 insertions(+), 33 deletions(-)

drivers/tty/n_tty.c +77 ~~-33~~

index 7fd7774..74dcedd 100644
@@ -74,6 +74,13 @@
 #define ECHO_OP_SET_CANON_COL 0x81
 #define ECHO_OP_ERASE_TAB 0x82
 
+#undef N_TTY_TRACE
+#ifdef N_TTY_TRACE
+# define n_tty_trace(f, args...)	trace_printk(f, ##args)
+#else
+# define n_tty_trace(f, args...)
+#endif
+
 struct n_tty_data {
 	unsigned int column;
 	unsigned long overrun_time;

@@ -1748,58 +1755,95 @@ static int copy_from_read_buf(struct tty_struct *tty,
 }
 
 /**
- *	canon_copy_to_user	-	copy read data in canonical mode
+ *	canon_copy_from_read_buf	-	copy read data in canonical mode
  *	@tty: terminal device
  *	@b: user data
  *	@nr: size of data
  *
  *	Helper function for n_tty_read.  It is only called when ICANON is on;
- *	it copies characters one at a time from the read buffer to the user
- *	space buffer.
+ *	it copies one line of input up to and including the line-delimiting
+ *	character into the user-space buffer.
  *
  *	Called under the atomic_read_lock mutex
  */
 
-static int canon_copy_to_user(struct tty_struct *tty,
-			      unsigned char __user **b,
-			      size_t *nr)
+static int canon_copy_from_read_buf(struct tty_struct *tty,
+				    unsigned char __user **b,
+				    size_t *nr)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	unsigned long flags;
-	int eol, c;
+	size_t n, size, more, c;
+	unsigned long eol;
+	int ret, tail, found = 0;
 
 	/* N.B. avoid overrun if nr == 0 */
+
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	while (*nr && ldata->read_cnt) {
-
-		eol = test_and_clear_bit(ldata->read_tail, ldata->read_flags);
-		c = ldata->read_buf[ldata->read_tail];
-		ldata->read_tail = (ldata->read_tail+1) & (N_TTY_BUF_SIZE-1);
-		ldata->read_cnt--;
-		if (eol) {
-			/* this test should be redundant:
-			 * we shouldn't be reading data if
-			 * canon_data is 0
-			 */
-			if (--ldata->canon_data < 0)
-				ldata->canon_data = 0;
-		}
+
+	n = min_t(size_t, *nr, ldata->read_cnt);
+	if (!n) {
 		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+		return 0;
+	}
 
-		if (!eol || (c != __DISABLED_CHAR)) {
-			if (tty_put_user(tty, c, *b))
-				return -EFAULT;
-			*b += 1;
-			*nr -= 1;
-		}
-		if (eol) {
-			tty_audit_push(tty);
-			return 0;
-		}
-		raw_spin_lock_irqsave(&ldata->read_lock, flags);
+	tail = ldata->read_tail;
+	size = min_t(size_t, tail + n, N_TTY_BUF_SIZE);
+
+	n_tty_trace("%s: nr:%zu tail:%d n:%zu size:%zu\n",
+		    __func__, *nr, tail, n, size);
+
+	eol = find_next_bit(ldata->read_flags, size, tail);
+	more = n - (size - tail);
+	if (eol == N_TTY_BUF_SIZE && more) {
+		/* scan wrapped without finding set bit */
+		eol = find_next_bit(ldata->read_flags, more, 0);
+		if (eol != more)
+			found = 1;
+	} else if (eol != size)
+		found = 1;
+
+	size = N_TTY_BUF_SIZE - tail;
+	n = (found + eol + size) & (N_TTY_BUF_SIZE - 1);
+	c = n;
+
+	if (found && ldata->read_buf[eol] == __DISABLED_CHAR)
+		n--;
+
+	n_tty_trace("%s: eol:%lu found:%d n:%zu c:%zu size:%zu more:%zu\n",
+		    __func__, eol, found, n, c, size, more);
+
+	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+
+	if (n > size) {
+		ret = copy_to_user(*b, &ldata->read_buf[tail], size);
+		if (ret)
+			return -EFAULT;
+		ret = copy_to_user(*b + size, ldata->read_buf, n - size);
+	} else
+		ret = copy_to_user(*b, &ldata->read_buf[tail], n);
+
+	if (ret)
+		return -EFAULT;
+	*b += n;
+	*nr -= n;
+
+	raw_spin_lock_irqsave(&ldata->read_lock, flags);
+	ldata->read_tail = (ldata->read_tail + c) & (N_TTY_BUF_SIZE - 1);
+	ldata->read_cnt -= c;
+	if (found) {
+		__clear_bit(eol, ldata->read_flags);
+		/* this test should be redundant:
+		 * we shouldn't be reading data if
+		 * canon_data is 0
+		 */
+		if (--ldata->canon_data < 0)
+			ldata->canon_data = 0;
 	}
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 
+	if (found)
+		tty_audit_push(tty);
 	return 0;
 }

@@ -1972,7 +2016,7 @@ do_it_again:
 		}
 
 		if (ldata->icanon && !L_EXTPROC(tty)) {
-			retval = canon_copy_to_user(tty, &b, &nr);
+			retval = canon_copy_from_read_buf(tty, &b, &nr);
 			if (retval)
 				break;
 		} else {

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #148 lore raw

PATCH v4 6/24 n_tty: Split n_tty_chars_in_buffer() for reader-only interface

In reply to Peter Hurley

N_TTY .chars_in_buffer() method requires serialized access if the current thread is not the single-consumer, n_tty_read().

Separate the internal interface; prepare for lockless read-side.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

drivers/tty/n_tty.c +8 -3

index 74dcedd..594a3c1 100644
@@ -277,7 +277,7 @@ static void n_tty_flush_buffer(struct tty_struct *tty)
  *	Locking: read_lock
  */
 
-static ssize_t n_tty_chars_in_buffer(struct tty_struct *tty)
+static ssize_t chars_in_buffer(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	unsigned long flags;

@@ -295,6 +295,11 @@ static ssize_t n_tty_chars_in_buffer(struct tty_struct *tty)
 	return n;
 }
 
+static ssize_t n_tty_chars_in_buffer(struct tty_struct *tty)
+{
+	return chars_in_buffer(tty);
+}
+
 /**
  *	is_utf8_continuation	-	utf8 multibyte check
  *	@c: byte to check

@@ -2032,7 +2037,7 @@ do_it_again:
 		}
 
 		/* If there is enough space in the read buffer now, let the
-		 * low-level driver know. We use n_tty_chars_in_buffer() to
+		 * low-level driver know. We use chars_in_buffer() to
 		 * check the buffer, as it now knows about canonical mode.
 		 * Otherwise, if the driver is throttled and the line is
 		 * longer than TTY_THRESHOLD_UNTHROTTLE in canonical mode,

@@ -2040,7 +2045,7 @@ do_it_again:
 		 */
 		while (1) {
 			tty_set_flow_change(tty, TTY_UNTHROTTLE_SAFE);
-			if (n_tty_chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
+			if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
 				break;
 			if (!tty->count)
 				break;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #149 lore raw

PATCH v4 7/24 tty: Deprecate ldisc .chars_in_buffer() method

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 1 +
 1 file changed, 1 insertion(+)

drivers/tty/n_tty.c +1 -0

index 594a3c1..290769c 100644
@@ -297,6 +297,7 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
 
 static ssize_t n_tty_chars_in_buffer(struct tty_struct *tty)
 {
+	WARN_ONCE(1, "%s is deprecated and scheduled for removal.", __func__);
 	return chars_in_buffer(tty);
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #150 lore raw

PATCH v4 8/24 n_tty: Get read_cnt through accessor

In reply to Peter Hurley

Prepare for replacing read_cnt field with computed value.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 41 +++++++++++++++++++++++------------------
 1 file changed, 23 insertions(+), 18 deletions(-)

drivers/tty/n_tty.c +23 ~~-18~~

index 290769c..371612b 100644
@@ -115,6 +115,11 @@ struct n_tty_data {
 	raw_spinlock_t read_lock;
 };
 
+static inline size_t read_cnt(struct n_tty_data *ldata)
+{
+	return ldata->read_cnt;
+}
+
 static inline int tty_put_user(struct tty_struct *tty, unsigned char x,
 			       unsigned char __user *ptr)
 {

@@ -133,9 +138,9 @@ static int receive_room(struct tty_struct *tty)
 		/* Multiply read_cnt by 3, since each byte might take up to
 		 * three times as many spaces when PARMRK is set (depending on
 		 * its flags, e.g. parity error). */
-		left = N_TTY_BUF_SIZE - ldata->read_cnt * 3 - 1;
+		left = N_TTY_BUF_SIZE - read_cnt(ldata) * 3 - 1;
 	} else
-		left = N_TTY_BUF_SIZE - ldata->read_cnt - 1;
+		left = N_TTY_BUF_SIZE - read_cnt(ldata) - 1;
 
 	/*
 	 * If we are doing input canonicalization, and there are no

@@ -180,7 +185,7 @@ static void n_tty_set_room(struct tty_struct *tty)
 
 static void put_tty_queue_nolock(unsigned char c, struct n_tty_data *ldata)
 {
-	if (ldata->read_cnt < N_TTY_BUF_SIZE) {
+	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
 		ldata->read_buf[ldata->read_head] = c;
 		ldata->read_head = (ldata->read_head + 1) & (N_TTY_BUF_SIZE-1);
 		ldata->read_cnt++;

@@ -285,7 +290,7 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 	if (!ldata->icanon) {
-		n = ldata->read_cnt;
+		n = read_cnt(ldata);
 	} else if (ldata->canon_data) {
 		n = (ldata->canon_head > ldata->read_tail) ?
 			ldata->canon_head - ldata->read_tail :

@@ -1204,7 +1209,7 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	if (!test_bit(c, ldata->process_char_map) || ldata->lnext) {
 		ldata->lnext = 0;
 		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (ldata->read_cnt >= (N_TTY_BUF_SIZE - parmrk - 1)) {
+		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
 			/* beep if no space */
 			if (L_ECHO(tty))
 				process_output('\a', tty);

@@ -1304,7 +1309,7 @@ send_signal:
 			return;
 		}
 		if (c == '\n') {
-			if (ldata->read_cnt >= N_TTY_BUF_SIZE) {
+			if (read_cnt(ldata) >= N_TTY_BUF_SIZE) {
 				if (L_ECHO(tty))
 					process_output('\a', tty);
 				return;

@@ -1316,7 +1321,7 @@ send_signal:
 			goto handle_newline;
 		}
 		if (c == EOF_CHAR(tty)) {
-			if (ldata->read_cnt >= N_TTY_BUF_SIZE)
+			if (read_cnt(ldata) >= N_TTY_BUF_SIZE)
 				return;
 			if (ldata->canon_head != ldata->read_head)
 				set_bit(TTY_PUSH, &tty->flags);

@@ -1327,7 +1332,7 @@ send_signal:
 		    (c == EOL2_CHAR(tty) && L_IEXTEN(tty))) {
 			parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty))
 				 ? 1 : 0;
-			if (ldata->read_cnt >= (N_TTY_BUF_SIZE - parmrk)) {
+			if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk)) {
 				if (L_ECHO(tty))
 					process_output('\a', tty);
 				return;

@@ -1364,7 +1369,7 @@ handle_newline:
 	}
 
 	parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-	if (ldata->read_cnt >= (N_TTY_BUF_SIZE - parmrk - 1)) {
+	if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
 		/* beep if no space */
 		if (L_ECHO(tty))
 			process_output('\a', tty);

@@ -1430,7 +1435,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 
 	if (ldata->real_raw) {
 		raw_spin_lock_irqsave(&ldata->read_lock, cpuflags);
-		i = min(N_TTY_BUF_SIZE - ldata->read_cnt,
+		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
 			N_TTY_BUF_SIZE - ldata->read_head);
 		i = min(count, i);
 		memcpy(ldata->read_buf + ldata->read_head, cp, i);

@@ -1439,7 +1444,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		cp += i;
 		count -= i;
 
-		i = min(N_TTY_BUF_SIZE - ldata->read_cnt,
+		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
 			N_TTY_BUF_SIZE - ldata->read_head);
 		i = min(count, i);
 		memcpy(ldata->read_buf + ldata->read_head, cp, i);

@@ -1474,7 +1479,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			tty->ops->flush_chars(tty);
 	}
 
-	if ((!ldata->icanon && (ldata->read_cnt >= ldata->minimum_to_wake)) ||
+	if ((!ldata->icanon && (read_cnt(ldata) >= ldata->minimum_to_wake)) ||
 		L_EXTPROC(tty)) {
 		kill_fasync(&tty->fasync, SIGIO, POLL_IN);
 		if (waitqueue_active(&tty->read_wait))

@@ -1552,7 +1557,7 @@ static void n_tty_set_termios(struct tty_struct *tty, struct ktermios *old)
 		ldata->erasing = 0;
 	}
 
-	if (canon_change && !L_ICANON(tty) && ldata->read_cnt)
+	if (canon_change && !L_ICANON(tty) && read_cnt(ldata))
 		wake_up_interruptible(&tty->read_wait);
 
 	ldata->icanon = (L_ICANON(tty) != 0);

@@ -1701,7 +1706,7 @@ static inline int input_available_p(struct tty_struct *tty, int amt)
 	if (ldata->icanon && !L_EXTPROC(tty)) {
 		if (ldata->canon_data)
 			return 1;
-	} else if (ldata->read_cnt >= (amt ? amt : 1))
+	} else if (read_cnt(ldata) >= (amt ? amt : 1))
 		return 1;
 
 	return 0;

@@ -1737,7 +1742,7 @@ static int copy_from_read_buf(struct tty_struct *tty,
 
 	retval = 0;
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	n = min(ldata->read_cnt, N_TTY_BUF_SIZE - ldata->read_tail);
+	n = min(read_cnt(ldata), N_TTY_BUF_SIZE - ldata->read_tail);
 	n = min(*nr, n);
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 	if (n) {

@@ -1751,7 +1756,7 @@ static int copy_from_read_buf(struct tty_struct *tty,
 		ldata->read_tail = (ldata->read_tail + n) & (N_TTY_BUF_SIZE-1);
 		ldata->read_cnt -= n;
 		/* Turn single EOF into zero-length read */
-		if (L_EXTPROC(tty) && ldata->icanon && is_eof && !ldata->read_cnt)
+		if (L_EXTPROC(tty) && ldata->icanon && is_eof && !read_cnt(ldata))
 			n = 0;
 		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 		*b += n;

@@ -1787,7 +1792,7 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 
-	n = min_t(size_t, *nr, ldata->read_cnt);
+	n = min(*nr, read_cnt(ldata));
 	if (!n) {
 		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 		return 0;

@@ -2253,7 +2258,7 @@ static int n_tty_ioctl(struct tty_struct *tty, struct file *file,
 		return put_user(tty_chars_in_buffer(tty), (int __user *) arg);
 	case TIOCINQ:
 		/* FIXME: Locking */
-		retval = ldata->read_cnt;
+		retval = read_cnt(ldata);
 		if (L_ICANON(tty))
 			retval = inq_canon(ldata);
 		return put_user(retval, (unsigned int __user *) arg);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:15 UTC #151 lore raw

PATCH v4 9/24 n_tty: Don't wrap input buffer indices at buffer size

In reply to Peter Hurley

Wrap read_buf indices (read_head, read_tail, canon_head) at max representable value, instead of at the N_TTY_BUF_SIZE. This step is necessary to allow lockless reads of these shared variables (by updating the variables atomically).

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 111 ++++++++++++++++++++++++++++------------------------
 1 file changed, 60 insertions(+), 51 deletions(-)

drivers/tty/n_tty.c +60 ~~-51~~

index 371612b..b2fef10 100644
@@ -96,8 +96,8 @@ struct n_tty_data {
 	DECLARE_BITMAP(read_flags, N_TTY_BUF_SIZE);
 
 	char *read_buf;
-	int read_head;
-	int read_tail;
+	size_t read_head;
+	size_t read_tail;
 	int read_cnt;
 	int minimum_to_wake;

@@ -106,7 +106,7 @@ struct n_tty_data {
 	unsigned int echo_cnt;
 
 	int canon_data;
-	unsigned long canon_head;
+	size_t canon_head;
 	unsigned int canon_column;
 
 	struct mutex atomic_read_lock;

@@ -120,6 +120,16 @@ static inline size_t read_cnt(struct n_tty_data *ldata)
 	return ldata->read_cnt;
 }
 
+static inline unsigned char read_buf(struct n_tty_data *ldata, size_t i)
+{
+	return ldata->read_buf[i & (N_TTY_BUF_SIZE - 1)];
+}
+
+static inline unsigned char *read_buf_addr(struct n_tty_data *ldata, size_t i)
+{
+	return &ldata->read_buf[i & (N_TTY_BUF_SIZE - 1)];
+}
+
 static inline int tty_put_user(struct tty_struct *tty, unsigned char x,
 			       unsigned char __user *ptr)
 {

@@ -186,8 +196,8 @@ static void n_tty_set_room(struct tty_struct *tty)
 static void put_tty_queue_nolock(unsigned char c, struct n_tty_data *ldata)
 {
 	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
-		ldata->read_buf[ldata->read_head] = c;
-		ldata->read_head = (ldata->read_head + 1) & (N_TTY_BUF_SIZE-1);
+		*read_buf_addr(ldata, ldata->read_head) = c;
+		ldata->read_head++;
 		ldata->read_cnt++;
 	}
 }

@@ -289,13 +299,10 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
 	ssize_t n = 0;
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	if (!ldata->icanon) {
+	if (!ldata->icanon)
 		n = read_cnt(ldata);
-	} else if (ldata->canon_data) {
-		n = (ldata->canon_head > ldata->read_tail) ?
-			ldata->canon_head - ldata->read_tail :
-			ldata->canon_head + (N_TTY_BUF_SIZE - ldata->read_tail);
-	}
+	else
+		n = ldata->canon_head - ldata->read_tail;
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 	return n;
 }

@@ -918,7 +925,9 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	enum { ERASE, WERASE, KILL } kill_type;
-	int head, seen_alnums, cnt;
+	size_t head;
+	size_t cnt;
+	int seen_alnums;
 	unsigned long flags;
 
 	/* FIXME: locking needed ? */

@@ -962,8 +971,8 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 
 		/* erase a single possibly multibyte character */
 		do {
-			head = (head - 1) & (N_TTY_BUF_SIZE-1);
-			c = ldata->read_buf[head];
+			head--;
+			c = read_buf(ldata, head);
 		} while (is_continuation(c, tty) && head != ldata->canon_head);
 
 		/* do not partially erase */

@@ -977,7 +986,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 			else if (seen_alnums)
 				break;
 		}
-		cnt = (ldata->read_head - head) & (N_TTY_BUF_SIZE-1);
+		cnt = ldata->read_head - head;
 		raw_spin_lock_irqsave(&ldata->read_lock, flags);
 		ldata->read_head = head;
 		ldata->read_cnt -= cnt;

@@ -991,9 +1000,8 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 				/* if cnt > 1, output a multi-byte character */
 				echo_char(c, tty);
 				while (--cnt > 0) {
-					head = (head+1) & (N_TTY_BUF_SIZE-1);
-					echo_char_raw(ldata->read_buf[head],
-							ldata);
+					head++;
+					echo_char_raw(read_buf(ldata, head), ldata);
 					echo_move_back_col(ldata);
 				}
 			} else if (kill_type == ERASE && !L_ECHOE(tty)) {

@@ -1001,7 +1009,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 			} else if (c == '\t') {
 				unsigned int num_chars = 0;
 				int after_tab = 0;
-				unsigned long tail = ldata->read_head;
+				size_t tail = ldata->read_head;
 
 				/*
 				 * Count the columns used for characters

@@ -1011,8 +1019,8 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 				 * number of columns.
 				 */
 				while (tail != ldata->canon_head) {
-					tail = (tail-1) & (N_TTY_BUF_SIZE-1);
-					c = ldata->read_buf[tail];
+					tail--;
+					c = read_buf(ldata, tail);
 					if (c == '\t') {
 						after_tab = 1;
 						break;

@@ -1296,14 +1304,14 @@ send_signal:
 		}
 		if (c == REPRINT_CHAR(tty) && L_ECHO(tty) &&
 		    L_IEXTEN(tty)) {
-			unsigned long tail = ldata->canon_head;
+			size_t tail = ldata->canon_head;
 
 			finish_erasing(ldata);
 			echo_char(c, tty);
 			echo_char_raw('\n', ldata);
 			while (tail != ldata->read_head) {
-				echo_char(ldata->read_buf[tail], tty);
-				tail = (tail+1) & (N_TTY_BUF_SIZE-1);
+				echo_char(read_buf(ldata, tail), tty);
+				tail++;
 			}
 			process_echoes(tty);
 			return;

@@ -1356,7 +1364,7 @@ send_signal:
 
 handle_newline:
 			raw_spin_lock_irqsave(&ldata->read_lock, flags);
-			set_bit(ldata->read_head, ldata->read_flags);
+			set_bit(ldata->read_head & (N_TTY_BUF_SIZE - 1), ldata->read_flags);
 			put_tty_queue_nolock(c, ldata);
 			ldata->canon_head = ldata->read_head;
 			ldata->canon_data++;

@@ -1436,19 +1444,19 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	if (ldata->real_raw) {
 		raw_spin_lock_irqsave(&ldata->read_lock, cpuflags);
 		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
-			N_TTY_BUF_SIZE - ldata->read_head);
+			N_TTY_BUF_SIZE - (ldata->read_head & (N_TTY_BUF_SIZE - 1)));
 		i = min(count, i);
-		memcpy(ldata->read_buf + ldata->read_head, cp, i);
-		ldata->read_head = (ldata->read_head + i) & (N_TTY_BUF_SIZE-1);
+		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
+		ldata->read_head += i;
 		ldata->read_cnt += i;
 		cp += i;
 		count -= i;
 
 		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
-			N_TTY_BUF_SIZE - ldata->read_head);
+			N_TTY_BUF_SIZE - (ldata->read_head & (N_TTY_BUF_SIZE - 1)));
 		i = min(count, i);
-		memcpy(ldata->read_buf + ldata->read_head, cp, i);
-		ldata->read_head = (ldata->read_head + i) & (N_TTY_BUF_SIZE-1);
+		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
+		ldata->read_head += i;
 		ldata->read_cnt += i;
 		raw_spin_unlock_irqrestore(&ldata->read_lock, cpuflags);
 	} else {

@@ -1739,21 +1747,21 @@ static int copy_from_read_buf(struct tty_struct *tty,
 	size_t n;
 	unsigned long flags;
 	bool is_eof;
+	size_t tail = ldata->read_tail & (N_TTY_BUF_SIZE - 1);
 
 	retval = 0;
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	n = min(read_cnt(ldata), N_TTY_BUF_SIZE - ldata->read_tail);
+	n = min(read_cnt(ldata), N_TTY_BUF_SIZE - tail);
 	n = min(*nr, n);
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 	if (n) {
-		retval = copy_to_user(*b, &ldata->read_buf[ldata->read_tail], n);
+		retval = copy_to_user(*b, read_buf_addr(ldata, tail), n);
 		n -= retval;
-		is_eof = n == 1 &&
-			ldata->read_buf[ldata->read_tail] == EOF_CHAR(tty);
-		tty_audit_add_data(tty, &ldata->read_buf[ldata->read_tail], n,
+		is_eof = n == 1 && read_buf(ldata, tail) == EOF_CHAR(tty);
+		tty_audit_add_data(tty, read_buf_addr(ldata, tail), n,
 				ldata->icanon);
 		raw_spin_lock_irqsave(&ldata->read_lock, flags);
-		ldata->read_tail = (ldata->read_tail + n) & (N_TTY_BUF_SIZE-1);
+		ldata->read_tail += n;
 		ldata->read_cnt -= n;
 		/* Turn single EOF into zero-length read */
 		if (L_EXTPROC(tty) && ldata->icanon && is_eof && !read_cnt(ldata))

@@ -1785,8 +1793,9 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	struct n_tty_data *ldata = tty->disc_data;
 	unsigned long flags;
 	size_t n, size, more, c;
-	unsigned long eol;
-	int ret, tail, found = 0;
+	size_t eol;
+	size_t tail;
+	int ret, found = 0;
 
 	/* N.B. avoid overrun if nr == 0 */

@@ -1798,10 +1807,10 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 		return 0;
 	}
 
-	tail = ldata->read_tail;
+	tail = ldata->read_tail & (N_TTY_BUF_SIZE - 1);
 	size = min_t(size_t, tail + n, N_TTY_BUF_SIZE);
 
-	n_tty_trace("%s: nr:%zu tail:%d n:%zu size:%zu\n",
+	n_tty_trace("%s: nr:%zu tail:%zu n:%zu size:%zu\n",
 		    __func__, *nr, tail, n, size);
 
 	eol = find_next_bit(ldata->read_flags, size, tail);

@@ -1818,21 +1827,21 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	n = (found + eol + size) & (N_TTY_BUF_SIZE - 1);
 	c = n;
 
-	if (found && ldata->read_buf[eol] == __DISABLED_CHAR)
+	if (found && read_buf(ldata, eol) == __DISABLED_CHAR)
 		n--;
 
-	n_tty_trace("%s: eol:%lu found:%d n:%zu c:%zu size:%zu more:%zu\n",
+	n_tty_trace("%s: eol:%zu found:%d n:%zu c:%zu size:%zu more:%zu\n",
 		    __func__, eol, found, n, c, size, more);
 
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 
 	if (n > size) {
-		ret = copy_to_user(*b, &ldata->read_buf[tail], size);
+		ret = copy_to_user(*b, read_buf_addr(ldata, tail), size);
 		if (ret)
 			return -EFAULT;
 		ret = copy_to_user(*b + size, ldata->read_buf, n - size);
 	} else
-		ret = copy_to_user(*b, &ldata->read_buf[tail], n);
+		ret = copy_to_user(*b, read_buf_addr(ldata, tail), n);
 
 	if (ret)
 		return -EFAULT;

@@ -1840,7 +1849,7 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	*nr -= n;
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	ldata->read_tail = (ldata->read_tail + c) & (N_TTY_BUF_SIZE - 1);
+	ldata->read_tail += c;
 	ldata->read_cnt -= c;
 	if (found) {
 		__clear_bit(eol, ldata->read_flags);

@@ -2230,19 +2239,19 @@ static unsigned int n_tty_poll(struct tty_struct *tty, struct file *file,
 
 static unsigned long inq_canon(struct n_tty_data *ldata)
 {
-	int nr, head, tail;
+	size_t nr, head, tail;
 
 	if (!ldata->canon_data)
 		return 0;
 	head = ldata->canon_head;
 	tail = ldata->read_tail;
-	nr = (head - tail) & (N_TTY_BUF_SIZE-1);
+	nr = head - tail;
 	/* Skip EOF-chars.. */
 	while (head != tail) {
-		if (test_bit(tail, ldata->read_flags) &&
-		    ldata->read_buf[tail] == __DISABLED_CHAR)
+		if (test_bit(tail & (N_TTY_BUF_SIZE - 1), ldata->read_flags) &&
+		    read_buf(ldata, tail) == __DISABLED_CHAR)
 			nr--;
-		tail = (tail+1) & (N_TTY_BUF_SIZE-1);
+		tail++;
 	}
 	return nr;
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:21 UTC #152 lore raw

PATCH v4 10/24 n_tty: Remove read_cnt

In reply to Peter Hurley

Storing the read_cnt creates an unnecessary shared variable between the single-producer (n_tty_receive_buf()) and the single-consumer (n_tty_read()).

Compute read_cnt from head & tail instead of storing.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)

drivers/tty/n_tty.c +2 ~~-13~~

index b2fef10..d159059 100644
@@ -98,7 +98,6 @@ struct n_tty_data {
 	char *read_buf;
 	size_t read_head;
 	size_t read_tail;
-	int read_cnt;
 	int minimum_to_wake;
 
 	unsigned char *echo_buf;

@@ -117,7 +116,7 @@ struct n_tty_data {
 
 static inline size_t read_cnt(struct n_tty_data *ldata)
 {
-	return ldata->read_cnt;
+	return ldata->read_head - ldata->read_tail;
 }
 
 static inline unsigned char read_buf(struct n_tty_data *ldata, size_t i)

@@ -198,7 +197,6 @@ static void put_tty_queue_nolock(unsigned char c, struct n_tty_data *ldata)
 	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
 		*read_buf_addr(ldata, ldata->read_head) = c;
 		ldata->read_head++;
-		ldata->read_cnt++;
 	}
 }

@@ -239,7 +237,7 @@ static void reset_buffer_flags(struct n_tty_data *ldata)
 	unsigned long flags;
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	ldata->read_head = ldata->read_tail = ldata->read_cnt = 0;
+	ldata->read_head = ldata->read_tail = 0;
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 
 	mutex_lock(&ldata->echo_lock);

@@ -942,16 +940,12 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 	else {
 		if (!L_ECHO(tty)) {
 			raw_spin_lock_irqsave(&ldata->read_lock, flags);
-			ldata->read_cnt -= ((ldata->read_head - ldata->canon_head) &
-					  (N_TTY_BUF_SIZE - 1));
 			ldata->read_head = ldata->canon_head;
 			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 			return;
 		}
 		if (!L_ECHOK(tty) || !L_ECHOKE(tty) || !L_ECHOE(tty)) {
 			raw_spin_lock_irqsave(&ldata->read_lock, flags);
-			ldata->read_cnt -= ((ldata->read_head - ldata->canon_head) &
-					  (N_TTY_BUF_SIZE - 1));
 			ldata->read_head = ldata->canon_head;
 			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 			finish_erasing(ldata);

@@ -989,7 +983,6 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 		cnt = ldata->read_head - head;
 		raw_spin_lock_irqsave(&ldata->read_lock, flags);
 		ldata->read_head = head;
-		ldata->read_cnt -= cnt;
 		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 		if (L_ECHO(tty)) {
 			if (L_ECHOPRT(tty)) {

@@ -1448,7 +1441,6 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		i = min(count, i);
 		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
 		ldata->read_head += i;
-		ldata->read_cnt += i;
 		cp += i;
 		count -= i;

@@ -1457,7 +1449,6 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		i = min(count, i);
 		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
 		ldata->read_head += i;
-		ldata->read_cnt += i;
 		raw_spin_unlock_irqrestore(&ldata->read_lock, cpuflags);
 	} else {
 		for (i = count, p = cp, f = fp; i; i--, p++) {

@@ -1762,7 +1753,6 @@ static int copy_from_read_buf(struct tty_struct *tty,
 				ldata->icanon);
 		raw_spin_lock_irqsave(&ldata->read_lock, flags);
 		ldata->read_tail += n;
-		ldata->read_cnt -= n;
 		/* Turn single EOF into zero-length read */
 		if (L_EXTPROC(tty) && ldata->icanon && is_eof && !read_cnt(ldata))
 			n = 0;

@@ -1850,7 +1840,6 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 	ldata->read_tail += c;
-	ldata->read_cnt -= c;
 	if (found) {
 		__clear_bit(eol, ldata->read_flags);
 		/* this test should be redundant:

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:21 UTC #153 lore raw

PATCH v4 11/24 tty: Convert termios_mutex to termios_rwsem

In reply to Peter Hurley

termios is commonly accessed unsafely (especially by N_TTY) because the existing mutex forces exclusive access. Convert existing usage.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/net/irda/irtty-sir.c |  8 ++--
 drivers/tty/n_tty.c          |  2 +-
 drivers/tty/pty.c            |  4 +-
 drivers/tty/tty_io.c         | 14 +++----
 drivers/tty/tty_ioctl.c      | 90 ++++++++++++++++++++++----------------------
 drivers/tty/tty_ldisc.c      | 10 ++---
 drivers/tty/vt/vt.c          |  4 +-
 include/linux/tty.h          |  7 ++--
 8 files changed, 70 insertions(+), 69 deletions(-)

drivers/net/irda/irtty-sir.c +4 -4

index a412671..177441a 100644
@@ -123,14 +123,14 @@ static int irtty_change_speed(struct sir_dev *dev, unsigned speed)
 
 	tty = priv->tty;
 
-	mutex_lock(&tty->termios_mutex);
+	down_write(&tty->termios_rwsem);
 	old_termios = tty->termios;
 	cflag = tty->termios.c_cflag;
 	tty_encode_baud_rate(tty, speed, speed);
 	if (tty->ops->set_termios)
 		tty->ops->set_termios(tty, &old_termios);
 	priv->io.speed = speed;
-	mutex_unlock(&tty->termios_mutex);
+	up_write(&tty->termios_rwsem);
 
 	return 0;
 }

@@ -280,7 +280,7 @@ static inline void irtty_stop_receiver(struct tty_struct *tty, int stop)
 	struct ktermios old_termios;
 	int cflag;
 
-	mutex_lock(&tty->termios_mutex);
+	down_write(&tty->termios_rwsem);
 	old_termios = tty->termios;
 	cflag = tty->termios.c_cflag;

@@ -292,7 +292,7 @@ static inline void irtty_stop_receiver(struct tty_struct *tty, int stop)
 	tty->termios.c_cflag = cflag;
 	if (tty->ops->set_termios)
 		tty->ops->set_termios(tty, &old_termios);
-	mutex_unlock(&tty->termios_mutex);
+	up_write(&tty->termios_rwsem);
 }
 
 /*****************************************************************/

drivers/tty/n_tty.c +1 -1

index d159059..ab923bb 100644
@@ -1539,7 +1539,7 @@ int is_ignored(int sig)
  *	guaranteed that this function will not be re-entered or in progress
  *	when the ldisc is closed.
  *
- *	Locking: Caller holds tty->termios_mutex
+ *	Locking: Caller holds tty->termios_rwsem
  */
 
 static void n_tty_set_termios(struct tty_struct *tty, struct ktermios *old)

drivers/tty/pty.c +2 -2

index 59bfaec..bbd9693 100644
@@ -290,7 +290,7 @@ static int pty_resize(struct tty_struct *tty,  struct winsize *ws)
 	struct tty_struct *pty = tty->link;
 
 	/* For a PTY we need to lock the tty side */
-	mutex_lock(&tty->termios_mutex);
+	down_write(&tty->termios_rwsem);
 	if (!memcmp(ws, &tty->winsize, sizeof(*ws)))
 		goto done;

@@ -317,7 +317,7 @@ static int pty_resize(struct tty_struct *tty,  struct winsize *ws)
 	tty->winsize = *ws;
 	pty->winsize = *ws;	/* Never used so will go away soon */
 done:
-	mutex_unlock(&tty->termios_mutex);
+	up_write(&tty->termios_rwsem);
 	return 0;
 }

drivers/tty/tty_io.c +7 -7

index 6cd08d9..1cec782 100644
@@ -604,7 +604,7 @@ static int tty_signal_session_leader(struct tty_struct *tty, int exit_session)
  *		  redirect lock for undoing redirection
  *		  file list lock for manipulating list of ttys
  *		  tty_ldiscs_lock from called functions
- *		  termios_mutex resetting termios data
+ *		  termios_rwsem resetting termios data
  *		  tasklist_lock to walk task list for hangup event
  *		    ->siglock to protect ->signal/->sighand
  */

@@ -2228,7 +2228,7 @@ static int tiocsti(struct tty_struct *tty, char __user *p)
  *
  *	Copies the kernel idea of the window size into the user buffer.
  *
- *	Locking: tty->termios_mutex is taken to ensure the winsize data
+ *	Locking: tty->termios_rwsem is taken to ensure the winsize data
  *		is consistent.
  */

@@ -2236,9 +2236,9 @@ static int tiocgwinsz(struct tty_struct *tty, struct winsize __user *arg)
 {
 	int err;
 
-	mutex_lock(&tty->termios_mutex);
+	down_read(&tty->termios_rwsem);
 	err = copy_to_user(arg, &tty->winsize, sizeof(*arg));
-	mutex_unlock(&tty->termios_mutex);
+	up_read(&tty->termios_rwsem);
 
 	return err ? -EFAULT: 0;
 }

@@ -2259,7 +2259,7 @@ int tty_do_resize(struct tty_struct *tty, struct winsize *ws)
 	unsigned long flags;
 
 	/* Lock the tty */
-	mutex_lock(&tty->termios_mutex);
+	down_write(&tty->termios_rwsem);
 	if (!memcmp(ws, &tty->winsize, sizeof(*ws)))
 		goto done;
 	/* Get the PID values and reference them so we can

@@ -2274,7 +2274,7 @@ int tty_do_resize(struct tty_struct *tty, struct winsize *ws)
 
 	tty->winsize = *ws;
 done:
-	mutex_unlock(&tty->termios_mutex);
+	up_write(&tty->termios_rwsem);
 	return 0;
 }
 EXPORT_SYMBOL(tty_do_resize);

@@ -3013,7 +3013,7 @@ void initialize_tty_struct(struct tty_struct *tty,
 	tty->session = NULL;
 	tty->pgrp = NULL;
 	mutex_init(&tty->legacy_mutex);
-	mutex_init(&tty->termios_mutex);
+	init_rwsem(&tty->termios_rwsem);
 	init_ldsem(&tty->ldisc_sem);
 	init_waitqueue_head(&tty->write_wait);
 	init_waitqueue_head(&tty->read_wait);

drivers/tty/tty_ioctl.c +45 ~~-45~~

320 lines not on this page · raw message

drivers/tty/tty_ldisc.c +5 -5

index 86356e3..56cd044 100644
@@ -415,14 +415,14 @@ EXPORT_SYMBOL_GPL(tty_ldisc_flush);
  *	they are not on hot paths so a little discipline won't do
  *	any harm.
  *
- *	Locking: takes termios_mutex
+ *	Locking: takes termios_rwsem
  */
 
 static void tty_set_termios_ldisc(struct tty_struct *tty, int num)
 {
-	mutex_lock(&tty->termios_mutex);
+	down_write(&tty->termios_rwsem);
 	tty->termios.c_line = num;
-	mutex_unlock(&tty->termios_mutex);
+	up_write(&tty->termios_rwsem);
 }
 
 /**

@@ -602,11 +602,11 @@ int tty_set_ldisc(struct tty_struct *tty, int ldisc)
 
 static void tty_reset_termios(struct tty_struct *tty)
 {
-	mutex_lock(&tty->termios_mutex);
+	down_write(&tty->termios_rwsem);
 	tty->termios = tty->driver->init_termios;
 	tty->termios.c_ispeed = tty_termios_input_baud_rate(&tty->termios);
 	tty->termios.c_ospeed = tty_termios_baud_rate(&tty->termios);
-	mutex_unlock(&tty->termios_mutex);
+	up_write(&tty->termios_rwsem);
 }

drivers/tty/vt/vt.c +2 -2

index c677829..02af6cc 100644
@@ -828,7 +828,7 @@ static inline int resize_screen(struct vc_data *vc, int width, int height,
  *	If the caller passes a tty structure then update the termios winsize
  *	information and perform any necessary signal handling.
  *
- *	Caller must hold the console semaphore. Takes the termios mutex and
+ *	Caller must hold the console semaphore. Takes the termios rwsem and
  *	ctrl_lock of the tty IFF a tty is passed.
  */

@@ -972,7 +972,7 @@ int vc_resize(struct vc_data *vc, unsigned int cols, unsigned int rows)
  *	the actual work.
  *
  *	Takes the console sem and the called methods then take the tty
- *	termios_mutex and the tty ctrl_lock in that order.
+ *	termios_rwsem and the tty ctrl_lock in that order.
  */
 static int vt_resize(struct tty_struct *tty, struct winsize *ws)
 {

include/linux/tty.h +4 -3

index 8323ee4..d304207 100644
@@ -10,6 +10,7 @@
 #include <linux/mutex.h>
 #include <linux/tty_flags.h>
 #include <uapi/linux/tty.h>
+#include <linux/rwsem.h>

@@ -243,9 +244,9 @@ struct tty_struct {
 
 	struct mutex atomic_write_lock;
 	struct mutex legacy_mutex;
-	struct mutex termios_mutex;
+	struct rw_semaphore termios_rwsem;
 	spinlock_t ctrl_lock;
-	/* Termios values are protected by the termios mutex */
+	/* Termios values are protected by the termios rwsem */
 	struct ktermios termios, termios_locked;
 	struct termiox *termiox;	/* May be NULL for unsupported */
 	char name[64];

@@ -253,7 +254,7 @@ struct tty_struct {
 	struct pid *session;
 	unsigned long flags;
 	int count;
-	struct winsize winsize;		/* termios mutex */
+	struct winsize winsize;		/* termios rwsem */
 	unsigned char stopped:1, hw_stopped:1, flow_stopped:1, packet:1;
 	unsigned char ctrl_status;	/* ctrl_lock */
 	unsigned int receive_room;	/* Bytes free for queue */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #154 lore raw

PATCH v4 12/24 n_tty: Access termios values safely

In reply to Peter Hurley

Use termios_rwsem to guarantee safe access to the termios values. This is particularly important for N_TTY as changing certain termios settings alters the mode of operation.

termios_rwsem must be dropped across throttle/unthrottle since those functions claim the termios_rwsem exclusively (to guarantee safe access to the termios and for mutual exclusion).

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 44 +++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 39 insertions(+), 5 deletions(-)

drivers/tty/n_tty.c +39 -5

index ab923bb..0599b58 100644
@@ -1491,10 +1491,14 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	 * canonical mode and don't have a newline yet!
 	 */
 	while (1) {
+		int throttled;
 		tty_set_flow_change(tty, TTY_THROTTLE_SAFE);
 		if (receive_room(tty) >= TTY_THRESHOLD_THROTTLE)
 			break;
-		if (!tty_throttle_safe(tty))
+		up_read(&tty->termios_rwsem);
+		throttled = tty_throttle_safe(tty);
+		down_read(&tty->termios_rwsem);
+		if (!throttled)
 			break;
 	}
 	__tty_set_flow_change(tty, 0);

@@ -1503,7 +1507,9 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			      char *fp, int count)
 {
+	down_read(&tty->termios_rwsem);
 	__receive_buf(tty, cp, fp, count);
+	up_read(&tty->termios_rwsem);
 }
 
 static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,

@@ -1512,6 +1518,8 @@ static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,
 	struct n_tty_data *ldata = tty->disc_data;
 	int room;
 
+	down_read(&tty->termios_rwsem);
+
 	tty->receive_room = room = receive_room(tty);
 	if (!room)
 		ldata->no_room = 1;

@@ -1519,6 +1527,8 @@ static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,
 	if (count)
 		__receive_buf(tty, cp, fp, count);
 
+	up_read(&tty->termios_rwsem);
+
 	return count;
 }

@@ -1934,6 +1944,8 @@ do_it_again:
 	if (c < 0)
 		return c;
 
+	down_read(&tty->termios_rwsem);
+
 	minimum = time = 0;
 	timeout = MAX_SCHEDULE_TIMEOUT;
 	if (!ldata->icanon) {

@@ -1955,11 +1967,15 @@ do_it_again:
 	 *	Internal serialization of reads.
 	 */
 	if (file->f_flags & O_NONBLOCK) {
-		if (!mutex_trylock(&ldata->atomic_read_lock))
+		if (!mutex_trylock(&ldata->atomic_read_lock)) {
+			up_read(&tty->termios_rwsem);
 			return -EAGAIN;
+		}
 	} else {
-		if (mutex_lock_interruptible(&ldata->atomic_read_lock))
+		if (mutex_lock_interruptible(&ldata->atomic_read_lock)) {
+			up_read(&tty->termios_rwsem);
 			return -ERESTARTSYS;
+		}
 	}
 	packet = tty->packet;

@@ -2009,7 +2025,11 @@ do_it_again:
 				break;
 			}
 			n_tty_set_room(tty);
+			up_read(&tty->termios_rwsem);
+
 			timeout = schedule_timeout(timeout);
+
+			down_read(&tty->termios_rwsem);
 			continue;
 		}
 		__set_current_state(TASK_RUNNING);

@@ -2048,13 +2068,17 @@ do_it_again:
 		 * we won't get any more characters.
 		 */
 		while (1) {
+			int unthrottled;
 			tty_set_flow_change(tty, TTY_UNTHROTTLE_SAFE);
 			if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
 				break;
 			if (!tty->count)
 				break;
 			n_tty_set_room(tty);
-			if (!tty_unthrottle_safe(tty))
+			up_read(&tty->termios_rwsem);
+			unthrottled = tty_unthrottle_safe(tty);
+			down_read(&tty->termios_rwsem);
+			if (!unthrottled)
 				break;
 		}
 		__tty_set_flow_change(tty, 0);

@@ -2076,10 +2100,13 @@ do_it_again:
 		retval = size;
 		if (nr)
 			clear_bit(TTY_PUSH, &tty->flags);
-	} else if (test_and_clear_bit(TTY_PUSH, &tty->flags))
+	} else if (test_and_clear_bit(TTY_PUSH, &tty->flags)) {
+		up_read(&tty->termios_rwsem);
 		goto do_it_again;
+	}
 
 	n_tty_set_room(tty);
+	up_read(&tty->termios_rwsem);
 	return retval;
 }

@@ -2120,6 +2147,8 @@ static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
 			return retval;
 	}
 
+	down_read(&tty->termios_rwsem);
+
 	/* Write out any echoed characters that are still pending */
 	process_echoes(tty);

@@ -2173,13 +2202,18 @@ static ssize_t n_tty_write(struct tty_struct *tty, struct file *file,
 			retval = -EAGAIN;
 			break;
 		}
+		up_read(&tty->termios_rwsem);
+
 		schedule();
+
+		down_read(&tty->termios_rwsem);
 	}
 break_out:
 	__set_current_state(TASK_RUNNING);
 	remove_wait_queue(&tty->write_wait, &wait);
 	if (b - buf != nr && tty->fasync)
 		set_bit(TTY_DO_WRITE_WAKEUP, &tty->flags);
+	up_read(&tty->termios_rwsem);
 	return (b - buf) ? b - buf : retval;
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #155 lore raw

PATCH v4 13/24 n_tty: Replace canon_data with index comparison

In reply to Peter Hurley

canon_data represented the # of lines which had been copied to the receive buffer but not yet copied to the user buffer. The value was tested to determine if input was available in canonical mode (and also to force input overrun if the receive buffer was full but a newline had not been received).

However, the actual count was irrelevent; only whether it was non-zero (meaning 'is there any input to transfer?'). This shared count is unnecessary and unsafe with a lockless algorithm. The same check is made by comparing canon_head with read_tail instead.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 22 ++++++----------------
 1 file changed, 6 insertions(+), 16 deletions(-)

drivers/tty/n_tty.c +6 ~~-16~~

index 0599b58..1098dd7 100644
@@ -104,7 +104,6 @@ struct n_tty_data {
 	unsigned int echo_pos;
 	unsigned int echo_cnt;
 
-	int canon_data;
 	size_t canon_head;
 	unsigned int canon_column;

@@ -158,7 +157,7 @@ static int receive_room(struct tty_struct *tty)
 	 * characters will be beeped.
 	 */
 	if (left <= 0)
-		left = ldata->icanon && !ldata->canon_data;
+		left = ldata->icanon && ldata->canon_head == ldata->read_tail;
 
 	return left;
 }

@@ -237,14 +236,14 @@ static void reset_buffer_flags(struct n_tty_data *ldata)
 	unsigned long flags;
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	ldata->read_head = ldata->read_tail = 0;
+	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 
 	mutex_lock(&ldata->echo_lock);
 	ldata->echo_pos = ldata->echo_cnt = ldata->echo_overrun = 0;
 	mutex_unlock(&ldata->echo_lock);
 
-	ldata->canon_head = ldata->canon_data = ldata->erasing = 0;
+	ldata->erasing = 0;
 	bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);
 }

@@ -1360,7 +1359,6 @@ handle_newline:
 			set_bit(ldata->read_head & (N_TTY_BUF_SIZE - 1), ldata->read_flags);
 			put_tty_queue_nolock(c, ldata);
 			ldata->canon_head = ldata->read_head;
-			ldata->canon_data++;
 			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 			kill_fasync(&tty->fasync, SIGIO, POLL_IN);
 			if (waitqueue_active(&tty->read_wait))

@@ -1562,7 +1560,6 @@ static void n_tty_set_termios(struct tty_struct *tty, struct ktermios *old)
 	if (canon_change) {
 		bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);
 		ldata->canon_head = ldata->read_tail;
-		ldata->canon_data = 0;
 		ldata->erasing = 0;
 	}

@@ -1713,7 +1710,7 @@ static inline int input_available_p(struct tty_struct *tty, int amt)
 
 	tty_flush_to_ldisc(tty);
 	if (ldata->icanon && !L_EXTPROC(tty)) {
-		if (ldata->canon_data)
+		if (ldata->canon_head != ldata->read_tail)
 			return 1;
 	} else if (read_cnt(ldata) >= (amt ? amt : 1))
 		return 1;

@@ -1850,15 +1847,8 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 
 	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 	ldata->read_tail += c;
-	if (found) {
+	if (found)
 		__clear_bit(eol, ldata->read_flags);
-		/* this test should be redundant:
-		 * we shouldn't be reading data if
-		 * canon_data is 0
-		 */
-		if (--ldata->canon_data < 0)
-			ldata->canon_data = 0;
-	}
 	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 
 	if (found)

@@ -2264,7 +2254,7 @@ static unsigned long inq_canon(struct n_tty_data *ldata)
 {
 	size_t nr, head, tail;
 
-	if (!ldata->canon_data)
+	if (ldata->canon_head == ldata->read_tail)
 		return 0;
 	head = ldata->canon_head;
 	tail = ldata->read_tail;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #156 lore raw

PATCH v4 14/24 n_tty: Make N_TTY ldisc receive path lockless

In reply to Peter Hurley

n_tty has a single-producer/single-consumer input model; use lockless publish instead.

Use termios_rwsem to exclude both consumer and producer while changing or resetting buffer indices, eg., when flushing. Also, claim exclusive termios_rwsem to safely retrieve the buffer indices from a thread other than consumer or producer (eg., TIOCINQ ioctl).

Note the read_tail is published _after_ clearing the newline indicator in read_flags to avoid racing the producer.

Drop read_lock spinlock.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 173 ++++++++++++++++++++++++++++------------------------
 1 file changed, 92 insertions(+), 81 deletions(-)

drivers/tty/n_tty.c +92 ~~-81~~

index 1098dd7..c7f71cb 100644
@@ -110,7 +110,6 @@ struct n_tty_data {
 	struct mutex atomic_read_lock;
 	struct mutex output_lock;
 	struct mutex echo_lock;
-	raw_spinlock_t read_lock;
 };
 
 static inline size_t read_cnt(struct n_tty_data *ldata)

@@ -168,7 +167,10 @@ static int receive_room(struct tty_struct *tty)
  *
  *	Re-schedules the flip buffer work if space just became available.
  *
- *	Locks: Concurrent update is protected with read_lock
+ *	Caller holds exclusive termios_rwsem
+ *	   or
+ *	n_tty_read()/consumer path:
+ *		holds non-exclusive termios_rwsem
  */
 
 static void n_tty_set_room(struct tty_struct *tty)

@@ -191,34 +193,27 @@ static void n_tty_set_room(struct tty_struct *tty)
 	}
 }
 
-static void put_tty_queue_nolock(unsigned char c, struct n_tty_data *ldata)
-{
-	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
-		*read_buf_addr(ldata, ldata->read_head) = c;
-		ldata->read_head++;
-	}
-}
-
 /**
  *	put_tty_queue		-	add character to tty
  *	@c: character
  *	@ldata: n_tty data
  *
- *	Add a character to the tty read_buf queue. This is done under the
- *	read_lock to serialize character addition and also to protect us
- *	against parallel reads or flushes
+ *	Add a character to the tty read_buf queue.
+ *
+ *	n_tty_receive_buf()/producer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		modifies read_head
+ *
+ *	read_head is only considered 'published' if canonical mode is
+ *	not active.
  */
 
 static void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
 {
-	unsigned long flags;
-	/*
-	 *	The problem of stomping on the buffers ends here.
-	 *	Why didn't anyone see this one coming? --AJK
-	*/
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	put_tty_queue_nolock(c, ldata);
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
+		*read_buf_addr(ldata, ldata->read_head) = c;
+		ldata->read_head++;
+	}
 }
 
 /**

@@ -228,16 +223,13 @@ static void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
  *	Reset the read buffer counters and clear the flags.
  *	Called from n_tty_open() and n_tty_flush_buffer().
  *
- *	Locking: tty_read_lock for read fields.
+ *	Locking: caller holds exclusive termios_rwsem
+ *		 (or locking is not required)
  */
 
 static void reset_buffer_flags(struct n_tty_data *ldata)
 {
-	unsigned long flags;
-
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 
 	mutex_lock(&ldata->echo_lock);
 	ldata->echo_pos = ldata->echo_cnt = ldata->echo_overrun = 0;

@@ -267,47 +259,55 @@ static void n_tty_packet_mode_flush(struct tty_struct *tty)
  *	buffer flushed (eg at hangup) or when the N_TTY line discipline
  *	internally has to clean the pending queue (for example some signals).
  *
- *	Locking: ctrl_lock, read_lock.
+ *	Holds termios_rwsem to exclude producer/consumer while
+ *	buffer indices are reset.
+ *
+ *	Locking: ctrl_lock, exclusive termios_rwsem
  */
 
 static void n_tty_flush_buffer(struct tty_struct *tty)
 {
+	down_write(&tty->termios_rwsem);
 	reset_buffer_flags(tty->disc_data);
 	n_tty_set_room(tty);
 
 	if (tty->link)
 		n_tty_packet_mode_flush(tty);
+	up_write(&tty->termios_rwsem);
 }
 
-/**
- *	n_tty_chars_in_buffer	-	report available bytes
- *	@tty: tty device
- *
- *	Report the number of characters buffered to be delivered to user
- *	at this instant in time.
- *
- *	Locking: read_lock
- */
-
 static ssize_t chars_in_buffer(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	unsigned long flags;
 	ssize_t n = 0;
 
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 	if (!ldata->icanon)
 		n = read_cnt(ldata);
 	else
 		n = ldata->canon_head - ldata->read_tail;
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 	return n;
 }
 
+/**
+ *	n_tty_chars_in_buffer	-	report available bytes
+ *	@tty: tty device
+ *
+ *	Report the number of characters buffered to be delivered to user
+ *	at this instant in time.
+ *
+ *	Locking: exclusive termios_rwsem
+ */
+
 static ssize_t n_tty_chars_in_buffer(struct tty_struct *tty)
 {
+	ssize_t n;
+
 	WARN_ONCE(1, "%s is deprecated and scheduled for removal.", __func__);
-	return chars_in_buffer(tty);
+
+	down_write(&tty->termios_rwsem);
+	n = chars_in_buffer(tty);
+	up_write(&tty->termios_rwsem);
+	return n;
 }
 
 /**

@@ -915,7 +915,12 @@ static inline void finish_erasing(struct n_tty_data *ldata)
  *	present in the stream from the driver layer. Handles the complexities
  *	of UTF-8 multibyte symbols.
  *
- *	Locking: read_lock for tty buffers
+ *	n_tty_receive_buf()/producer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		modifies read_head
+ *
+ *	Modifying the read_head is not considered a publish in this context
+ *	because canonical mode is active -- only canon_head publishes
  */
 
 static void eraser(unsigned char c, struct tty_struct *tty)

@@ -925,9 +930,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 	size_t head;
 	size_t cnt;
 	int seen_alnums;
-	unsigned long flags;
 
-	/* FIXME: locking needed ? */
 	if (ldata->read_head == ldata->canon_head) {
 		/* process_output('\a', tty); */ /* what do you think? */
 		return;

@@ -938,15 +941,11 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 		kill_type = WERASE;
 	else {
 		if (!L_ECHO(tty)) {
-			raw_spin_lock_irqsave(&ldata->read_lock, flags);
 			ldata->read_head = ldata->canon_head;
-			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 			return;
 		}
 		if (!L_ECHOK(tty) || !L_ECHOKE(tty) || !L_ECHOE(tty)) {
-			raw_spin_lock_irqsave(&ldata->read_lock, flags);
 			ldata->read_head = ldata->canon_head;
-			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 			finish_erasing(ldata);
 			echo_char(KILL_CHAR(tty), tty);
 			/* Add a newline if ECHOK is on and ECHOKE is off. */

@@ -958,7 +957,6 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 	}
 
 	seen_alnums = 0;
-	/* FIXME: Locking ?? */
 	while (ldata->read_head != ldata->canon_head) {
 		head = ldata->read_head;

@@ -980,9 +978,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
 				break;
 		}
 		cnt = ldata->read_head - head;
-		raw_spin_lock_irqsave(&ldata->read_lock, flags);
 		ldata->read_head = head;
-		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 		if (L_ECHO(tty)) {
 			if (L_ECHOPRT(tty)) {
 				if (!ldata->erasing) {

@@ -1071,7 +1067,11 @@ static inline void isig(int sig, struct tty_struct *tty)
  *	An RS232 break event has been hit in the incoming bitstream. This
  *	can cause a variety of events depending upon the termios settings.
  *
- *	Called from the receive_buf path so single threaded.
+ *	n_tty_receive_buf()/producer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		publishes read_head via put_tty_queue()
+ *
+ *	Note: may get exclusive termios_rwsem if flushing input buffer
  */
 
 static inline void n_tty_receive_break(struct tty_struct *tty)

@@ -1083,8 +1083,11 @@ static inline void n_tty_receive_break(struct tty_struct *tty)
 	if (I_BRKINT(tty)) {
 		isig(SIGINT, tty);
 		if (!L_NOFLSH(tty)) {
+			/* flushing needs exclusive termios_rwsem */
+			up_read(&tty->termios_rwsem);
 			n_tty_flush_buffer(tty);
 			tty_driver_flush_buffer(tty);
+			down_read(&tty->termios_rwsem);
 		}
 		return;
 	}

@@ -1131,7 +1134,11 @@ static inline void n_tty_receive_overrun(struct tty_struct *tty)
  *	@c: character
  *
  *	Process a parity error and queue the right data to indicate
- *	the error case if necessary. Locking as per n_tty_receive_buf.
+ *	the error case if necessary.
+ *
+ *	n_tty_receive_buf()/producer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		publishes read_head via put_tty_queue()
  */
 static inline void n_tty_receive_parity_error(struct tty_struct *tty,
 					      unsigned char c)

@@ -1159,12 +1166,16 @@ static inline void n_tty_receive_parity_error(struct tty_struct *tty,
  *	Process an individual character of input received from the driver.
  *	This is serialized with respect to itself by the rules for the
  *	driver above.
+ *
+ *	n_tty_receive_buf()/producer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		publishes canon_head if canonical mode is active
+ *		otherwise, publishes read_head via put_tty_queue()
  */
 
 static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	unsigned long flags;
 	int parmrk;
 
 	if (ldata->raw) {

@@ -1253,8 +1264,11 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 		if (c == SUSP_CHAR(tty)) {
 send_signal:
 			if (!L_NOFLSH(tty)) {
+				/* flushing needs exclusive termios_rwsem */
+				up_read(&tty->termios_rwsem);
 				n_tty_flush_buffer(tty);
 				tty_driver_flush_buffer(tty);
+				down_read(&tty->termios_rwsem);
 			}
 			if (I_IXON(tty))
 				start_tty(tty);

@@ -1355,11 +1369,9 @@ send_signal:
 				put_tty_queue(c, ldata);
 
 handle_newline:
-			raw_spin_lock_irqsave(&ldata->read_lock, flags);
 			set_bit(ldata->read_head & (N_TTY_BUF_SIZE - 1), ldata->read_flags);
-			put_tty_queue_nolock(c, ldata);
+			put_tty_queue(c, ldata);
 			ldata->canon_head = ldata->read_head;
-			raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 			kill_fasync(&tty->fasync, SIGIO, POLL_IN);
 			if (waitqueue_active(&tty->read_wait))
 				wake_up_interruptible(&tty->read_wait);

@@ -1420,6 +1432,10 @@ static void n_tty_write_wakeup(struct tty_struct *tty)
  *	been received. This function must be called from soft contexts
  *	not from interrupt context. The driver is responsible for making
  *	calls one at a time and in order (or using flush_to_ldisc)
+ *
+ *	n_tty_receive_buf()/producer path:
+ *		claims non-exclusive termios_rwsem
+ *		publishes read_head and canon_head
  */
 
 static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,

@@ -1430,10 +1446,8 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	char *f, flags = TTY_NORMAL;
 	int	i;
 	char	buf[64];
-	unsigned long cpuflags;
 
 	if (ldata->real_raw) {
-		raw_spin_lock_irqsave(&ldata->read_lock, cpuflags);
 		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
 			N_TTY_BUF_SIZE - (ldata->read_head & (N_TTY_BUF_SIZE - 1)));
 		i = min(count, i);

@@ -1447,7 +1461,6 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		i = min(count, i);
 		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
 		ldata->read_head += i;
-		raw_spin_unlock_irqrestore(&ldata->read_lock, cpuflags);
 	} else {
 		for (i = count, p = cp, f = fp; i; i--, p++) {
 			if (f)

@@ -1677,7 +1690,6 @@ static int n_tty_open(struct tty_struct *tty)
 	mutex_init(&ldata->atomic_read_lock);
 	mutex_init(&ldata->output_lock);
 	mutex_init(&ldata->echo_lock);
-	raw_spin_lock_init(&ldata->read_lock);
 
 	/* These are ugly. Currently a malloc failure here can panic */
 	ldata->read_buf = kzalloc(N_TTY_BUF_SIZE, GFP_KERNEL);

@@ -1733,6 +1745,9 @@ static inline int input_available_p(struct tty_struct *tty, int amt)
  *
  *	Called under the ldata->atomic_read_lock sem
  *
+ *	n_tty_read()/consumer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		read_tail published
  */
 
 static int copy_from_read_buf(struct tty_struct *tty,

@@ -1743,27 +1758,22 @@ static int copy_from_read_buf(struct tty_struct *tty,
 	struct n_tty_data *ldata = tty->disc_data;
 	int retval;
 	size_t n;
-	unsigned long flags;
 	bool is_eof;
 	size_t tail = ldata->read_tail & (N_TTY_BUF_SIZE - 1);
 
 	retval = 0;
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
 	n = min(read_cnt(ldata), N_TTY_BUF_SIZE - tail);
 	n = min(*nr, n);
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 	if (n) {
 		retval = copy_to_user(*b, read_buf_addr(ldata, tail), n);
 		n -= retval;
 		is_eof = n == 1 && read_buf(ldata, tail) == EOF_CHAR(tty);
 		tty_audit_add_data(tty, read_buf_addr(ldata, tail), n,
 				ldata->icanon);
-		raw_spin_lock_irqsave(&ldata->read_lock, flags);
 		ldata->read_tail += n;
 		/* Turn single EOF into zero-length read */
 		if (L_EXTPROC(tty) && ldata->icanon && is_eof && !read_cnt(ldata))
 			n = 0;
-		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
 		*b += n;
 		*nr -= n;
 	}

@@ -1781,6 +1791,10 @@ static int copy_from_read_buf(struct tty_struct *tty,
  *	character into the user-space buffer.
  *
  *	Called under the atomic_read_lock mutex
+ *
+ *	n_tty_read()/consumer path:
+ *		caller holds non-exclusive termios_rwsem
+ *		read_tail published
  */
 
 static int canon_copy_from_read_buf(struct tty_struct *tty,

@@ -1788,21 +1802,15 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 				    size_t *nr)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	unsigned long flags;
 	size_t n, size, more, c;
 	size_t eol;
 	size_t tail;
 	int ret, found = 0;
 
 	/* N.B. avoid overrun if nr == 0 */
-
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-
 	n = min(*nr, read_cnt(ldata));
-	if (!n) {
-		raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+	if (!n)
 		return 0;
-	}
 
 	tail = ldata->read_tail & (N_TTY_BUF_SIZE - 1);
 	size = min_t(size_t, tail + n, N_TTY_BUF_SIZE);

@@ -1830,8 +1838,6 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	n_tty_trace("%s: eol:%zu found:%d n:%zu c:%zu size:%zu more:%zu\n",
 		    __func__, eol, found, n, c, size, more);
 
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
-
 	if (n > size) {
 		ret = copy_to_user(*b, read_buf_addr(ldata, tail), size);
 		if (ret)

@@ -1845,11 +1851,10 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	*b += n;
 	*nr -= n;
 
-	raw_spin_lock_irqsave(&ldata->read_lock, flags);
-	ldata->read_tail += c;
 	if (found)
-		__clear_bit(eol, ldata->read_flags);
-	raw_spin_unlock_irqrestore(&ldata->read_lock, flags);
+		clear_bit(eol, ldata->read_flags);
+	smp_mb__after_clear_bit();
+	ldata->read_tail += c;
 
 	if (found)
 		tty_audit_push(tty);

@@ -1913,6 +1918,10 @@ static int job_control(struct tty_struct *tty, struct file *file)
  *	a hangup. Always called in user context, may sleep.
  *
  *	This code must be sure never to sleep through a hangup.
+ *
+ *	n_tty_read()/consumer path:
+ *		claims non-exclusive termios_rwsem
+ *		publishes read_tail
  */
 
 static ssize_t n_tty_read(struct tty_struct *tty, struct file *file,

@@ -2279,10 +2288,12 @@ static int n_tty_ioctl(struct tty_struct *tty, struct file *file,
 	case TIOCOUTQ:
 		return put_user(tty_chars_in_buffer(tty), (int __user *) arg);
 	case TIOCINQ:
-		/* FIXME: Locking */
-		retval = read_cnt(ldata);
+		down_write(&tty->termios_rwsem);
 		if (L_ICANON(tty))
 			retval = inq_canon(ldata);
+		else
+			retval = read_cnt(ldata);
+		up_write(&tty->termios_rwsem);
 		return put_user(retval, (unsigned int __user *) arg);
 	default:
 		return n_tty_ioctl_helper(tty, file, cmd, arg);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #157 lore raw

PATCH v4 15/24 n_tty: Reset lnext if canonical mode changes

In reply to Peter Hurley

lnext escapes the next input character as a literal, and must be reset when canonical mode changes (to avoid misinterpreting a special character as a literal if canonical mode is changed back again).

lnext is specifically not reset on a buffer flush so as to avoid misinterpreting the next input character as a special character.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 1 +
 1 file changed, 1 insertion(+)

drivers/tty/n_tty.c +1 -0

index c7f71cb..e63beab 100644
@@ -1574,6 +1574,7 @@ static void n_tty_set_termios(struct tty_struct *tty, struct ktermios *old)
 		bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);
 		ldata->canon_head = ldata->read_tail;
 		ldata->erasing = 0;
+		ldata->lnext = 0;
 	}
 
 	if (canon_change && !L_ICANON(tty) && read_cnt(ldata))

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #158 lore raw

PATCH v4 16/24 n_tty: Fix type mismatches in receive_buf raw copy

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)

drivers/tty/n_tty.c +17 ~~-14~~

index e63beab..fe1c399 100644
@@ -1444,24 +1444,27 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	struct n_tty_data *ldata = tty->disc_data;
 	const unsigned char *p;
 	char *f, flags = TTY_NORMAL;
-	int	i;
 	char	buf[64];
 
 	if (ldata->real_raw) {
-		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
-			N_TTY_BUF_SIZE - (ldata->read_head & (N_TTY_BUF_SIZE - 1)));
-		i = min(count, i);
-		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
-		ldata->read_head += i;
-		cp += i;
-		count -= i;
-
-		i = min(N_TTY_BUF_SIZE - read_cnt(ldata),
-			N_TTY_BUF_SIZE - (ldata->read_head & (N_TTY_BUF_SIZE - 1)));
-		i = min(count, i);
-		memcpy(read_buf_addr(ldata, ldata->read_head), cp, i);
-		ldata->read_head += i;
+		size_t n, head;
+
+		head = ldata->read_head & (N_TTY_BUF_SIZE - 1);
+		n = N_TTY_BUF_SIZE - max(read_cnt(ldata), head);
+		n = min_t(size_t, count, n);
+		memcpy(read_buf_addr(ldata, head), cp, n);
+		ldata->read_head += n;
+		cp += n;
+		count -= n;
+
+		head = ldata->read_head & (N_TTY_BUF_SIZE - 1);
+		n = N_TTY_BUF_SIZE - max(read_cnt(ldata), head);
+		n = min_t(size_t, count, n);
+		memcpy(read_buf_addr(ldata, head), cp, n);
+		ldata->read_head += n;
 	} else {
+		int i;
+
 		for (i = count, p = cp, f = fp; i; i--, p++) {
 			if (f)
 				flags = *f++;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #159 lore raw

PATCH v4 17/24 n_tty: Don't wait for buffer work in read() loop

In reply to Peter Hurley

User-space read() can run concurrently with receiving from device; waiting for receive_buf() to complete is not required.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 1 -
 1 file changed, 1 deletion(-)

drivers/tty/n_tty.c +0 -1

index fe1c399..a6eea30 100644
@@ -1724,7 +1724,6 @@ static inline int input_available_p(struct tty_struct *tty, int amt)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 
-	tty_flush_to_ldisc(tty);
 	if (ldata->icanon && !L_EXTPROC(tty)) {
 		if (ldata->canon_head != ldata->read_tail)
 			return 1;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #160 lore raw

PATCH v4 18/24 n_tty: Separate buffer indices to prevent cache-line sharing

In reply to Peter Hurley

If the read buffer indices are in the same cache-line, cpus will contended over the cache-line (so called 'false sharing').

Separate the producer-published fields from the consumer-published fields; document the locks relevant to each field.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

drivers/tty/n_tty.c +15 -6

index a6eea30..d0c8805 100644
@@ -82,29 +82,38 @@
 #endif
 
 struct n_tty_data {
-	unsigned int column;
+	/* producer-published */
+	size_t read_head;
+	size_t canon_head;
+	DECLARE_BITMAP(process_char_map, 256);
+
+	/* private to n_tty_receive_overrun (single-threaded) */
 	unsigned long overrun_time;
 	int num_overrun;
 
 	/* non-atomic */
 	bool no_room;
 
+	/* must hold exclusive termios_rwsem to reset these */
 	unsigned char lnext:1, erasing:1, raw:1, real_raw:1, icanon:1;
 	unsigned char echo_overrun:1;
 
-	DECLARE_BITMAP(process_char_map, 256);
+	/* shared by producer and consumer */
+	char *read_buf;
 	DECLARE_BITMAP(read_flags, N_TTY_BUF_SIZE);
 
-	char *read_buf;
-	size_t read_head;
-	size_t read_tail;
 	int minimum_to_wake;
 
+	/* consumer-published */
+	size_t read_tail;
+
+	/* protected by echo_lock */
 	unsigned char *echo_buf;
 	unsigned int echo_pos;
 	unsigned int echo_cnt;
 
-	size_t canon_head;
+	/* protected by output lock */
+	unsigned int column;
 	unsigned int canon_column;
 
 	struct mutex atomic_read_lock;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #161 lore raw

PATCH v4 19/24 tty: Only guarantee termios read safety for throttle/unthrottle

In reply to Peter Hurley

No tty driver modifies termios during throttle() or unthrottle(). Therefore, only read safety is required.

However, tty_throttle_safe and tty_unthrottle_safe must still be mutually exclusive; introduce throttle_mutex for that purpose.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c     | 4 ----
 drivers/tty/tty_io.c    | 1 +
 drivers/tty/tty_ioctl.c | 8 ++++----
 include/linux/tty.h     | 1 +
 4 files changed, 6 insertions(+), 8 deletions(-)

drivers/tty/n_tty.c +0 -4

index d0c8805..b78ee464 100644
@@ -1518,9 +1518,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		tty_set_flow_change(tty, TTY_THROTTLE_SAFE);
 		if (receive_room(tty) >= TTY_THRESHOLD_THROTTLE)
 			break;
-		up_read(&tty->termios_rwsem);
 		throttled = tty_throttle_safe(tty);
-		down_read(&tty->termios_rwsem);
 		if (!throttled)
 			break;
 	}

@@ -2086,9 +2084,7 @@ do_it_again:
 			if (!tty->count)
 				break;
 			n_tty_set_room(tty);
-			up_read(&tty->termios_rwsem);
 			unthrottled = tty_unthrottle_safe(tty);
-			down_read(&tty->termios_rwsem);
 			if (!unthrottled)
 				break;
 		}

drivers/tty/tty_io.c +1 -0

index 1cec782..7dbea58 100644
@@ -3013,6 +3013,7 @@ void initialize_tty_struct(struct tty_struct *tty,
 	tty->session = NULL;
 	tty->pgrp = NULL;
 	mutex_init(&tty->legacy_mutex);
+	mutex_init(&tty->throttle_mutex);
 	init_rwsem(&tty->termios_rwsem);
 	init_ldsem(&tty->ldisc_sem);
 	init_waitqueue_head(&tty->write_wait);

drivers/tty/tty_ioctl.c +4 -4

index 9ce20df..03ba081 100644
@@ -151,7 +151,7 @@ int tty_throttle_safe(struct tty_struct *tty)
 {
 	int ret = 0;
 
-	down_write(&tty->termios_rwsem);
+	mutex_lock(&tty->throttle_mutex);
 	if (!test_bit(TTY_THROTTLED, &tty->flags)) {
 		if (tty->flow_change != TTY_THROTTLE_SAFE)
 			ret = 1;

@@ -161,7 +161,7 @@ int tty_throttle_safe(struct tty_struct *tty)
 				tty->ops->throttle(tty);
 		}
 	}
-	up_write(&tty->termios_rwsem);
+	mutex_unlock(&tty->throttle_mutex);
 
 	return ret;
 }

@@ -182,7 +182,7 @@ int tty_unthrottle_safe(struct tty_struct *tty)
 {
 	int ret = 0;
 
-	down_write(&tty->termios_rwsem);
+	mutex_lock(&tty->throttle_mutex);
 	if (test_bit(TTY_THROTTLED, &tty->flags)) {
 		if (tty->flow_change != TTY_UNTHROTTLE_SAFE)
 			ret = 1;

@@ -192,7 +192,7 @@ int tty_unthrottle_safe(struct tty_struct *tty)
 				tty->ops->unthrottle(tty);
 		}
 	}
-	up_write(&tty->termios_rwsem);
+	mutex_unlock(&tty->throttle_mutex);
 
 	return ret;
 }

include/linux/tty.h +1 -0

index d304207..57a70d1 100644
@@ -244,6 +244,7 @@ struct tty_struct {
 
 	struct mutex atomic_write_lock;
 	struct mutex legacy_mutex;
+	struct mutex throttle_mutex;
 	struct rw_semaphore termios_rwsem;
 	spinlock_t ctrl_lock;
 	/* Termios values are protected by the termios rwsem */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #162 lore raw

PATCH v4 20/24 n_tty: Move chars_in_buffer() to factor throttle/unthrottle

In reply to Peter Hurley

Prepare to factor throttle and unthrottle into helper functions; relocate chars_in_buffer() to avoid forward declaration.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

drivers/tty/n_tty.c +12 ~~-12~~

index b78ee464..9ec0c68 100644
@@ -202,6 +202,18 @@ static void n_tty_set_room(struct tty_struct *tty)
 	}
 }
 
+static ssize_t chars_in_buffer(struct tty_struct *tty)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	ssize_t n = 0;
+
+	if (!ldata->icanon)
+		n = read_cnt(ldata);
+	else
+		n = ldata->canon_head - ldata->read_tail;
+	return n;
+}
+
 /**
  *	put_tty_queue		-	add character to tty
  *	@c: character

@@ -285,18 +297,6 @@ static void n_tty_flush_buffer(struct tty_struct *tty)
 	up_write(&tty->termios_rwsem);
 }
 
-static ssize_t chars_in_buffer(struct tty_struct *tty)
-{
-	struct n_tty_data *ldata = tty->disc_data;
-	ssize_t n = 0;
-
-	if (!ldata->icanon)
-		n = read_cnt(ldata);
-	else
-		n = ldata->canon_head - ldata->read_tail;
-	return n;
-}
-
 /**
  *	n_tty_chars_in_buffer	-	report available bytes
  *	@tty: tty device

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #163 lore raw

PATCH v4 21/24 n_tty: Factor throttle/unthrottle into helper functions

In reply to Peter Hurley

Prepare for special handling of pty throttle/unthrottle; factor flow control into helper functions.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 81 ++++++++++++++++++++++++++++++-----------------------
 1 file changed, 46 insertions(+), 35 deletions(-)

drivers/tty/n_tty.c +46 ~~-35~~

index 9ec0c68..9e13c80 100644
@@ -214,6 +214,50 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
 	return n;
 }
 
+static inline void n_tty_check_throttle(struct tty_struct *tty)
+{
+	/*
+	 * Check the remaining room for the input canonicalization
+	 * mode.  We don't want to throttle the driver if we're in
+	 * canonical mode and don't have a newline yet!
+	 */
+	while (1) {
+		int throttled;
+		tty_set_flow_change(tty, TTY_THROTTLE_SAFE);
+		if (receive_room(tty) >= TTY_THRESHOLD_THROTTLE)
+			break;
+		throttled = tty_throttle_safe(tty);
+		if (!throttled)
+			break;
+	}
+	__tty_set_flow_change(tty, 0);
+}
+
+static inline void n_tty_check_unthrottle(struct tty_struct *tty)
+{
+	/* If there is enough space in the read buffer now, let the
+	 * low-level driver know. We use chars_in_buffer() to
+	 * check the buffer, as it now knows about canonical mode.
+	 * Otherwise, if the driver is throttled and the line is
+	 * longer than TTY_THRESHOLD_UNTHROTTLE in canonical mode,
+	 * we won't get any more characters.
+	 */
+
+	while (1) {
+		int unthrottled;
+		tty_set_flow_change(tty, TTY_UNTHROTTLE_SAFE);
+		if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
+			break;
+		if (!tty->count)
+			break;
+		n_tty_set_room(tty);
+		unthrottled = tty_unthrottle_safe(tty);
+		if (!unthrottled)
+			break;
+	}
+	__tty_set_flow_change(tty, 0);
+}
+
 /**
  *	put_tty_queue		-	add character to tty
  *	@c: character

@@ -1508,21 +1552,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			wake_up_interruptible(&tty->read_wait);
 	}
 
-	/*
-	 * Check the remaining room for the input canonicalization
-	 * mode.  We don't want to throttle the driver if we're in
-	 * canonical mode and don't have a newline yet!
-	 */
-	while (1) {
-		int throttled;
-		tty_set_flow_change(tty, TTY_THROTTLE_SAFE);
-		if (receive_room(tty) >= TTY_THRESHOLD_THROTTLE)
-			break;
-		throttled = tty_throttle_safe(tty);
-		if (!throttled)
-			break;
-	}
-	__tty_set_flow_change(tty, 0);
+	n_tty_check_throttle(tty);
 }
 
 static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,

@@ -2069,26 +2099,7 @@ do_it_again:
 			}
 		}
 
-		/* If there is enough space in the read buffer now, let the
-		 * low-level driver know. We use chars_in_buffer() to
-		 * check the buffer, as it now knows about canonical mode.
-		 * Otherwise, if the driver is throttled and the line is
-		 * longer than TTY_THRESHOLD_UNTHROTTLE in canonical mode,
-		 * we won't get any more characters.
-		 */
-		while (1) {
-			int unthrottled;
-			tty_set_flow_change(tty, TTY_UNTHROTTLE_SAFE);
-			if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
-				break;
-			if (!tty->count)
-				break;
-			n_tty_set_room(tty);
-			unthrottled = tty_unthrottle_safe(tty);
-			if (!unthrottled)
-				break;
-		}
-		__tty_set_flow_change(tty, 0);
+		n_tty_check_unthrottle(tty);
 
 		if (b - buf >= minimum)
 			break;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #164 lore raw

PATCH v4 22/24 n_tty: Move n_tty_write_wakeup() to avoid forward declaration

In reply to Peter Hurley

Prepare to special case pty flow control; avoid forward declaration.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)

drivers/tty/n_tty.c +15 ~~-16~~

index 9e13c80..0e3efc1 100644
@@ -214,6 +214,21 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
 	return n;
 }
 
+/**
+ *	n_tty_write_wakeup	-	asynchronous I/O notifier
+ *	@tty: tty device
+ *
+ *	Required for the ptys, serial driver etc. since processes
+ *	that attach themselves to the master and rely on ASYNC
+ *	IO must be woken up
+ */
+
+static void n_tty_write_wakeup(struct tty_struct *tty)
+{
+	if (tty->fasync && test_and_clear_bit(TTY_DO_WRITE_WAKEUP, &tty->flags))
+		kill_fasync(&tty->fasync, SIGIO, POLL_OUT);
+}
+
 static inline void n_tty_check_throttle(struct tty_struct *tty)
 {
 	/*

@@ -1458,22 +1473,6 @@ handle_newline:
 	put_tty_queue(c, ldata);
 }
 
-
-/**
- *	n_tty_write_wakeup	-	asynchronous I/O notifier
- *	@tty: tty device
- *
- *	Required for the ptys, serial driver etc. since processes
- *	that attach themselves to the master and rely on ASYNC
- *	IO must be woken up
- */
-
-static void n_tty_write_wakeup(struct tty_struct *tty)
-{
-	if (tty->fasync && test_and_clear_bit(TTY_DO_WRITE_WAKEUP, &tty->flags))
-		kill_fasync(&tty->fasync, SIGIO, POLL_OUT);
-}
-
 /**
  *	n_tty_receive_buf	-	data receive
  *	@tty: terminal device

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:21 UTC #165 lore raw

PATCH v4 23/24 n_tty: Special case pty flow control

In reply to Peter Hurley

Instead, short-circuit the unthrottle if the tty is a pty and perform the write wakeup directly.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

drivers/tty/n_tty.c +13 -0

index 0e3efc1..072e52f 100644
@@ -231,6 +231,8 @@ static void n_tty_write_wakeup(struct tty_struct *tty)
 
 static inline void n_tty_check_throttle(struct tty_struct *tty)
 {
+	if (tty->driver->type == TTY_DRIVER_TYPE_PTY)
+		return;
 	/*
 	 * Check the remaining room for the input canonicalization
 	 * mode.  We don't want to throttle the driver if we're in

@@ -250,6 +252,17 @@ static inline void n_tty_check_throttle(struct tty_struct *tty)
 
 static inline void n_tty_check_unthrottle(struct tty_struct *tty)
 {
+	if (tty->driver->type == TTY_DRIVER_TYPE_PTY) {
+		if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
+			return;
+		if (!tty->count)
+			return;
+		n_tty_set_room(tty);
+		n_tty_write_wakeup(tty->link);
+		wake_up_interruptible_poll(&tty->link->write_wait, POLLOUT);
+		return;
+	}
+
 	/* If there is enough space in the read buffer now, let the
 	 * low-level driver know. We use chars_in_buffer() to
 	 * check the buffer, as it now knows about canonical mode.

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-23 13:15 UTC #166 lore raw

PATCH v5 23/24 n_tty: Special case pty flow control

Instead, short-circuit the unthrottle if the tty is a pty and perform the write wakeup directly.

Signed-off-by Peter Hurley <[email protected]>

---

v5: fix regression in unusual pty/ldisc configurations where
i/o fails to make forward progress

 drivers/tty/n_tty.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

drivers/tty/n_tty.c +14 -0

index 0e3efc1..59eb444 100644
@@ -231,6 +231,8 @@ static void n_tty_write_wakeup(struct tty_struct *tty)
 
 static inline void n_tty_check_throttle(struct tty_struct *tty)
 {
+	if (tty->driver->type == TTY_DRIVER_TYPE_PTY)
+		return;
 	/*
 	 * Check the remaining room for the input canonicalization
 	 * mode.  We don't want to throttle the driver if we're in

@@ -250,6 +252,18 @@ static inline void n_tty_check_throttle(struct tty_struct *tty)
 
 static inline void n_tty_check_unthrottle(struct tty_struct *tty)
 {
+	if (tty->driver->type == TTY_DRIVER_TYPE_PTY &&
+	    tty->link->ldisc->ops->write_wakeup == n_tty_write_wakeup) {
+		if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)
+			return;
+		if (!tty->count)
+			return;
+		n_tty_set_room(tty);
+		n_tty_write_wakeup(tty->link);
+		wake_up_interruptible_poll(&tty->link->write_wait, POLLOUT);
+		return;
+	}
+
 	/* If there is enough space in the read buffer now, let the
 	 * low-level driver know. We use chars_in_buffer() to
 	 * check the buffer, as it now knows about canonical mode.

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:14 UTC #167 lore raw

PATCH v4 24/24 n_tty: Queue buffer work on any available cpu

In reply to Peter Hurley

Scheduling buffer work on the same cpu as the read() thread limits the parallelism now possible between the receive_buf path and the n_tty_read() path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/tty/n_tty.c +1 -1

index 072e52f..2321d6a 100644
@@ -198,7 +198,7 @@ static void n_tty_set_room(struct tty_struct *tty)
 		 */
 		WARN_RATELIMIT(test_bit(TTY_LDISC_HALTED, &tty->flags),
 			       "scheduling buffer work for halted ldisc\n");
-		schedule_work(&tty->port->buf.work);
+		queue_work(system_unbound_wq, &tty->port->buf.work);
 	}
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #168 lore raw

v2 cover lockless tty flip buffers

In reply to Peter Hurley

** v2 changes **
- Rebased on v4 of 'lockless n_tty receive path'

This 2nd of 4 patchsets implements lockless receive from the tty driver. By lockless, I'm referring to the 'lock' spin lock formerly used to serialize access to the flip buffer list.

Since the driver-side flip buffer usage is already single-threaded and line discipline receiving is already single-threaded, implementing a lockless flip buffer list was the primary hurdle. [The only 2 flip buffer consumers, flush_to_ldisc() and tty_buffer_flush() were already mutually exclusive and this exclusion remains although the mechanism is changed.]

Since the flip buffer consumers, flush_to_ldisc() and tty_buffer_flush(), already leave the last-consumed flip buffer on the list, and since the existing flip buffer api is already divided into an add/commit interface, most of the requirement for a lockless algorithm was already in-place. The main differences are;
1) the initial state of the flip buffer list points head and tail to a 0-sized sentinel flip buffer. This eliminates head & tail NULL testing and assigning the head ptr from the driver-side thread. This sentinel is 'consumed' on the first iteration of ldisc receiving and does not require special-case logic.
2) the free list uses the atomic singly-linked llist interface. While this guarantees safe concurrent usage by both producer and consumer, it's not optimal. Both producer and consumer unnecessarily contend over the free list head ptr; a better approach would be to maintain an unconsumed buffer in the same way the flip buffer list works. Light testing has shown this contention accounts for roughly 5% of total cpu time in end-to-end copying.
3) The mutual exclusion between consumers is reimplemented as a mutex; this eliminates the need to drop the lock across the ldisc receive_buf() method. This mutual exclusion is extended to a public interface which the vt driver now uses to safely utilize the ldisc receive_buf() interface when pasting a selection.

Peter Hurley (16):
tty: Compute flip buffer ptrs
tty: Fix flip buffer free list
tty: Factor flip buffer initialization into helper function
tty: Merge tty_buffer_find() into tty_buffer_alloc()
tty: Use generic names for flip buffer list cursors
tty: Use lockless flip buffer free list
tty: Simplify flip buffer list with 0-sized sentinel
tty: Track flip buffer memory limit atomically
tty: Make driver-side flip buffers lockless
tty: Ensure single-threaded flip buffer consumer with mutex
tty: Only perform flip buffer flush from tty_buffer_flush()
tty: Avoid false-sharing flip buffer ptrs
tty: Use non-atomic state to signal flip buffer flush pending
tty: Merge __tty_flush_buffer() into lone call site
tty: Fix unsafe vt paste_selection()
tty: Remove private constant from global namespace

 drivers/staging/dgrp/dgrp_tty.c |   2 +
 drivers/tty/pty.c               |  10 +-
 drivers/tty/tty_buffer.c        | 395 +++++++++++++++++++---------------------
 drivers/tty/vt/selection.c      |   4 +-
 include/linux/llist.h           |  23 +++
 include/linux/tty.h             |  39 ++--
 include/linux/tty_flip.h        |   8 +-
 7 files changed, 244 insertions(+), 237 deletions(-)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #169 lore raw

PATCH v2 1/16 tty: Compute flip buffer ptrs

The char_buf_ptr and flag_buf_ptr values are trivially derived from the .data field offset; compute values as needed.

Fixes a long-standing type-mismatch with the char and flag ptrs.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 22 ++++++++++------------
 include/linux/tty.h      | 12 ++++++++++--
 include/linux/tty_flip.h |  4 ++--
 3 files changed, 22 insertions(+), 16 deletions(-)

drivers/tty/tty_buffer.c +10 ~~-12~~

index ff1b2e3..170674c 100644
@@ -71,8 +71,6 @@ static struct tty_buffer *tty_buffer_alloc(struct tty_port *port, size_t size)
 	p->next = NULL;
 	p->commit = 0;
 	p->read = 0;
-	p->char_buf_ptr = (char *)(p->data);
-	p->flag_buf_ptr = (unsigned char *)p->char_buf_ptr + size;
 	port->buf.memory_used += size;
 	return p;
 }

@@ -265,8 +263,8 @@ int tty_insert_flip_string_fixed_flag(struct tty_port *port,
 		if (unlikely(space == 0)) {
 			break;
 		}
-		memcpy(tb->char_buf_ptr + tb->used, chars, space);
-		memset(tb->flag_buf_ptr + tb->used, flag, space);
+		memcpy(char_buf_ptr(tb, tb->used), chars, space);
+		memset(flag_buf_ptr(tb, tb->used), flag, space);
 		tb->used += space;
 		copied += space;
 		chars += space;

@@ -303,8 +301,8 @@ int tty_insert_flip_string_flags(struct tty_port *port,
 		if (unlikely(space == 0)) {
 			break;
 		}
-		memcpy(tb->char_buf_ptr + tb->used, chars, space);
-		memcpy(tb->flag_buf_ptr + tb->used, flags, space);
+		memcpy(char_buf_ptr(tb, tb->used), chars, space);
+		memcpy(flag_buf_ptr(tb, tb->used), flags, space);
 		tb->used += space;
 		copied += space;
 		chars += space;

@@ -364,8 +362,8 @@ int tty_prepare_flip_string(struct tty_port *port, unsigned char **chars,
 	int space = tty_buffer_request_room(port, size);
 	if (likely(space)) {
 		struct tty_buffer *tb = port->buf.tail;
-		*chars = tb->char_buf_ptr + tb->used;
-		memset(tb->flag_buf_ptr + tb->used, TTY_NORMAL, space);
+		*chars = char_buf_ptr(tb, tb->used);
+		memset(flag_buf_ptr(tb, tb->used), TTY_NORMAL, space);
 		tb->used += space;
 	}
 	return space;

@@ -394,8 +392,8 @@ int tty_prepare_flip_string_flags(struct tty_port *port,
 	int space = tty_buffer_request_room(port, size);
 	if (likely(space)) {
 		struct tty_buffer *tb = port->buf.tail;
-		*chars = tb->char_buf_ptr + tb->used;
-		*flags = tb->flag_buf_ptr + tb->used;
+		*chars = char_buf_ptr(tb, tb->used);
+		*flags = flag_buf_ptr(tb, tb->used);
 		tb->used += space;
 	}
 	return space;

@@ -407,8 +405,8 @@ static int
 receive_buf(struct tty_struct *tty, struct tty_buffer *head, int count)
 {
 	struct tty_ldisc *disc = tty->ldisc;
-	char 	      *p = head->char_buf_ptr + head->read;
-	unsigned char *f = head->flag_buf_ptr + head->read;
+	unsigned char *p = char_buf_ptr(head, head->read);
+	char	      *f = flag_buf_ptr(head, head->read);
 
 	if (disc->ops->receive_buf2)
 		count = disc->ops->receive_buf2(tty, p, f, count);

include/linux/tty.h +10 -2

index 57a70d1..87bbaa3 100644
@@ -31,8 +31,6 @@
 
 struct tty_buffer {
 	struct tty_buffer *next;
-	char *char_buf_ptr;
-	unsigned char *flag_buf_ptr;
 	int used;
 	int size;
 	int commit;

@@ -41,6 +39,16 @@ struct tty_buffer {
 	unsigned long data[0];
 };
 
+static inline unsigned char *char_buf_ptr(struct tty_buffer *b, int ofs)
+{
+	return ((unsigned char *)b->data) + ofs;
+}
+
+static inline char *flag_buf_ptr(struct tty_buffer *b, int ofs)
+{
+	return (char *)char_buf_ptr(b, ofs) + b->size;
+}
+
 /*
  * We default to dicing tty buffer allocations to this many characters
  * in order to avoid multiple page allocations. We know the size of

include/linux/tty_flip.h +2 -2

index e0f2526..ad03039 100644
@@ -18,8 +18,8 @@ static inline int tty_insert_flip_char(struct tty_port *port,
 {
 	struct tty_buffer *tb = port->buf.tail;
 	if (tb && tb->used < tb->size) {
-		tb->flag_buf_ptr[tb->used] = flag;
-		tb->char_buf_ptr[tb->used++] = ch;
+		*flag_buf_ptr(tb, tb->used) = flag;
+		*char_buf_ptr(tb, tb->used++) = ch;
 		return 1;
 	}
 	return tty_insert_flip_string_flags(port, &ch, &flag, 1);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #170 lore raw

PATCH v2 2/16 tty: Fix flip buffer free list

In reply to Peter Hurley

Since flip buffers are size-aligned to 256 bytes and all flip buffers 512-bytes or larger are not added to the free list, the free list only contains 256-byte flip buffers.

Remove the list search when allocating a new flip buffer.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

drivers/tty/tty_buffer.c +10 -6

index 170674c..a5e3962 100644
@@ -18,6 +18,10 @@
 #include <linux/module.h>
 #include <linux/ratelimit.h>
 
+
+#define MIN_TTYB_SIZE	256
+#define TTYB_ALIGN_MASK	255
+
 /**
  *	tty_buffer_free_all		-	free buffers used by a tty
  *	@tty: tty to free from

@@ -94,7 +98,7 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
 	buf->memory_used -= b->size;
 	WARN_ON(buf->memory_used < 0);
 
-	if (b->size >= 512)
+	if (b->size > MIN_TTYB_SIZE)
 		kfree(b);
 	else {
 		b->next = buf->free;

@@ -176,9 +180,10 @@ void tty_buffer_flush(struct tty_struct *tty)
 static struct tty_buffer *tty_buffer_find(struct tty_port *port, size_t size)
 {
 	struct tty_buffer **tbh = &port->buf.free;
-	while ((*tbh) != NULL) {
-		struct tty_buffer *t = *tbh;
-		if (t->size >= size) {
+	if (size <= MIN_TTYB_SIZE) {
+		if (*tbh) {
+			struct tty_buffer *t = *tbh;
+
 			*tbh = t->next;
 			t->next = NULL;
 			t->used = 0;

@@ -187,10 +192,9 @@ static struct tty_buffer *tty_buffer_find(struct tty_port *port, size_t size)
 			port->buf.memory_used += t->size;
 			return t;
 		}
-		tbh = &((*tbh)->next);
 	}
 	/* Round the buffer size out */
-	size = (size + 0xFF) & ~0xFF;
+	size = __ALIGN_MASK(size, TTYB_ALIGN_MASK);
 	return tty_buffer_alloc(port, size);
 	/* Should possibly check if this fails for the largest buffer we
 	   have queued and recycle that ? */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #171 lore raw

PATCH v2 3/16 tty: Factor flip buffer initialization into helper function

In reply to Peter Hurley

Factor shared code; prepare for adding 0-sized sentinel flip buffer.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

drivers/tty/tty_buffer.c +12 -9

index a5e3962..56d4602 100644
@@ -22,6 +22,15 @@
 #define MIN_TTYB_SIZE	256
 #define TTYB_ALIGN_MASK	255
 
+static void tty_buffer_reset(struct tty_buffer *p, size_t size)
+{
+	p->used = 0;
+	p->size = size;
+	p->next = NULL;
+	p->commit = 0;
+	p->read = 0;
+}
+
 /**
  *	tty_buffer_free_all		-	free buffers used by a tty
  *	@tty: tty to free from

@@ -70,11 +79,8 @@ static struct tty_buffer *tty_buffer_alloc(struct tty_port *port, size_t size)
 	p = kmalloc(sizeof(struct tty_buffer) + 2 * size, GFP_ATOMIC);
 	if (p == NULL)
 		return NULL;
-	p->used = 0;
-	p->size = size;
-	p->next = NULL;
-	p->commit = 0;
-	p->read = 0;
+
+	tty_buffer_reset(p, size);
 	port->buf.memory_used += size;
 	return p;
 }

@@ -185,10 +191,7 @@ static struct tty_buffer *tty_buffer_find(struct tty_port *port, size_t size)
 			struct tty_buffer *t = *tbh;
 
 			*tbh = t->next;
-			t->next = NULL;
-			t->used = 0;
-			t->commit = 0;
-			t->read = 0;
+			tty_buffer_reset(t, t->size);
 			port->buf.memory_used += t->size;
 			return t;
 		}

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #172 lore raw

PATCH v2 4/16 tty: Merge tty_buffer_find() into tty_buffer_alloc()

In reply to Peter Hurley

tty_buffer_find() implements a simple free list lookaside cache. Merge this functionality into tty_buffer_alloc() to reflect the more traditional alloc/free symmetry.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 50 +++++++++++++++++-------------------------------
 1 file changed, 18 insertions(+), 32 deletions(-)

drivers/tty/tty_buffer.c +18 ~~-32~~

index 56d4602..a428fa2 100644
@@ -64,6 +64,8 @@ void tty_buffer_free_all(struct tty_port *port)
  *	@size: desired size (characters)
  *
  *	Allocate a new tty buffer to hold the desired number of characters.
+ *	We round our buffers off in 256 character chunks to get better
+ *	allocation behaviour.
  *	Return NULL if out of memory or the allocation would exceed the
  *	per device queue
  *

@@ -72,14 +74,29 @@ void tty_buffer_free_all(struct tty_port *port)
 
 static struct tty_buffer *tty_buffer_alloc(struct tty_port *port, size_t size)
 {
+	struct tty_buffer **tbh = &port->buf.free;
 	struct tty_buffer *p;
 
+	/* Round the buffer size out */
+	size = __ALIGN_MASK(size, TTYB_ALIGN_MASK);
+
+	if (size <= MIN_TTYB_SIZE) {
+		if (*tbh) {
+			p = *tbh;
+			*tbh = p->next;
+			goto found;
+		}
+	}
+
+	/* Should possibly check if this fails for the largest buffer we
+	   have queued and recycle that ? */
 	if (port->buf.memory_used + size > 65536)
 		return NULL;
 	p = kmalloc(sizeof(struct tty_buffer) + 2 * size, GFP_ATOMIC);
 	if (p == NULL)
 		return NULL;
 
+found:
 	tty_buffer_reset(p, size);
 	port->buf.memory_used += size;
 	return p;

@@ -172,37 +189,6 @@ void tty_buffer_flush(struct tty_struct *tty)
 }
 
 /**
- *	tty_buffer_find		-	find a free tty buffer
- *	@tty: tty owning the buffer
- *	@size: characters wanted
- *
- *	Locate an existing suitable tty buffer or if we are lacking one then
- *	allocate a new one. We round our buffers off in 256 character chunks
- *	to get better allocation behaviour.
- *
- *	Locking: Caller must hold tty->buf.lock
- */
-
-static struct tty_buffer *tty_buffer_find(struct tty_port *port, size_t size)
-{
-	struct tty_buffer **tbh = &port->buf.free;
-	if (size <= MIN_TTYB_SIZE) {
-		if (*tbh) {
-			struct tty_buffer *t = *tbh;
-
-			*tbh = t->next;
-			tty_buffer_reset(t, t->size);
-			port->buf.memory_used += t->size;
-			return t;
-		}
-	}
-	/* Round the buffer size out */
-	size = __ALIGN_MASK(size, TTYB_ALIGN_MASK);
-	return tty_buffer_alloc(port, size);
-	/* Should possibly check if this fails for the largest buffer we
-	   have queued and recycle that ? */
-}
-/**
  *	tty_buffer_request_room		-	grow tty buffer if needed
  *	@tty: tty structure
  *	@size: size desired

@@ -230,7 +216,7 @@ int tty_buffer_request_room(struct tty_port *port, size_t size)
 
 	if (left < size) {
 		/* This is the slow path - looking for new buffers to use */
-		if ((n = tty_buffer_find(port, size)) != NULL) {
+		if ((n = tty_buffer_alloc(port, size)) != NULL) {
 			if (b != NULL) {
 				b->next = n;
 				b->commit = b->used;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #173 lore raw

PATCH v2 5/16 tty: Use generic names for flip buffer list cursors

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

drivers/tty/tty_buffer.c +10 ~~-10~~

index a428fa2..0259a76 100644
@@ -44,15 +44,15 @@ static void tty_buffer_reset(struct tty_buffer *p, size_t size)
 void tty_buffer_free_all(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
-	struct tty_buffer *thead;
+	struct tty_buffer *p;
 
-	while ((thead = buf->head) != NULL) {
-		buf->head = thead->next;
-		kfree(thead);
+	while ((p = buf->head) != NULL) {
+		buf->head = p->next;
+		kfree(p);
 	}
-	while ((thead = buf->free) != NULL) {
-		buf->free = thead->next;
-		kfree(thead);
+	while ((p = buf->free) != NULL) {
+		buf->free = p->next;
+		kfree(p);
 	}
 	buf->tail = NULL;
 	buf->memory_used = 0;

@@ -143,13 +143,13 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
 static void __tty_buffer_flush(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
-	struct tty_buffer *thead;
+	struct tty_buffer *next;
 
 	if (unlikely(buf->head == NULL))
 		return;
-	while ((thead = buf->head->next) != NULL) {
+	while ((next = buf->head->next) != NULL) {
 		tty_buffer_free(port, buf->head);
-		buf->head = thead;
+		buf->head = next;
 	}
 	WARN_ON(buf->head != buf->tail);
 	buf->head->read = buf->head->commit;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #174 lore raw

PATCH v2 6/16 tty: Use lockless flip buffer free list

In reply to Peter Hurley

In preparation for lockless flip buffers, make the flip buffer free list lockless.

NB: using llist is not the optimal solution, as the driver and buffer work may contend over the llist head unnecessarily. However, test measurements indicate this contention is low.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 29 ++++++++++++-----------------
 include/linux/llist.h    | 23 +++++++++++++++++++++++
 include/linux/tty.h      |  8 ++++++--
 3 files changed, 41 insertions(+), 19 deletions(-)

drivers/tty/tty_buffer.c +12 ~~-17~~

index 0259a76..069640e 100644
@@ -44,16 +44,17 @@ static void tty_buffer_reset(struct tty_buffer *p, size_t size)
 void tty_buffer_free_all(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
-	struct tty_buffer *p;
+	struct tty_buffer *p, *next;
+	struct llist_node *llist;
 
 	while ((p = buf->head) != NULL) {
 		buf->head = p->next;
 		kfree(p);
 	}
-	while ((p = buf->free) != NULL) {
-		buf->free = p->next;
+	llist = llist_del_all(&buf->free);
+	llist_for_each_entry_safe(p, next, llist, free)
 		kfree(p);
-	}
+
 	buf->tail = NULL;
 	buf->memory_used = 0;
 }

@@ -68,22 +69,20 @@ void tty_buffer_free_all(struct tty_port *port)
  *	allocation behaviour.
  *	Return NULL if out of memory or the allocation would exceed the
  *	per device queue
- *
- *	Locking: Caller must hold tty->buf.lock
  */
 
 static struct tty_buffer *tty_buffer_alloc(struct tty_port *port, size_t size)
 {
-	struct tty_buffer **tbh = &port->buf.free;
+	struct llist_node *free;
 	struct tty_buffer *p;
 
 	/* Round the buffer size out */
 	size = __ALIGN_MASK(size, TTYB_ALIGN_MASK);
 
 	if (size <= MIN_TTYB_SIZE) {
-		if (*tbh) {
-			p = *tbh;
-			*tbh = p->next;
+		free = llist_del_first(&port->buf.free);
+		if (free) {
+			p = llist_entry(free, struct tty_buffer, free);
 			goto found;
 		}
 	}

@@ -109,8 +108,6 @@ found:
  *
  *	Free a tty buffer, or add it to the free list according to our
  *	internal strategy
- *
- *	Locking: Caller must hold tty->buf.lock
  */
 
 static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)

@@ -123,10 +120,8 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
 
 	if (b->size > MIN_TTYB_SIZE)
 		kfree(b);
-	else {
-		b->next = buf->free;
-		buf->free = b;
-	}
+	else
+		llist_add(&b->free, &buf->free);
 }
 
 /**

@@ -542,7 +537,7 @@ void tty_buffer_init(struct tty_port *port)
 	spin_lock_init(&buf->lock);
 	buf->head = NULL;
 	buf->tail = NULL;
-	buf->free = NULL;
+	init_llist_head(&buf->free);
 	buf->memory_used = 0;
 	INIT_WORK(&buf->work, flush_to_ldisc);
 }

include/linux/llist.h +23 -0

index a5199f6..97cf31d 100644
@@ -125,6 +125,29 @@ static inline void init_llist_head(struct llist_head *list)
 	     (pos) = llist_entry((pos)->member.next, typeof(*(pos)), member))
 
 /**
+ * llist_for_each_entry_safe - iterate over some deleted entries of lock-less list of given type
+ *			       safe against removal of list entry
+ * @pos:	the type * to use as a loop cursor.
+ * @n:		another type * to use as temporary storage
+ * @node:	the first entry of deleted list entries.
+ * @member:	the name of the llist_node with the struct.
+ *
+ * In general, some entries of the lock-less list can be traversed
+ * safely only after being removed from list, so start with an entry
+ * instead of list head.
+ *
+ * If being used on entries deleted from lock-less list directly, the
+ * traverse order is from the newest to the oldest added entry.  If
+ * you want to traverse from the oldest to the newest, you must
+ * reverse the order by yourself before traversing.
+ */
+#define llist_for_each_entry_safe(pos, n, node, member)			       \
+	for (pos = llist_entry((node), typeof(*pos), member);		       \
+	     &pos->member != NULL &&					       \
+	        (n = llist_entry(pos->member.next, typeof(*n), member), true); \
+	     pos = n)
+
+/**
  * llist_empty - tests whether a lock-less list is empty
  * @head:	the list to test
  *

include/linux/tty.h +6 -2

index 87bbaa3..5043b12 100644
@@ -11,6 +11,7 @@
 #include <linux/tty_flags.h>
 #include <uapi/linux/tty.h>
 #include <linux/rwsem.h>
+#include <linux/llist.h>

@@ -30,7 +31,10 @@
 #define __DISABLED_CHAR '\0'
 
 struct tty_buffer {
-	struct tty_buffer *next;
+	union {
+		struct tty_buffer *next;
+		struct llist_node free;
+	};
 	int used;
 	int size;
 	int commit;

@@ -65,7 +69,7 @@ struct tty_bufhead {
 	spinlock_t lock;
 	struct tty_buffer *head;	/* Queue head */
 	struct tty_buffer *tail;	/* Active buffer */
-	struct tty_buffer *free;	/* Free queue head */
+	struct llist_head free;		/* Free queue head */
 	int memory_used;		/* Buffer space used excluding
 								free queue */
 };

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #175 lore raw

PATCH v2 7/16 tty: Simplify flip buffer list with 0-sized sentinel

In reply to Peter Hurley

Use a 0-sized sentinel to avoid assigning the head ptr from the driver side thread. This also eliminates testing head/tail for NULL.

When the sentinel is first 'consumed' by the buffer work (or by tty_buffer_flush()), it is detached from the list but not freed nor added to the free list. Both buffer work and tty_buffer_flush() continue to preserve at least 1 flip buffer to which head & tail is pointed.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 49 ++++++++++++++++++------------------------------
 include/linux/tty.h      |  1 +
 2 files changed, 19 insertions(+), 31 deletions(-)

drivers/tty/tty_buffer.c +18 ~~-31~~

index 069640e..231b7a8 100644
@@ -49,13 +49,16 @@ void tty_buffer_free_all(struct tty_port *port)
 
 	while ((p = buf->head) != NULL) {
 		buf->head = p->next;
-		kfree(p);
+		if (p->size > 0)
+			kfree(p);
 	}
 	llist = llist_del_all(&buf->free);
 	llist_for_each_entry_safe(p, next, llist, free)
 		kfree(p);
 
-	buf->tail = NULL;
+	tty_buffer_reset(&buf->sentinel, 0);
+	buf->head = &buf->sentinel;
+	buf->tail = &buf->sentinel;
 	buf->memory_used = 0;
 }

@@ -120,7 +123,7 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
 
 	if (b->size > MIN_TTYB_SIZE)
 		kfree(b);
-	else
+	else if (b->size > 0)
 		llist_add(&b->free, &buf->free);
 }

@@ -140,8 +143,6 @@ static void __tty_buffer_flush(struct tty_port *port)
 	struct tty_bufhead *buf = &port->buf;
 	struct tty_buffer *next;
 
-	if (unlikely(buf->head == NULL))
-		return;
 	while ((next = buf->head->next) != NULL) {
 		tty_buffer_free(port, buf->head);
 		buf->head = next;

@@ -200,23 +201,14 @@ int tty_buffer_request_room(struct tty_port *port, size_t size)
 	int left;
 	unsigned long flags;
 	spin_lock_irqsave(&buf->lock, flags);
-	/* OPTIMISATION: We could keep a per tty "zero" sized buffer to
-	   remove this conditional if its worth it. This would be invisible
-	   to the callers */
 	b = buf->tail;
-	if (b != NULL)
-		left = b->size - b->used;
-	else
-		left = 0;
+	left = b->size - b->used;
 
 	if (left < size) {
 		/* This is the slow path - looking for new buffers to use */
 		if ((n = tty_buffer_alloc(port, size)) != NULL) {
-			if (b != NULL) {
-				b->next = n;
-				b->commit = b->used;
-			} else
-				buf->head = n;
+			b->next = n;
+			b->commit = b->used;
 			buf->tail = n;
 		} else
 			size = left;

@@ -247,10 +239,8 @@ int tty_insert_flip_string_fixed_flag(struct tty_port *port,
 		int goal = min_t(size_t, size - copied, TTY_BUFFER_PAGE);
 		int space = tty_buffer_request_room(port, goal);
 		struct tty_buffer *tb = port->buf.tail;
-		/* If there is no space then tb may be NULL */
-		if (unlikely(space == 0)) {
+		if (unlikely(space == 0))
 			break;
-		}
 		memcpy(char_buf_ptr(tb, tb->used), chars, space);
 		memset(flag_buf_ptr(tb, tb->used), flag, space);
 		tb->used += space;

@@ -285,10 +275,8 @@ int tty_insert_flip_string_flags(struct tty_port *port,
 		int goal = min_t(size_t, size - copied, TTY_BUFFER_PAGE);
 		int space = tty_buffer_request_room(port, goal);
 		struct tty_buffer *tb = port->buf.tail;
-		/* If there is no space then tb may be NULL */
-		if (unlikely(space == 0)) {
+		if (unlikely(space == 0))
 			break;
-		}
 		memcpy(char_buf_ptr(tb, tb->used), chars, space);
 		memcpy(flag_buf_ptr(tb, tb->used), flags, space);
 		tb->used += space;

@@ -322,8 +310,7 @@ void tty_schedule_flip(struct tty_port *port)
 	WARN_ON(port->low_latency);
 
 	spin_lock_irqsave(&buf->lock, flags);
-	if (buf->tail != NULL)
-		buf->tail->commit = buf->tail->used;
+	buf->tail->commit = buf->tail->used;
 	spin_unlock_irqrestore(&buf->lock, flags);
 	schedule_work(&buf->work);
 }

@@ -438,8 +425,8 @@ static void flush_to_ldisc(struct work_struct *work)
 	spin_lock_irqsave(&buf->lock, flags);
 
 	if (!test_and_set_bit(TTYP_FLUSHING, &port->iflags)) {
-		struct tty_buffer *head;
-		while ((head = buf->head) != NULL) {
+		while (1) {
+			struct tty_buffer *head = buf->head;
 			int count;
 
 			count = head->commit - head->read;

@@ -509,8 +496,7 @@ void tty_flip_buffer_push(struct tty_port *port)
 	unsigned long flags;
 
 	spin_lock_irqsave(&buf->lock, flags);
-	if (buf->tail != NULL)
-		buf->tail->commit = buf->tail->used;
+	buf->tail->commit = buf->tail->used;
 	spin_unlock_irqrestore(&buf->lock, flags);
 
 	if (port->low_latency)

@@ -535,8 +521,9 @@ void tty_buffer_init(struct tty_port *port)
 	struct tty_bufhead *buf = &port->buf;
 
 	spin_lock_init(&buf->lock);
-	buf->head = NULL;
-	buf->tail = NULL;
+	tty_buffer_reset(&buf->sentinel, 0);
+	buf->head = &buf->sentinel;
+	buf->tail = &buf->sentinel;
 	init_llist_head(&buf->free);
 	buf->memory_used = 0;
 	INIT_WORK(&buf->work, flush_to_ldisc);

include/linux/tty.h +1 -0

index 5043b12..2e93eb8 100644
@@ -67,6 +67,7 @@ static inline char *flag_buf_ptr(struct tty_buffer *b, int ofs)
 struct tty_bufhead {
 	struct work_struct work;
 	spinlock_t lock;
+	struct tty_buffer sentinel;
 	struct tty_buffer *head;	/* Queue head */
 	struct tty_buffer *tail;	/* Active buffer */
 	struct llist_head free;		/* Free queue head */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #176 lore raw

PATCH v2 8/16 tty: Track flip buffer memory limit atomically

In reply to Peter Hurley

Lockless flip buffers require atomically updating the bytes-in-use watermark.

The pty driver also peeks at the watermark value to limit memory consumption to a much lower value than the default; query the watermark with new fn, tty_buffer_space_avail().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/pty.c        | 10 +++-------
 drivers/tty/tty_buffer.c | 37 +++++++++++++++++++++++++++++++------
 include/linux/tty.h      |  3 +--
 include/linux/tty_flip.h |  1 +
 4 files changed, 36 insertions(+), 15 deletions(-)

drivers/tty/pty.c +3 -7

index bbd9693..0634dd9 100644
@@ -89,17 +89,13 @@ static void pty_unthrottle(struct tty_struct *tty)
  *	pty_space	-	report space left for writing
  *	@to: tty we are writing into
  *
- *	The tty buffers allow 64K but we sneak a peak and clip at 8K this
- *	allows a lot of overspill room for echo and other fun messes to
- *	be handled properly
+ *	Limit the buffer space used by ptys to 8k.
  */
 
 static int pty_space(struct tty_struct *to)
 {
-	int n = 8192 - to->port->buf.memory_used;
-	if (n < 0)
-		return 0;
-	return n;
+	int n = tty_buffer_space_avail(to->port);
+	return min(n, 8192);
 }
 
 /**

drivers/tty/tty_buffer.c +31 -6

index 231b7a8..5d5a564 100644
@@ -22,6 +22,31 @@
 #define MIN_TTYB_SIZE	256
 #define TTYB_ALIGN_MASK	255
 
+/*
+ * Byte threshold to limit memory consumption for flip buffers.
+ * The actual memory limit is > 2x this amount.
+ */
+#define TTYB_MEM_LIMIT	65536
+
+
+/**
+ *	tty_buffer_space_avail	-	return unused buffer space
+ *	@port - tty_port owning the flip buffer
+ *
+ *	Returns the # of bytes which can be written by the driver without
+ *	reaching the buffer limit.
+ *
+ *	Note: this does not guarantee that memory is available to write
+ *	the returned # of bytes (use tty_prepare_flip_string_xxx() to
+ *	pre-allocate if memory guarantee is required).
+ */
+
+int tty_buffer_space_avail(struct tty_port *port)
+{
+	int space = TTYB_MEM_LIMIT - atomic_read(&port->buf.memory_used);
+	return max(space, 0);
+}
+
 static void tty_buffer_reset(struct tty_buffer *p, size_t size)
 {
 	p->used = 0;

@@ -59,7 +84,8 @@ void tty_buffer_free_all(struct tty_port *port)
 	tty_buffer_reset(&buf->sentinel, 0);
 	buf->head = &buf->sentinel;
 	buf->tail = &buf->sentinel;
-	buf->memory_used = 0;
+
+	atomic_set(&buf->memory_used, 0);
 }
 
 /**

@@ -92,7 +118,7 @@ static struct tty_buffer *tty_buffer_alloc(struct tty_port *port, size_t size)
 
 	/* Should possibly check if this fails for the largest buffer we
 	   have queued and recycle that ? */
-	if (port->buf.memory_used + size > 65536)
+	if (atomic_read(&port->buf.memory_used) > TTYB_MEM_LIMIT)
 		return NULL;
 	p = kmalloc(sizeof(struct tty_buffer) + 2 * size, GFP_ATOMIC);
 	if (p == NULL)

@@ -100,7 +126,7 @@ static struct tty_buffer *tty_buffer_alloc(struct tty_port *port, size_t size)
 
 found:
 	tty_buffer_reset(p, size);
-	port->buf.memory_used += size;
+	atomic_add(size, &port->buf.memory_used);
 	return p;
 }

@@ -118,8 +144,7 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
 	struct tty_bufhead *buf = &port->buf;
 
 	/* Dumb strategy for now - should keep some stats */
-	buf->memory_used -= b->size;
-	WARN_ON(buf->memory_used < 0);
+	WARN_ON(atomic_sub_return(b->size, &buf->memory_used) < 0);
 
 	if (b->size > MIN_TTYB_SIZE)
 		kfree(b);

@@ -525,7 +550,7 @@ void tty_buffer_init(struct tty_port *port)
 	buf->head = &buf->sentinel;
 	buf->tail = &buf->sentinel;
 	init_llist_head(&buf->free);
-	buf->memory_used = 0;
+	atomic_set(&buf->memory_used, 0);
 	INIT_WORK(&buf->work, flush_to_ldisc);
 }

include/linux/tty.h +1 -2

index 2e93eb8..7c12454 100644
@@ -71,8 +71,7 @@ struct tty_bufhead {
 	struct tty_buffer *head;	/* Queue head */
 	struct tty_buffer *tail;	/* Active buffer */
 	struct llist_head free;		/* Free queue head */
-	int memory_used;		/* Buffer space used excluding
-								free queue */
+	atomic_t	   memory_used; /* In-use buffers excluding free list */
 };
 /*
  * When a break, frame error, or parity error happens, these codes are

include/linux/tty_flip.h +1 -0

index ad03039..6944ed2 100644
@@ -1,6 +1,7 @@
 #ifndef _LINUX_TTY_FLIP_H
 #define _LINUX_TTY_FLIP_H
 
+extern int tty_buffer_space_avail(struct tty_port *port);
 extern int tty_buffer_request_room(struct tty_port *port, size_t size);
 extern int tty_insert_flip_string_flags(struct tty_port *port,
 		const unsigned char *chars, const char *flags, size_t size);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #177 lore raw

PATCH v2 9/16 tty: Make driver-side flip buffers lockless

In reply to Peter Hurley

Driver-side flip buffer input is already single-threaded; 'publish' the .next link as the last operation on the tail buffer so the 'consumer' sees the already-completed flip buffer.

The commit buffer index is already 'published' by driver-side functions.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 31 ++++---------------------------
 1 file changed, 4 insertions(+), 27 deletions(-)

drivers/tty/tty_buffer.c +4 ~~-27~~

index 5d5a564..685757c 100644
@@ -62,8 +62,6 @@ static void tty_buffer_reset(struct tty_buffer *p, size_t size)
  *
  *	Remove all the buffers pending on a tty whether queued with data
  *	or in the free ring. Must be called when the tty is no longer in use
- *
- *	Locking: none
  */
 
 void tty_buffer_free_all(struct tty_port *port)

@@ -216,29 +214,26 @@ void tty_buffer_flush(struct tty_struct *tty)
  *
  *	Make at least size bytes of linear space available for the tty
  *	buffer. If we fail return the size we managed to find.
- *
- *	Locking: Takes port->buf.lock
  */
 int tty_buffer_request_room(struct tty_port *port, size_t size)
 {
 	struct tty_bufhead *buf = &port->buf;
 	struct tty_buffer *b, *n;
 	int left;
-	unsigned long flags;
-	spin_lock_irqsave(&buf->lock, flags);
+
 	b = buf->tail;
 	left = b->size - b->used;
 
 	if (left < size) {
 		/* This is the slow path - looking for new buffers to use */
 		if ((n = tty_buffer_alloc(port, size)) != NULL) {
-			b->next = n;
-			b->commit = b->used;
 			buf->tail = n;
+			b->commit = b->used;
+			smp_mb();
+			b->next = n;
 		} else
 			size = left;
 	}
-	spin_unlock_irqrestore(&buf->lock, flags);
 	return size;
 }
 EXPORT_SYMBOL_GPL(tty_buffer_request_room);

@@ -252,8 +247,6 @@ EXPORT_SYMBOL_GPL(tty_buffer_request_room);
  *
  *	Queue a series of bytes to the tty buffering. All the characters
  *	passed are marked with the supplied flag. Returns the number added.
- *
- *	Locking: Called functions may take port->buf.lock
  */
 
 int tty_insert_flip_string_fixed_flag(struct tty_port *port,

@@ -288,8 +281,6 @@ EXPORT_SYMBOL(tty_insert_flip_string_fixed_flag);
  *	Queue a series of bytes to the tty buffering. For each character
  *	the flags array indicates the status of the character. Returns the
  *	number added.
- *
- *	Locking: Called functions may take port->buf.lock
  */
 
 int tty_insert_flip_string_flags(struct tty_port *port,

@@ -324,19 +315,14 @@ EXPORT_SYMBOL(tty_insert_flip_string_flags);
  *	processing by the line discipline.
  *	Note that this function can only be used when the low_latency flag
  *	is unset. Otherwise the workqueue won't be flushed.
- *
- *	Locking: Takes port->buf.lock
  */
 
 void tty_schedule_flip(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
-	unsigned long flags;
 	WARN_ON(port->low_latency);
 
-	spin_lock_irqsave(&buf->lock, flags);
 	buf->tail->commit = buf->tail->used;
-	spin_unlock_irqrestore(&buf->lock, flags);
 	schedule_work(&buf->work);
 }
 EXPORT_SYMBOL(tty_schedule_flip);

@@ -352,8 +338,6 @@ EXPORT_SYMBOL(tty_schedule_flip);
  *	accounted for as ready for normal characters. This is used for drivers
  *	that need their own block copy routines into the buffer. There is no
  *	guarantee the buffer is a DMA target!
- *
- *	Locking: May call functions taking port->buf.lock
  */
 
 int tty_prepare_flip_string(struct tty_port *port, unsigned char **chars,

@@ -382,8 +366,6 @@ EXPORT_SYMBOL_GPL(tty_prepare_flip_string);
  *	accounted for as ready for characters. This is used for drivers
  *	that need their own block copy routines into the buffer. There is no
  *	guarantee the buffer is a DMA target!
- *
- *	Locking: May call functions taking port->buf.lock
  */
 
 int tty_prepare_flip_string_flags(struct tty_port *port,

@@ -511,18 +493,13 @@ void tty_flush_to_ldisc(struct tty_struct *tty)
  *
  *	In the event of the queue being busy for flipping the work will be
  *	held off and retried later.
- *
- *	Locking: tty buffer lock. Driver locks in low latency mode.
  */
 
 void tty_flip_buffer_push(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
-	unsigned long flags;
 
-	spin_lock_irqsave(&buf->lock, flags);
 	buf->tail->commit = buf->tail->used;
-	spin_unlock_irqrestore(&buf->lock, flags);
 
 	if (port->low_latency)
 		flush_to_ldisc(&buf->work);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:57 UTC #178 lore raw

PATCH v2 10/16 tty: Ensure single-threaded flip buffer consumer with mutex

In reply to Peter Hurley

The buffer work may race with parallel tty_buffer_flush. Use a mutex to guarantee exclusive modify access to the head flip buffer.

Remove the unneeded spin lock.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/staging/dgrp/dgrp_tty.c |  2 ++
 drivers/tty/tty_buffer.c        | 40 +++++++++++++++++++---------------------
 include/linux/tty.h             |  2 +-
 3 files changed, 22 insertions(+), 22 deletions(-)

drivers/staging/dgrp/dgrp_tty.c +2 -0

index 654f601..0d52de3 100644
@@ -1120,7 +1120,9 @@ static void dgrp_tty_close(struct tty_struct *tty, struct file *file)
 				if (!sent_printer_offstr)
 					dgrp_tty_flush_buffer(tty);
 
+				spin_unlock_irqrestore(&nd->nd_lock, lock_flags);
 				tty_ldisc_flush(tty);
+				spin_lock_irqsave(&nd->nd_lock, lock_flags);
 				break;
 		}

drivers/tty/tty_buffer.c +19 ~~-21~~

index 685757c..c3c606c 100644
@@ -157,8 +157,6 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
  *	flush all the buffers containing receive data. Caller must
  *	hold the buffer lock and must have ensured no parallel flush to
  *	ldisc is running.
- *
- *	Locking: Caller must hold tty->buf.lock
  */
 
 static void __tty_buffer_flush(struct tty_port *port)

@@ -182,29 +180,29 @@ static void __tty_buffer_flush(struct tty_port *port)
  *	being processed by flush_to_ldisc then we defer the processing
  *	to that function
  *
- *	Locking: none
+ *	Locking: takes flush_mutex to ensure single-threaded flip buffer
+ *		 'consumer'
  */
 
 void tty_buffer_flush(struct tty_struct *tty)
 {
 	struct tty_port *port = tty->port;
 	struct tty_bufhead *buf = &port->buf;
-	unsigned long flags;
-
-	spin_lock_irqsave(&buf->lock, flags);
 
+	mutex_lock(&buf->flush_mutex);
 	/* If the data is being pushed to the tty layer then we can't
 	   process it here. Instead set a flag and the flush_to_ldisc
 	   path will process the flush request before it exits */
 	if (test_bit(TTYP_FLUSHING, &port->iflags)) {
 		set_bit(TTYP_FLUSHPENDING, &port->iflags);
-		spin_unlock_irqrestore(&buf->lock, flags);
+		mutex_unlock(&buf->flush_mutex);
 		wait_event(tty->read_wait,
 				test_bit(TTYP_FLUSHPENDING, &port->iflags) == 0);
 		return;
-	} else
-		__tty_buffer_flush(port);
-	spin_unlock_irqrestore(&buf->lock, flags);
+	}
+
+	__tty_buffer_flush(port);
+	mutex_unlock(&buf->flush_mutex);
 }
 
 /**

@@ -408,9 +406,10 @@ receive_buf(struct tty_struct *tty, struct tty_buffer *head, int count)
  *	This routine is called out of the software interrupt to flush data
  *	from the buffer chain to the line discipline.
  *
- *	Locking: holds tty->buf.lock to guard buffer list. Drops the lock
- *	while invoking the line discipline receive_buf method. The
- *	receive_buf method is single threaded for each tty instance.
+ *	The receive_buf method is single threaded for each tty instance.
+ *
+ *	Locking: takes flush_mutex to ensure single-threaded flip buffer
+ *		 'consumer'
  */
 
 static void flush_to_ldisc(struct work_struct *work)

@@ -418,7 +417,6 @@ static void flush_to_ldisc(struct work_struct *work)
 	struct tty_port *port = container_of(work, struct tty_port, buf.work);
 	struct tty_bufhead *buf = &port->buf;
 	struct tty_struct *tty;
-	unsigned long 	flags;
 	struct tty_ldisc *disc;
 
 	tty = port->itty;

@@ -429,7 +427,7 @@ static void flush_to_ldisc(struct work_struct *work)
 	if (disc == NULL)
 		return;
 
-	spin_lock_irqsave(&buf->lock, flags);
+	mutex_lock(&buf->flush_mutex);
 
 	if (!test_and_set_bit(TTYP_FLUSHING, &port->iflags)) {
 		while (1) {

@@ -444,11 +442,13 @@ static void flush_to_ldisc(struct work_struct *work)
 				tty_buffer_free(port, head);
 				continue;
 			}
-			spin_unlock_irqrestore(&buf->lock, flags);
+
+			mutex_unlock(&buf->flush_mutex);
 
 			count = receive_buf(tty, head, count);
 
-			spin_lock_irqsave(&buf->lock, flags);
+			mutex_lock(&buf->flush_mutex);
+
 			/* Ldisc or user is trying to flush the buffers.
 			   We may have a deferred request to flush the
 			   input buffer, if so pull the chain under the lock

@@ -464,7 +464,7 @@ static void flush_to_ldisc(struct work_struct *work)
 		clear_bit(TTYP_FLUSHING, &port->iflags);
 	}
 
-	spin_unlock_irqrestore(&buf->lock, flags);
+	mutex_unlock(&buf->flush_mutex);
 
 	tty_ldisc_deref(disc);
 }

@@ -514,15 +514,13 @@ EXPORT_SYMBOL(tty_flip_buffer_push);
  *
  *	Set up the initial state of the buffer management for a tty device.
  *	Must be called before the other tty buffer functions are used.
- *
- *	Locking: none
  */
 
 void tty_buffer_init(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
 
-	spin_lock_init(&buf->lock);
+	mutex_init(&buf->flush_mutex);
 	tty_buffer_reset(&buf->sentinel, 0);
 	buf->head = &buf->sentinel;
 	buf->tail = &buf->sentinel;

include/linux/tty.h +1 -1

index 7c12454..1c8fef0 100644
@@ -66,7 +66,7 @@ static inline char *flag_buf_ptr(struct tty_buffer *b, int ofs)
 
 struct tty_bufhead {
 	struct work_struct work;
-	spinlock_t lock;
+	struct mutex flush_mutex;
 	struct tty_buffer sentinel;
 	struct tty_buffer *head;	/* Queue head */
 	struct tty_buffer *tail;	/* Active buffer */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #179 lore raw

PATCH v2 11/16 tty: Only perform flip buffer flush from tty_buffer_flush()

In reply to Peter Hurley

Now that dropping the buffer lock is not necessary (as result of converting the spin lock to a mutex), the flip buffer flush no longer needs to be handled by the buffer work.

Simply signal a flush is required; the buffer work will exit the i/o loop, which allows tty_buffer_flush() to proceed.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 63 ++++++++++++++++--------------------------------
 include/linux/tty.h      |  1 -
 2 files changed, 21 insertions(+), 43 deletions(-)

drivers/tty/tty_buffer.c +21 ~~-42~~

index c3c606c..39cae61 100644
@@ -189,19 +189,11 @@ void tty_buffer_flush(struct tty_struct *tty)
 	struct tty_port *port = tty->port;
 	struct tty_bufhead *buf = &port->buf;
 
-	mutex_lock(&buf->flush_mutex);
-	/* If the data is being pushed to the tty layer then we can't
-	   process it here. Instead set a flag and the flush_to_ldisc
-	   path will process the flush request before it exits */
-	if (test_bit(TTYP_FLUSHING, &port->iflags)) {
-		set_bit(TTYP_FLUSHPENDING, &port->iflags);
-		mutex_unlock(&buf->flush_mutex);
-		wait_event(tty->read_wait,
-				test_bit(TTYP_FLUSHPENDING, &port->iflags) == 0);
-		return;
-	}
+	set_bit(TTYP_FLUSHPENDING, &port->iflags);
 
+	mutex_lock(&buf->flush_mutex);
 	__tty_buffer_flush(port);
+	clear_bit(TTYP_FLUSHPENDING, &port->iflags);
 	mutex_unlock(&buf->flush_mutex);
 }

@@ -429,39 +421,26 @@ static void flush_to_ldisc(struct work_struct *work)
 
 	mutex_lock(&buf->flush_mutex);
 
-	if (!test_and_set_bit(TTYP_FLUSHING, &port->iflags)) {
-		while (1) {
-			struct tty_buffer *head = buf->head;
-			int count;
-
-			count = head->commit - head->read;
-			if (!count) {
-				if (head->next == NULL)
-					break;
-				buf->head = head->next;
-				tty_buffer_free(port, head);
-				continue;
-			}
-
-			mutex_unlock(&buf->flush_mutex);
-
-			count = receive_buf(tty, head, count);
-
-			mutex_lock(&buf->flush_mutex);
-
-			/* Ldisc or user is trying to flush the buffers.
-			   We may have a deferred request to flush the
-			   input buffer, if so pull the chain under the lock
-			   and empty the queue */
-			if (test_bit(TTYP_FLUSHPENDING, &port->iflags)) {
-				__tty_buffer_flush(port);
-				clear_bit(TTYP_FLUSHPENDING, &port->iflags);
-				wake_up(&tty->read_wait);
-				break;
-			} else if (!count)
+	while (1) {
+		struct tty_buffer *head = buf->head;
+		int count;
+
+		/* Ldisc or user is trying to flush the buffers. */
+		if (test_bit(TTYP_FLUSHPENDING, &port->iflags))
+			break;
+
+		count = head->commit - head->read;
+		if (!count) {
+			if (head->next == NULL)
 				break;
+			buf->head = head->next;
+			tty_buffer_free(port, head);
+			continue;
 		}
-		clear_bit(TTYP_FLUSHING, &port->iflags);
+
+		count = receive_buf(tty, head, count);
+		if (!count)
+			break;
 	}
 
 	mutex_unlock(&buf->flush_mutex);

include/linux/tty.h +0 -1

index 1c8fef0..1d5bacc 100644
@@ -213,7 +213,6 @@ struct tty_port {
 	wait_queue_head_t	delta_msr_wait;	/* Modem status change */
 	unsigned long		flags;		/* TTY flags ASY_*/
 	unsigned long		iflags;		/* TTYP_ internal flags */
-#define TTYP_FLUSHING			1  /* Flushing to ldisc in progress */
 #define TTYP_FLUSHPENDING		2  /* Queued buffer flush pending */
 	unsigned char		console:1,	/* port is a console */
 				low_latency:1;	/* direct buffer flush */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #180 lore raw

PATCH v2 12/16 tty: Avoid false-sharing flip buffer ptrs

In reply to Peter Hurley

Separate the head and tail ptrs to avoid cache-line contention (so called 'false-sharing') between concurrent threads.

Signed-off-by Peter Hurley <[email protected]>

---
 include/linux/tty.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

include/linux/tty.h +2 -2

index 1d5bacc..b8e8adf 100644
@@ -65,13 +65,13 @@ static inline char *flag_buf_ptr(struct tty_buffer *b, int ofs)
 
 
 struct tty_bufhead {
+	struct tty_buffer *head;	/* Queue head */
 	struct work_struct work;
 	struct mutex flush_mutex;
 	struct tty_buffer sentinel;
-	struct tty_buffer *head;	/* Queue head */
-	struct tty_buffer *tail;	/* Active buffer */
 	struct llist_head free;		/* Free queue head */
 	atomic_t	   memory_used; /* In-use buffers excluding free list */
+	struct tty_buffer *tail;	/* Active buffer */
 };
 /*
  * When a break, frame error, or parity error happens, these codes are

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #181 lore raw

PATCH v2 13/16 tty: Use non-atomic state to signal flip buffer flush pending

In reply to Peter Hurley

Atomic bit ops are no longer required to indicate a flip buffer flush is pending, as the flush_mutex is sufficient barrier.

Remove the unnecessary port .iflags field and localize flip buffer state to struct tty_bufhead.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 7 ++++---
 include/linux/tty.h      | 3 +--
 2 files changed, 5 insertions(+), 5 deletions(-)

drivers/tty/tty_buffer.c +4 -3

index 39cae61..fb042b9 100644
@@ -189,11 +189,11 @@ void tty_buffer_flush(struct tty_struct *tty)
 	struct tty_port *port = tty->port;
 	struct tty_bufhead *buf = &port->buf;
 
-	set_bit(TTYP_FLUSHPENDING, &port->iflags);
+	buf->flushpending = 1;
 
 	mutex_lock(&buf->flush_mutex);
 	__tty_buffer_flush(port);
-	clear_bit(TTYP_FLUSHPENDING, &port->iflags);
+	buf->flushpending = 0;
 	mutex_unlock(&buf->flush_mutex);
 }

@@ -426,7 +426,7 @@ static void flush_to_ldisc(struct work_struct *work)
 		int count;
 
 		/* Ldisc or user is trying to flush the buffers. */
-		if (test_bit(TTYP_FLUSHPENDING, &port->iflags))
+		if (buf->flushpending)
 			break;
 
 		count = head->commit - head->read;

@@ -505,6 +505,7 @@ void tty_buffer_init(struct tty_port *port)
 	buf->tail = &buf->sentinel;
 	init_llist_head(&buf->free);
 	atomic_set(&buf->memory_used, 0);
+	buf->flushpending = 0;
 	INIT_WORK(&buf->work, flush_to_ldisc);
 }

include/linux/tty.h +1 -2

index b8e8adf..991575f 100644
@@ -68,6 +68,7 @@ struct tty_bufhead {
 	struct tty_buffer *head;	/* Queue head */
 	struct work_struct work;
 	struct mutex flush_mutex;
+	unsigned int flushpending:1;
 	struct tty_buffer sentinel;
 	struct llist_head free;		/* Free queue head */
 	atomic_t	   memory_used; /* In-use buffers excluding free list */

@@ -212,8 +213,6 @@ struct tty_port {
 	wait_queue_head_t	close_wait;	/* Close waiters */
 	wait_queue_head_t	delta_msr_wait;	/* Modem status change */
 	unsigned long		flags;		/* TTY flags ASY_*/
-	unsigned long		iflags;		/* TTYP_ internal flags */
-#define TTYP_FLUSHPENDING		2  /* Queued buffer flush pending */
 	unsigned char		console:1,	/* port is a console */
 				low_latency:1;	/* direct buffer flush */
 	struct mutex		mutex;		/* Locking */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:57 UTC #182 lore raw

PATCH v2 14/16 tty: Merge __tty_flush_buffer() into lone call site

In reply to Peter Hurley

__tty_flush_buffer() is now only called by tty_flush_buffer(); merge functions.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 29 ++++++-----------------------
 1 file changed, 6 insertions(+), 23 deletions(-)

drivers/tty/tty_buffer.c +6 ~~-23~~

index fb042b9..dbe4a71 100644
@@ -151,28 +151,6 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
 }
 
 /**
- *	__tty_buffer_flush		-	flush full tty buffers
- *	@tty: tty to flush
- *
- *	flush all the buffers containing receive data. Caller must
- *	hold the buffer lock and must have ensured no parallel flush to
- *	ldisc is running.
- */
-
-static void __tty_buffer_flush(struct tty_port *port)
-{
-	struct tty_bufhead *buf = &port->buf;
-	struct tty_buffer *next;
-
-	while ((next = buf->head->next) != NULL) {
-		tty_buffer_free(port, buf->head);
-		buf->head = next;
-	}
-	WARN_ON(buf->head != buf->tail);
-	buf->head->read = buf->head->commit;
-}
-
-/**
  *	tty_buffer_flush		-	flush full tty buffers
  *	@tty: tty to flush
  *

@@ -188,11 +166,16 @@ void tty_buffer_flush(struct tty_struct *tty)
 {
 	struct tty_port *port = tty->port;
 	struct tty_bufhead *buf = &port->buf;
+	struct tty_buffer *next;
 
 	buf->flushpending = 1;
 
 	mutex_lock(&buf->flush_mutex);
-	__tty_buffer_flush(port);
+	while ((next = buf->head->next) != NULL) {
+		tty_buffer_free(port, buf->head);
+		buf->head = next;
+	}
+	buf->head->read = buf->head->commit;
 	buf->flushpending = 0;
 	mutex_unlock(&buf->flush_mutex);
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:36 UTC #183 lore raw

PATCH v2 15/16 tty: Fix unsafe vt paste_selection()

In reply to Peter Hurley

Convert the tty_buffer_flush() exclusion mechanism to a public interface - tty_buffer_lock/unlock_exclusive() - and use the interface to safely write the paste selection to the line discipline.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c   | 61 ++++++++++++++++++++++++++++++++++++----------
 drivers/tty/vt/selection.c |  4 ++-
 include/linux/tty.h        |  4 +--
 include/linux/tty_flip.h   |  3 +++
 4 files changed, 56 insertions(+), 16 deletions(-)

drivers/tty/tty_buffer.c +48 ~~-13~~

index dbe4a71..f22e116 100644
@@ -30,6 +30,42 @@
 
 
 /**
+ *	tty_buffer_lock_exclusive	-	gain exclusive access to buffer
+ *	tty_buffer_unlock_exclusive	-	release exclusive access
+ *
+ *	@port - tty_port owning the flip buffer
+ *
+ *	Guarantees safe use of the line discipline's receive_buf() method by
+ *	excluding the buffer work and any pending flush from using the flip
+ *	buffer. Data can continue to be added concurrently to the flip buffer
+ *	from the driver side.
+ *
+ *	On release, the buffer work is restarted if there is data in the
+ *	flip buffer
+ */
+
+void tty_buffer_lock_exclusive(struct tty_port *port)
+{
+	struct tty_bufhead *buf = &port->buf;
+
+	atomic_inc(&buf->priority);
+	mutex_lock(&buf->lock);
+}
+
+void tty_buffer_unlock_exclusive(struct tty_port *port)
+{
+	struct tty_bufhead *buf = &port->buf;
+	int restart;
+
+	restart = buf->head->commit != buf->head->read;
+
+	atomic_dec(&buf->priority);
+	mutex_unlock(&buf->lock);
+	if (restart)
+		queue_work(system_unbound_wq, &buf->work);
+}
+
+/**
  *	tty_buffer_space_avail	-	return unused buffer space
  *	@port - tty_port owning the flip buffer
  *

@@ -158,7 +194,7 @@ static void tty_buffer_free(struct tty_port *port, struct tty_buffer *b)
  *	being processed by flush_to_ldisc then we defer the processing
  *	to that function
  *
- *	Locking: takes flush_mutex to ensure single-threaded flip buffer
+ *	Locking: takes buffer lock to ensure single-threaded flip buffer
  *		 'consumer'
  */

@@ -168,16 +204,16 @@ void tty_buffer_flush(struct tty_struct *tty)
 	struct tty_bufhead *buf = &port->buf;
 	struct tty_buffer *next;
 
-	buf->flushpending = 1;
+	atomic_inc(&buf->priority);
 
-	mutex_lock(&buf->flush_mutex);
+	mutex_lock(&buf->lock);
 	while ((next = buf->head->next) != NULL) {
 		tty_buffer_free(port, buf->head);
 		buf->head = next;
 	}
 	buf->head->read = buf->head->commit;
-	buf->flushpending = 0;
-	mutex_unlock(&buf->flush_mutex);
+	atomic_dec(&buf->priority);
+	mutex_unlock(&buf->lock);
 }
 
 /**

@@ -383,7 +419,7 @@ receive_buf(struct tty_struct *tty, struct tty_buffer *head, int count)
  *
  *	The receive_buf method is single threaded for each tty instance.
  *
- *	Locking: takes flush_mutex to ensure single-threaded flip buffer
+ *	Locking: takes buffer lock to ensure single-threaded flip buffer
  *		 'consumer'
  */

@@ -402,14 +438,14 @@ static void flush_to_ldisc(struct work_struct *work)
 	if (disc == NULL)
 		return;
 
-	mutex_lock(&buf->flush_mutex);
+	mutex_lock(&buf->lock);
 
 	while (1) {
 		struct tty_buffer *head = buf->head;
 		int count;
 
-		/* Ldisc or user is trying to flush the buffers. */
-		if (buf->flushpending)
+		/* Ldisc or user is trying to gain exclusive access */
+		if (atomic_read(&buf->priority))
 			break;
 
 		count = head->commit - head->read;

@@ -426,7 +462,7 @@ static void flush_to_ldisc(struct work_struct *work)
 			break;
 	}
 
-	mutex_unlock(&buf->flush_mutex);
+	mutex_unlock(&buf->lock);
 
 	tty_ldisc_deref(disc);
 }

@@ -482,13 +518,12 @@ void tty_buffer_init(struct tty_port *port)
 {
 	struct tty_bufhead *buf = &port->buf;
 
-	mutex_init(&buf->flush_mutex);
+	mutex_init(&buf->lock);
 	tty_buffer_reset(&buf->sentinel, 0);
 	buf->head = &buf->sentinel;
 	buf->tail = &buf->sentinel;
 	init_llist_head(&buf->free);
 	atomic_set(&buf->memory_used, 0);
-	buf->flushpending = 0;
+	atomic_set(&buf->priority, 0);
 	INIT_WORK(&buf->work, flush_to_ldisc);
 }
-

drivers/tty/vt/selection.c +3 -1

index 2ca8d6b..ea27804 100644
@@ -24,6 +24,7 @@
 #include <linux/selection.h>
 #include <linux/tiocl.h>
 #include <linux/console.h>
+#include <linux/tty_flip.h>
 
 /* Don't take this from <ctype.h>: 011-015 on the screen aren't spaces */
 #define isspace(c)	((c) == ' ')

@@ -346,8 +347,8 @@ int paste_selection(struct tty_struct *tty)
 	console_unlock();
 
 	ld = tty_ldisc_ref_wait(tty);
+	tty_buffer_lock_exclusive(&vc->port);
 
-	/* FIXME: this is completely unsafe */
 	add_wait_queue(&vc->paste_wait, &wait);
 	while (sel_buffer && sel_buffer_lth > pasted) {
 		set_current_state(TASK_INTERRUPTIBLE);

@@ -363,6 +364,7 @@ int paste_selection(struct tty_struct *tty)
 	remove_wait_queue(&vc->paste_wait, &wait);
 	__set_current_state(TASK_RUNNING);
 
+	tty_buffer_unlock_exclusive(&vc->port);
 	tty_ldisc_deref(ld);
 	return 0;
 }

include/linux/tty.h +2 -2

index 991575f..7a9a3b0 100644
@@ -67,8 +67,8 @@ static inline char *flag_buf_ptr(struct tty_buffer *b, int ofs)
 struct tty_bufhead {
 	struct tty_buffer *head;	/* Queue head */
 	struct work_struct work;
-	struct mutex flush_mutex;
-	unsigned int flushpending:1;
+	struct mutex	   lock;
+	atomic_t	   priority;
 	struct tty_buffer sentinel;
 	struct llist_head free;		/* Free queue head */
 	atomic_t	   memory_used; /* In-use buffers excluding free list */

include/linux/tty_flip.h +3 -0

index 6944ed2..21ddd7d 100644
@@ -32,4 +32,7 @@ static inline int tty_insert_flip_string(struct tty_port *port,
 	return tty_insert_flip_string_fixed_flag(port, chars, TTY_NORMAL, size);
 }
 
+extern void tty_buffer_lock_exclusive(struct tty_port *port);
+extern void tty_buffer_unlock_exclusive(struct tty_port *port);
+
 #endif /* _LINUX_TTY_FLIP_H */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 13:57 UTC #184 lore raw

PATCH v2 16/16 tty: Remove private constant from global namespace

In reply to Peter Hurley

TTY_BUFFER_PAGE is only used within drivers/tty/tty_buffer.c; relocate to that file scope.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/tty_buffer.c | 10 ++++++++++
 include/linux/tty.h      | 11 -----------
 2 files changed, 10 insertions(+), 11 deletions(-)

drivers/tty/tty_buffer.c +10 -0

index f22e116..c043136f 100644
@@ -28,6 +28,16 @@
  */
 #define TTYB_MEM_LIMIT	65536
 
+/*
+ * We default to dicing tty buffer allocations to this many characters
+ * in order to avoid multiple page allocations. We know the size of
+ * tty_buffer itself but it must also be taken into account that the
+ * the buffer is 256 byte aligned. See tty_buffer_find for the allocation
+ * logic this must match
+ */
+
+#define TTY_BUFFER_PAGE	(((PAGE_SIZE - sizeof(struct tty_buffer)) / 2) & ~0xFF)
+
 
 /**
  *	tty_buffer_lock_exclusive	-	gain exclusive access to buffer

include/linux/tty.h +0 ~~-11~~

index 7a9a3b0..5fd5d6f 100644
@@ -53,17 +53,6 @@ static inline char *flag_buf_ptr(struct tty_buffer *b, int ofs)
 	return (char *)char_buf_ptr(b, ofs) + b->size;
 }
 
-/*
- * We default to dicing tty buffer allocations to this many characters
- * in order to avoid multiple page allocations. We know the size of
- * tty_buffer itself but it must also be taken into account that the
- * the buffer is 256 byte aligned. See tty_buffer_find for the allocation
- * logic this must match
- */
-
-#define TTY_BUFFER_PAGE	(((PAGE_SIZE - sizeof(struct tty_buffer)) / 2) & ~0xFF)
-
-
 struct tty_bufhead {
 	struct tty_buffer *head;	/* Queue head */
 	struct work_struct work;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #185 lore raw

v2 cover mostly lockless tty echo

In reply to Peter Hurley

** v2 changes **
- Rebased on v2 of 'lockless tty flip buffers'

This 3rd of 4 patchsets implements a mostly-lockless tty echo output.

Because echoing is performed solely by the single-threaded ldisc receive_buf() method, most of the lockless requirements are already in-place. The main existing complications were;
1) Echoing could overrun itself. A fixed-size buffer is used to record the operations necessary when outputting the echoes; the most recent echo data is preserved.
2) An attempt to push unprocessed echoes is made by the n_tty_write method (on a different thread) before attempting write output.

The overrun condition is solved by outputting the echoes in blocks and ensuring that at least that much space is available each time an echo operation is committed. At the conclusion of each flip buffer received, any remaining unprocessed echoes are output.

This block output method is particularly effective when there is no reader (the tty is output-only) and termios is misconfigured with echo enabled.

The concurrent access by the n_tty_write() method is already excluded by the existing output_lock mutex.

Peter Hurley (9):
n_tty: Remove unused echo_overrun field
n_tty: Use separate head and tail indices for echo_buf
n_tty: Replace echo_cnt with computed value
n_tty: Remove echo_lock
n_tty: Eliminate echo_commit memory barrier
n_tty: Process echoes in blocks
n_tty: Only flush echo output if actually output
n_tty: Eliminate counter in __process_echoes
n_tty: Avoid false-sharing echo buffer indices

 drivers/tty/n_tty.c | 248 ++++++++++++++++++++++++----------------------------
 1 file changed, 113 insertions(+), 135 deletions(-)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #186 lore raw

PATCH v2 1/9 n_tty: Remove unused echo_overrun field

The echo_overrun field is only assigned and never tested; remove it.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

drivers/tty/n_tty.c +1 -7

index 2321d6a..160d9d2 100644
@@ -96,7 +96,6 @@ struct n_tty_data {
 
 	/* must hold exclusive termios_rwsem to reset these */
 	unsigned char lnext:1, erasing:1, raw:1, real_raw:1, icanon:1;
-	unsigned char echo_overrun:1;
 
 	/* shared by producer and consumer */
 	char *read_buf;

@@ -325,7 +324,7 @@ static void reset_buffer_flags(struct n_tty_data *ldata)
 	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
 
 	mutex_lock(&ldata->echo_lock);
-	ldata->echo_pos = ldata->echo_cnt = ldata->echo_overrun = 0;
+	ldata->echo_pos = ldata->echo_cnt = 0;
 	mutex_unlock(&ldata->echo_lock);
 
 	ldata->erasing = 0;

@@ -781,14 +780,11 @@ static void process_echoes(struct tty_struct *tty)
 	if (nr == 0) {
 		ldata->echo_pos = 0;
 		ldata->echo_cnt = 0;
-		ldata->echo_overrun = 0;
 	} else {
 		int num_processed = ldata->echo_cnt - nr;
 		ldata->echo_pos += num_processed;
 		ldata->echo_pos &= N_TTY_BUF_SIZE - 1;
 		ldata->echo_cnt = nr;
-		if (num_processed > 0)
-			ldata->echo_overrun = 0;
 	}
 
 	mutex_unlock(&ldata->echo_lock);

@@ -834,8 +830,6 @@ static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
 			ldata->echo_pos++;
 		}
 		ldata->echo_pos &= N_TTY_BUF_SIZE - 1;
-
-		ldata->echo_overrun = 1;
 	} else {
 		new_byte_pos = ldata->echo_pos + ldata->echo_cnt;
 		new_byte_pos &= N_TTY_BUF_SIZE - 1;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #187 lore raw

PATCH v2 2/9 n_tty: Use separate head and tail indices for echo_buf

In reply to Peter Hurley

Instead of using a single index to track the current echo_buf position, use a head index when adding to the buffer and a tail index when consuming from the buffer. Allow these head and tail indices to wrap at max representable value; perform modulo reduction via helper functions when accessing the buffer.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 88 +++++++++++++++++++++--------------------------------
 1 file changed, 35 insertions(+), 53 deletions(-)

drivers/tty/n_tty.c +35 ~~-53~~

index 160d9d2..f90e171 100644
@@ -108,7 +108,8 @@ struct n_tty_data {
 
 	/* protected by echo_lock */
 	unsigned char *echo_buf;
-	unsigned int echo_pos;
+	size_t echo_head;
+	size_t echo_tail;
 	unsigned int echo_cnt;
 
 	/* protected by output lock */

@@ -135,6 +136,16 @@ static inline unsigned char *read_buf_addr(struct n_tty_data *ldata, size_t i)
 	return &ldata->read_buf[i & (N_TTY_BUF_SIZE - 1)];
 }
 
+static inline unsigned char echo_buf(struct n_tty_data *ldata, size_t i)
+{
+	return ldata->echo_buf[i & (N_TTY_BUF_SIZE - 1)];
+}
+
+static inline unsigned char *echo_buf_addr(struct n_tty_data *ldata, size_t i)
+{
+	return &ldata->echo_buf[i & (N_TTY_BUF_SIZE - 1)];
+}
+
 static inline int tty_put_user(struct tty_struct *tty, unsigned char x,
 			       unsigned char __user *ptr)
 {

@@ -324,7 +335,7 @@ static void reset_buffer_flags(struct n_tty_data *ldata)
 	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
 
 	mutex_lock(&ldata->echo_lock);
-	ldata->echo_pos = ldata->echo_cnt = 0;
+	ldata->echo_head = ldata->echo_tail = ldata->echo_cnt = 0;
 	mutex_unlock(&ldata->echo_lock);
 
 	ldata->erasing = 0;

@@ -639,8 +650,8 @@ static void process_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int	space, nr;
+	size_t tail;
 	unsigned char c;
-	unsigned char *cp, *buf_end;
 
 	if (!ldata->echo_cnt)
 		return;

@@ -650,14 +661,12 @@ static void process_echoes(struct tty_struct *tty)
 
 	space = tty_write_room(tty);
 
-	buf_end = ldata->echo_buf + N_TTY_BUF_SIZE;
-	cp = ldata->echo_buf + ldata->echo_pos;
+	tail = ldata->echo_tail;
 	nr = ldata->echo_cnt;
 	while (nr > 0) {
-		c = *cp;
+		c = echo_buf(ldata, tail);
 		if (c == ECHO_OP_START) {
 			unsigned char op;
-			unsigned char *opp;
 			int no_space_left = 0;
 
 			/*

@@ -665,18 +674,13 @@ static void process_echoes(struct tty_struct *tty)
 			 * operation, get the next byte, which is either the
 			 * op code or a control character value.
 			 */
-			opp = cp + 1;
-			if (opp == buf_end)
-				opp -= N_TTY_BUF_SIZE;
-			op = *opp;
+			op = echo_buf(ldata, tail + 1);
 
 			switch (op) {
 				unsigned int num_chars, num_bs;
 
 			case ECHO_OP_ERASE_TAB:
-				if (++opp == buf_end)
-					opp -= N_TTY_BUF_SIZE;
-				num_chars = *opp;
+				num_chars = echo_buf(ldata, tail + 2);
 
 				/*
 				 * Determine how many columns to go back

@@ -702,20 +706,20 @@ static void process_echoes(struct tty_struct *tty)
 					if (ldata->column > 0)
 						ldata->column--;
 				}
-				cp += 3;
+				tail += 3;
 				nr -= 3;
 				break;
 
 			case ECHO_OP_SET_CANON_COL:
 				ldata->canon_column = ldata->column;
-				cp += 2;
+				tail += 2;
 				nr -= 2;
 				break;
 
 			case ECHO_OP_MOVE_BACK_COL:
 				if (ldata->column > 0)
 					ldata->column--;
-				cp += 2;
+				tail += 2;
 				nr -= 2;
 				break;

@@ -728,7 +732,7 @@ static void process_echoes(struct tty_struct *tty)
 				tty_put_char(tty, ECHO_OP_START);
 				ldata->column++;
 				space--;
-				cp += 2;
+				tail += 2;
 				nr -= 2;
 				break;

@@ -750,7 +754,7 @@ static void process_echoes(struct tty_struct *tty)
 				tty_put_char(tty, op ^ 0100);
 				ldata->column += 2;
 				space -= 2;
-				cp += 2;
+				tail += 2;
 				nr -= 2;
 			}

@@ -768,24 +772,13 @@ static void process_echoes(struct tty_struct *tty)
 				tty_put_char(tty, c);
 				space -= 1;
 			}
-			cp += 1;
+			tail += 1;
 			nr -= 1;
 		}
-
-		/* When end of circular buffer reached, wrap around */
-		if (cp >= buf_end)
-			cp -= N_TTY_BUF_SIZE;
 	}
 
-	if (nr == 0) {
-		ldata->echo_pos = 0;
-		ldata->echo_cnt = 0;
-	} else {
-		int num_processed = ldata->echo_cnt - nr;
-		ldata->echo_pos += num_processed;
-		ldata->echo_pos &= N_TTY_BUF_SIZE - 1;
-		ldata->echo_cnt = nr;
-	}
+	ldata->echo_tail = tail;
+	ldata->echo_cnt = nr;
 
 	mutex_unlock(&ldata->echo_lock);
 	mutex_unlock(&ldata->output_lock);

@@ -806,37 +799,26 @@ static void process_echoes(struct tty_struct *tty)
 
 static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
 {
-	int	new_byte_pos;
-
 	if (ldata->echo_cnt == N_TTY_BUF_SIZE) {
-		/* Circular buffer is already at capacity */
-		new_byte_pos = ldata->echo_pos;
-
+		size_t head = ldata->echo_head;
 		/*
 		 * Since the buffer start position needs to be advanced,
 		 * be sure to step by a whole operation byte group.
 		 */
-		if (ldata->echo_buf[ldata->echo_pos] == ECHO_OP_START) {
-			if (ldata->echo_buf[(ldata->echo_pos + 1) &
-					  (N_TTY_BUF_SIZE - 1)] ==
-						ECHO_OP_ERASE_TAB) {
-				ldata->echo_pos += 3;
+		if (echo_buf(ldata, head) == ECHO_OP_START) {
+			if (echo_buf(ldata, head + 1) == ECHO_OP_ERASE_TAB) {
+				ldata->echo_tail += 3;
 				ldata->echo_cnt -= 2;
 			} else {
-				ldata->echo_pos += 2;
+				ldata->echo_tail += 2;
 				ldata->echo_cnt -= 1;
 			}
-		} else {
-			ldata->echo_pos++;
-		}
-		ldata->echo_pos &= N_TTY_BUF_SIZE - 1;
-	} else {
-		new_byte_pos = ldata->echo_pos + ldata->echo_cnt;
-		new_byte_pos &= N_TTY_BUF_SIZE - 1;
+		} else
+			ldata->echo_tail++;
+	} else
 		ldata->echo_cnt++;
-	}
 
-	ldata->echo_buf[new_byte_pos] = c;
+	*echo_buf_addr(ldata, ldata->echo_head++) = c;
 }
 
 /**

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #188 lore raw

PATCH v2 3/9 n_tty: Replace echo_cnt with computed value

In reply to Peter Hurley

Prepare for lockless echo_buf handling; compute current byte count of echo_buf from head and tail indices.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

drivers/tty/n_tty.c +7 ~~-13~~

index f90e171..68dc9fd 100644
@@ -110,7 +110,6 @@ struct n_tty_data {
 	unsigned char *echo_buf;
 	size_t echo_head;
 	size_t echo_tail;
-	unsigned int echo_cnt;
 
 	/* protected by output lock */
 	unsigned int column;

@@ -335,7 +334,7 @@ static void reset_buffer_flags(struct n_tty_data *ldata)
 	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
 
 	mutex_lock(&ldata->echo_lock);
-	ldata->echo_head = ldata->echo_tail = ldata->echo_cnt = 0;
+	ldata->echo_head = ldata->echo_tail  = 0;
 	mutex_unlock(&ldata->echo_lock);
 
 	ldata->erasing = 0;

@@ -653,7 +652,7 @@ static void process_echoes(struct tty_struct *tty)
 	size_t tail;
 	unsigned char c;
 
-	if (!ldata->echo_cnt)
+	if (ldata->echo_head == ldata->echo_tail)
 		return;
 
 	mutex_lock(&ldata->output_lock);

@@ -662,7 +661,7 @@ static void process_echoes(struct tty_struct *tty)
 	space = tty_write_room(tty);
 
 	tail = ldata->echo_tail;
-	nr = ldata->echo_cnt;
+	nr = ldata->echo_head - ldata->echo_tail;
 	while (nr > 0) {
 		c = echo_buf(ldata, tail);
 		if (c == ECHO_OP_START) {

@@ -778,7 +777,6 @@ static void process_echoes(struct tty_struct *tty)
 	}
 
 	ldata->echo_tail = tail;
-	ldata->echo_cnt = nr;
 
 	mutex_unlock(&ldata->echo_lock);
 	mutex_unlock(&ldata->output_lock);

@@ -799,24 +797,20 @@ static void process_echoes(struct tty_struct *tty)
 
 static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
 {
-	if (ldata->echo_cnt == N_TTY_BUF_SIZE) {
+	if (ldata->echo_head - ldata->echo_tail == N_TTY_BUF_SIZE) {
 		size_t head = ldata->echo_head;
 		/*
 		 * Since the buffer start position needs to be advanced,
 		 * be sure to step by a whole operation byte group.
 		 */
 		if (echo_buf(ldata, head) == ECHO_OP_START) {
-			if (echo_buf(ldata, head + 1) == ECHO_OP_ERASE_TAB) {
+			if (echo_buf(ldata, head + 1) == ECHO_OP_ERASE_TAB)
 				ldata->echo_tail += 3;
-				ldata->echo_cnt -= 2;
-			} else {
+			else
 				ldata->echo_tail += 2;
-				ldata->echo_cnt -= 1;
-			}
 		} else
 			ldata->echo_tail++;
-	} else
-		ldata->echo_cnt++;
+	}
 
 	*echo_buf_addr(ldata, ldata->echo_head++) = c;
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #189 lore raw

PATCH v2 4/9 n_tty: Remove echo_lock

In reply to Peter Hurley

Adding data to echo_buf (via add_echo_byte()) is guaranteed to be single-threaded, since all callers are from the n_tty_receive_buf() path. Processing the echo_buf can be called from either the n_tty_receive_buf() path or the n_tty_write() path; however, these callers are already serialized by output_lock.

Publish cumulative echo_head changes to echo_commit; process echo_buf from echo_tail to echo_commit; remove echo_lock.

On echo_buf overrun, claim output_lock to serialize changes to echo_tail.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 75 ++++++++++++++++++++---------------------------------
 1 file changed, 28 insertions(+), 47 deletions(-)

drivers/tty/n_tty.c +28 ~~-47~~

index 68dc9fd..d22c14c 100644
@@ -106,10 +106,10 @@ struct n_tty_data {
 	/* consumer-published */
 	size_t read_tail;
 
-	/* protected by echo_lock */
 	unsigned char *echo_buf;
 	size_t echo_head;
 	size_t echo_tail;
+	size_t echo_commit;
 
 	/* protected by output lock */
 	unsigned int column;

@@ -117,7 +117,6 @@ struct n_tty_data {
 
 	struct mutex atomic_read_lock;
 	struct mutex output_lock;
-	struct mutex echo_lock;
 };
 
 static inline size_t read_cnt(struct n_tty_data *ldata)

@@ -332,10 +331,7 @@ static void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
 static void reset_buffer_flags(struct n_tty_data *ldata)
 {
 	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
-
-	mutex_lock(&ldata->echo_lock);
-	ldata->echo_head = ldata->echo_tail  = 0;
-	mutex_unlock(&ldata->echo_lock);
+	ldata->echo_head = ldata->echo_tail = ldata->echo_commit = 0;
 
 	ldata->erasing = 0;
 	bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);

@@ -641,8 +637,7 @@ break_out:
  *	are prioritized.  Also, when control characters are echoed with a
  *	prefixed "^", the pair is treated atomically and thus not separated.
  *
- *	Locking: output_lock to protect column state and space left,
- *		 echo_lock to protect the echo buffer
+ *	Locking: output_lock to protect column state and space left
  */
 
 static void process_echoes(struct tty_struct *tty)

@@ -652,16 +647,15 @@ static void process_echoes(struct tty_struct *tty)
 	size_t tail;
 	unsigned char c;
 
-	if (ldata->echo_head == ldata->echo_tail)
+	if (ldata->echo_commit == ldata->echo_tail)
 		return;
 
 	mutex_lock(&ldata->output_lock);
-	mutex_lock(&ldata->echo_lock);
 
 	space = tty_write_room(tty);
 
 	tail = ldata->echo_tail;
-	nr = ldata->echo_head - ldata->echo_tail;
+	nr = ldata->echo_commit - ldata->echo_tail;
 	while (nr > 0) {
 		c = echo_buf(ldata, tail);
 		if (c == ECHO_OP_START) {

@@ -778,13 +772,21 @@ static void process_echoes(struct tty_struct *tty)
 
 	ldata->echo_tail = tail;
 
-	mutex_unlock(&ldata->echo_lock);
 	mutex_unlock(&ldata->output_lock);
 
 	if (tty->ops->flush_chars)
 		tty->ops->flush_chars(tty);
 }
 
+static void commit_echoes(struct tty_struct *tty)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+
+	smp_mb();
+	ldata->echo_commit = ldata->echo_head;
+	process_echoes(tty);
+}
+
 /**
  *	add_echo_byte	-	add a byte to the echo buffer
  *	@c: unicode byte to echo

@@ -792,13 +794,16 @@ static void process_echoes(struct tty_struct *tty)
  *
  *	Add a character or operation byte to the echo buffer.
  *
- *	Should be called under the echo lock to protect the echo buffer.
+ *	Locks: may claim output_lock to prevent concurrent modify of
+ *	       echo_tail by process_echoes().
  */
 
 static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
 {
 	if (ldata->echo_head - ldata->echo_tail == N_TTY_BUF_SIZE) {
 		size_t head = ldata->echo_head;
+
+		mutex_lock(&ldata->output_lock);
 		/*
 		 * Since the buffer start position needs to be advanced,
 		 * be sure to step by a whole operation byte group.

@@ -810,6 +815,7 @@ static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
 				ldata->echo_tail += 2;
 		} else
 			ldata->echo_tail++;
+		mutex_unlock(&ldata->output_lock);
 	}
 
 	*echo_buf_addr(ldata, ldata->echo_head++) = c;

@@ -820,16 +826,12 @@ static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
  *	@ldata: n_tty data
  *
  *	Add an operation to the echo buffer to move back one column.
- *
- *	Locking: echo_lock to protect the echo buffer
  */
 
 static void echo_move_back_col(struct n_tty_data *ldata)
 {
-	mutex_lock(&ldata->echo_lock);
 	add_echo_byte(ECHO_OP_START, ldata);
 	add_echo_byte(ECHO_OP_MOVE_BACK_COL, ldata);
-	mutex_unlock(&ldata->echo_lock);
 }
 
 /**

@@ -838,16 +840,12 @@ static void echo_move_back_col(struct n_tty_data *ldata)
  *
  *	Add an operation to the echo buffer to set the canon column
  *	to the current column.
- *
- *	Locking: echo_lock to protect the echo buffer
  */
 
 static void echo_set_canon_col(struct n_tty_data *ldata)
 {
-	mutex_lock(&ldata->echo_lock);
 	add_echo_byte(ECHO_OP_START, ldata);
 	add_echo_byte(ECHO_OP_SET_CANON_COL, ldata);
-	mutex_unlock(&ldata->echo_lock);
 }
 
 /**

@@ -863,15 +861,11 @@ static void echo_set_canon_col(struct n_tty_data *ldata)
  *	of input.  This information will be used later, along with
  *	canon column (if applicable), to go back the correct number
  *	of columns.
- *
- *	Locking: echo_lock to protect the echo buffer
  */
 
 static void echo_erase_tab(unsigned int num_chars, int after_tab,
 			   struct n_tty_data *ldata)
 {
-	mutex_lock(&ldata->echo_lock);
-
 	add_echo_byte(ECHO_OP_START, ldata);
 	add_echo_byte(ECHO_OP_ERASE_TAB, ldata);

@@ -883,8 +877,6 @@ static void echo_erase_tab(unsigned int num_chars, int after_tab,
 		num_chars |= 0x80;
 
 	add_echo_byte(num_chars, ldata);
-
-	mutex_unlock(&ldata->echo_lock);
 }
 
 /**

@@ -896,20 +888,16 @@ static void echo_erase_tab(unsigned int num_chars, int after_tab,
  *	L_ECHO(tty) is true. Called from the driver receive_buf path.
  *
  *	This variant does not treat control characters specially.
- *
- *	Locking: echo_lock to protect the echo buffer
  */
 
 static void echo_char_raw(unsigned char c, struct n_tty_data *ldata)
 {
-	mutex_lock(&ldata->echo_lock);
 	if (c == ECHO_OP_START) {
 		add_echo_byte(ECHO_OP_START, ldata);
 		add_echo_byte(ECHO_OP_START, ldata);
 	} else {
 		add_echo_byte(c, ldata);
 	}
-	mutex_unlock(&ldata->echo_lock);
 }
 
 /**

@@ -922,16 +910,12 @@ static void echo_char_raw(unsigned char c, struct n_tty_data *ldata)
  *
  *	This variant tags control characters to be echoed as "^X"
  *	(where X is the letter representing the control char).
- *
- *	Locking: echo_lock to protect the echo buffer
  */
 
 static void echo_char(unsigned char c, struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 
-	mutex_lock(&ldata->echo_lock);
-
 	if (c == ECHO_OP_START) {
 		add_echo_byte(ECHO_OP_START, ldata);
 		add_echo_byte(ECHO_OP_START, ldata);

@@ -940,8 +924,6 @@ static void echo_char(unsigned char c, struct tty_struct *tty)
 			add_echo_byte(ECHO_OP_START, ldata);
 		add_echo_byte(c, ldata);
 	}
-
-	mutex_unlock(&ldata->echo_lock);
 }
 
 /**

@@ -1283,7 +1265,7 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 			if (ldata->canon_head == ldata->read_head)
 				echo_set_canon_col(ldata);
 			echo_char(c, tty);
-			process_echoes(tty);
+			commit_echoes(tty);
 		}
 		if (parmrk)
 			put_tty_queue(c, ldata);

@@ -1294,7 +1276,7 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	if (I_IXON(tty)) {
 		if (c == START_CHAR(tty)) {
 			start_tty(tty);
-			process_echoes(tty);
+			commit_echoes(tty);
 			return;
 		}
 		if (c == STOP_CHAR(tty)) {

@@ -1325,7 +1307,7 @@ send_signal:
 				start_tty(tty);
 			if (L_ECHO(tty)) {
 				echo_char(c, tty);
-				process_echoes(tty);
+				commit_echoes(tty);
 			}
 			isig(signal, tty);
 			return;

@@ -1344,7 +1326,7 @@ send_signal:
 		if (c == ERASE_CHAR(tty) || c == KILL_CHAR(tty) ||
 		    (c == WERASE_CHAR(tty) && L_IEXTEN(tty))) {
 			eraser(c, tty);
-			process_echoes(tty);
+			commit_echoes(tty);
 			return;
 		}
 		if (c == LNEXT_CHAR(tty) && L_IEXTEN(tty)) {

@@ -1354,7 +1336,7 @@ send_signal:
 				if (L_ECHOCTL(tty)) {
 					echo_char_raw('^', ldata);
 					echo_char_raw('\b', ldata);
-					process_echoes(tty);
+					commit_echoes(tty);
 				}
 			}
 			return;

@@ -1370,7 +1352,7 @@ send_signal:
 				echo_char(read_buf(ldata, tail), tty);
 				tail++;
 			}
-			process_echoes(tty);
+			commit_echoes(tty);
 			return;
 		}
 		if (c == '\n') {

@@ -1381,7 +1363,7 @@ send_signal:
 			}
 			if (L_ECHO(tty) || L_ECHONL(tty)) {
 				echo_char_raw('\n', ldata);
-				process_echoes(tty);
+				commit_echoes(tty);
 			}
 			goto handle_newline;
 		}

@@ -1410,7 +1392,7 @@ send_signal:
 				if (ldata->canon_head == ldata->read_head)
 					echo_set_canon_col(ldata);
 				echo_char(c, tty);
-				process_echoes(tty);
+				commit_echoes(tty);
 			}
 			/*
 			 * XXX does PARMRK doubling happen for

@@ -1447,7 +1429,7 @@ handle_newline:
 				echo_set_canon_col(ldata);
 			echo_char(c, tty);
 		}
-		process_echoes(tty);
+		commit_echoes(tty);
 	}
 
 	if (parmrk)

@@ -1712,7 +1694,6 @@ static int n_tty_open(struct tty_struct *tty)
 	ldata->overrun_time = jiffies;
 	mutex_init(&ldata->atomic_read_lock);
 	mutex_init(&ldata->output_lock);
-	mutex_init(&ldata->echo_lock);
 
 	/* These are ugly. Currently a malloc failure here can panic */
 	ldata->read_buf = kzalloc(N_TTY_BUF_SIZE, GFP_KERNEL);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #190 lore raw

PATCH v2 5/9 n_tty: Eliminate echo_commit memory barrier

In reply to Peter Hurley

Use output_lock mutex as a memory barrier when storing echo_commit.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 31 ++++++++++++++++++++-----------
 1 file changed, 20 insertions(+), 11 deletions(-)

drivers/tty/n_tty.c +20 ~~-11~~

index d22c14c..c595264 100644
@@ -637,21 +637,16 @@ break_out:
  *	are prioritized.  Also, when control characters are echoed with a
  *	prefixed "^", the pair is treated atomically and thus not separated.
  *
- *	Locking: output_lock to protect column state and space left
+ *	Locking: callers must hold output_lock
  */
 
-static void process_echoes(struct tty_struct *tty)
+static void __process_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int	space, nr;
 	size_t tail;
 	unsigned char c;
 
-	if (ldata->echo_commit == ldata->echo_tail)
-		return;
-
-	mutex_lock(&ldata->output_lock);
-
 	space = tty_write_room(tty);
 
 	tail = ldata->echo_tail;

@@ -771,20 +766,34 @@ static void process_echoes(struct tty_struct *tty)
 	}
 
 	ldata->echo_tail = tail;
+}
+
+static void commit_echoes(struct tty_struct *tty)
+{
+	struct n_tty_data *ldata = tty->disc_data;
 
+	mutex_lock(&ldata->output_lock);
+	ldata->echo_commit = ldata->echo_head;
+	__process_echoes(tty);
 	mutex_unlock(&ldata->output_lock);
 
 	if (tty->ops->flush_chars)
 		tty->ops->flush_chars(tty);
 }
 
-static void commit_echoes(struct tty_struct *tty)
+static void process_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 
-	smp_mb();
-	ldata->echo_commit = ldata->echo_head;
-	process_echoes(tty);
+	if (!L_ECHO(tty) || ldata->echo_commit == ldata->echo_tail)
+		return;
+
+	mutex_lock(&ldata->output_lock);
+	__process_echoes(tty);
+	mutex_unlock(&ldata->output_lock);
+
+	if (tty->ops->flush_chars)
+		tty->ops->flush_chars(tty);
 }
 
 /**

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #191 lore raw

PATCH v2 6/9 n_tty: Process echoes in blocks

In reply to Peter Hurley

Byte-by-byte echo output is painfully slow, requiring a lock/unlock cycle for every input byte.

Instead, perform the echo output in blocks of 256 characters, and at least once per flip buffer receive. Enough space is reserved in the echo buffer to guarantee a full block can be saved without overrunning the echo output. Overrun is prevented by discarding the oldest echoes until enough space exists in the echo buffer to receive at least a full block of new echoes.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 70 +++++++++++++++++++++++++++++++++++------------------
 1 file changed, 47 insertions(+), 23 deletions(-)

drivers/tty/n_tty.c +47 ~~-23~~

index c595264..f298890 100644
@@ -74,6 +74,11 @@
 #define ECHO_OP_SET_CANON_COL 0x81
 #define ECHO_OP_ERASE_TAB 0x82
 
+#define ECHO_COMMIT_WATERMARK	256
+#define ECHO_BLOCK		256
+#define ECHO_DISCARD_WATERMARK	N_TTY_BUF_SIZE - (ECHO_BLOCK + 32)
+
+
 #undef N_TTY_TRACE
 #ifdef N_TTY_TRACE
 # define n_tty_trace(f, args...)	trace_printk(f, ##args)

@@ -765,15 +770,40 @@ static void __process_echoes(struct tty_struct *tty)
 		}
 	}
 
+	/* If the echo buffer is nearly full (so that the possibility exists
+	 * of echo overrun before the next commit), then discard enough
+	 * data at the tail to prevent a subsequent overrun */
+	while (ldata->echo_commit - tail >= ECHO_DISCARD_WATERMARK) {
+		if (echo_buf(ldata, tail == ECHO_OP_START)) {
+			if (echo_buf(ldata, tail) == ECHO_OP_ERASE_TAB)
+				tail += 3;
+			else
+				tail += 2;
+		} else
+			tail++;
+	}
+
 	ldata->echo_tail = tail;
 }
 
 static void commit_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
+	size_t nr, old;
+	size_t head;
+
+	head = ldata->echo_head;
+	old = ldata->echo_commit - ldata->echo_tail;
+
+	/* Process committed echoes if the accumulated # of bytes
+	 * is over the threshold (and try again each time another
+	 * block is accumulated) */
+	nr = head - ldata->echo_tail;
+	if (nr < ECHO_COMMIT_WATERMARK || (nr % ECHO_BLOCK > old % ECHO_BLOCK))
+		return;
 
 	mutex_lock(&ldata->output_lock);
-	ldata->echo_commit = ldata->echo_head;
+	ldata->echo_commit = head;
 	__process_echoes(tty);
 	mutex_unlock(&ldata->output_lock);

@@ -796,37 +826,29 @@ static void process_echoes(struct tty_struct *tty)
 		tty->ops->flush_chars(tty);
 }
 
+static void flush_echoes(struct tty_struct *tty)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+
+	if (!L_ECHO(tty) || ldata->echo_commit == ldata->echo_head)
+		return;
+
+	mutex_lock(&ldata->output_lock);
+	ldata->echo_commit = ldata->echo_head;
+	__process_echoes(tty);
+	mutex_unlock(&ldata->output_lock);
+}
+
 /**
  *	add_echo_byte	-	add a byte to the echo buffer
  *	@c: unicode byte to echo
  *	@ldata: n_tty data
  *
  *	Add a character or operation byte to the echo buffer.
- *
- *	Locks: may claim output_lock to prevent concurrent modify of
- *	       echo_tail by process_echoes().
  */
 
-static void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
+static inline void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
 {
-	if (ldata->echo_head - ldata->echo_tail == N_TTY_BUF_SIZE) {
-		size_t head = ldata->echo_head;
-
-		mutex_lock(&ldata->output_lock);
-		/*
-		 * Since the buffer start position needs to be advanced,
-		 * be sure to step by a whole operation byte group.
-		 */
-		if (echo_buf(ldata, head) == ECHO_OP_START) {
-			if (echo_buf(ldata, head + 1) == ECHO_OP_ERASE_TAB)
-				ldata->echo_tail += 3;
-			else
-				ldata->echo_tail += 2;
-		} else
-			ldata->echo_tail++;
-		mutex_unlock(&ldata->output_lock);
-	}
-
 	*echo_buf_addr(ldata, ldata->echo_head++) = c;
 }

@@ -1514,6 +1536,8 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 				break;
 			}
 		}
+
+		flush_echoes(tty);
 		if (tty->ops->flush_chars)
 			tty->ops->flush_chars(tty);
 	}

Signature · 1 line

1.8.1.2

Greg Kroah-Hartman

maintainer

2013-07-23 23:53 UTC #192 lore raw

Re: [PATCH v2 6/9] n_tty: Process echoes in blocks

Quoted 9 lines · Peter Hurley

Byte-by-byte echo output is painfully slow, requiring a lock/unlock cycle for every input byte.
Instead, perform the echo output in blocks of 256 characters, and at least once per flip buffer receive. Enough space is reserved in the echo buffer to guarantee a full block can be saved without overrunning the echo output. Overrun is prevented by discarding the oldest echoes until enough space exists in the echo buffer to receive at least a full block of new echoes.

I'm a bit worried about this, I wonder if anything is expecting the echos to not come in "bursts" like this, but I really can't think of why they would want that.

So let's apply it and see what breaks! :)

thanks,

greg k-h

Peter Hurley

author

2013-07-25 01:33 UTC #193 lore raw

Re: [PATCH v2 6/9] n_tty: Process echoes in blocks

Quoted 11 lines · Greg Kroah-Hartman

Quoted 9 lines · Peter Hurley
Byte-by-byte echo output is painfully slow, requiring a lock/unlock cycle for every input byte.
Instead, perform the echo output in blocks of 256 characters, and at least once per flip buffer receive. Enough space is reserved in the echo buffer to guarantee a full block can be saved without overrunning the echo output. Overrun is prevented by discarding the oldest echoes until enough space exists in the echo buffer to receive at least a full block of new echoes.

Quoted 3 lines

I'm a bit worried about this, I wonder if anything is expecting the echos to not come in "bursts" like this, but I really can't think of why they would want that.

The block size for echoes is trivially settable so if you'd prefer a smaller value than every 256 chars, I could send a patch for a value you are more comfortable with - 64 chars, 32 chars, whatever.

I picked that value from casual performance testing and that it coincides with the smallest possible flip buffer; ie., echo blocks are retired at the same rate as flip buffers when receiving is not saturated.

Quoted 1 line

So let's apply it and see what breaks! :)

I'm ready :)

Regards,
Peter Hurley

Peter Hurley

author

2013-06-15 14:04 UTC #194 lore raw

PATCH v2 7/9 n_tty: Only flush echo output if actually output

In reply to Peter Hurley

Don't have the driver flush received echoes if no echoes were actually output.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

drivers/tty/n_tty.c +10 -8

index f298890..4a2da9e 100644
@@ -645,14 +645,14 @@ break_out:
  *	Locking: callers must hold output_lock
  */
 
-static void __process_echoes(struct tty_struct *tty)
+static size_t __process_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	int	space, nr;
+	int	space, old_space;
 	size_t tail;
 	unsigned char c;
 
-	space = tty_write_room(tty);
+	old_space = space = tty_write_room(tty);
 
 	tail = ldata->echo_tail;
 	nr = ldata->echo_commit - ldata->echo_tail;

@@ -784,12 +784,13 @@ static void __process_echoes(struct tty_struct *tty)
 	}
 
 	ldata->echo_tail = tail;
+	return old_space - space;
 }
 
 static void commit_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	size_t nr, old;
+	size_t nr, old, echoed;
 	size_t head;
 
 	head = ldata->echo_head;

@@ -804,25 +805,26 @@ static void commit_echoes(struct tty_struct *tty)
 
 	mutex_lock(&ldata->output_lock);
 	ldata->echo_commit = head;
-	__process_echoes(tty);
+	echoed = __process_echoes(tty);
 	mutex_unlock(&ldata->output_lock);
 
-	if (tty->ops->flush_chars)
+	if (echoed && tty->ops->flush_chars)
 		tty->ops->flush_chars(tty);
 }
 
 static void process_echoes(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
+	size_t echoed;
 
 	if (!L_ECHO(tty) || ldata->echo_commit == ldata->echo_tail)
 		return;
 
 	mutex_lock(&ldata->output_lock);
-	__process_echoes(tty);
+	echoed = __process_echoes(tty);
 	mutex_unlock(&ldata->output_lock);
 
-	if (tty->ops->flush_chars)
+	if (echoed && tty->ops->flush_chars)
 		tty->ops->flush_chars(tty);
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:05 UTC #195 lore raw

PATCH v2 8/9 n_tty: Eliminate counter in __process_echoes

In reply to Peter Hurley

Since neither echo_commit nor echo_tail can change for the duration of __process_echoes loop, substitute index comparison for the snapshot counter.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

drivers/tty/n_tty.c +1 -8

index 4a2da9e..e689026 100644
@@ -655,8 +655,7 @@ static size_t __process_echoes(struct tty_struct *tty)
 	old_space = space = tty_write_room(tty);
 
 	tail = ldata->echo_tail;
-	nr = ldata->echo_commit - ldata->echo_tail;
-	while (nr > 0) {
+	while (ldata->echo_commit != tail) {
 		c = echo_buf(ldata, tail);
 		if (c == ECHO_OP_START) {
 			unsigned char op;

@@ -700,20 +699,17 @@ static size_t __process_echoes(struct tty_struct *tty)
 						ldata->column--;
 				}
 				tail += 3;
-				nr -= 3;
 				break;
 
 			case ECHO_OP_SET_CANON_COL:
 				ldata->canon_column = ldata->column;
 				tail += 2;
-				nr -= 2;
 				break;
 
 			case ECHO_OP_MOVE_BACK_COL:
 				if (ldata->column > 0)
 					ldata->column--;
 				tail += 2;
-				nr -= 2;
 				break;
 
 			case ECHO_OP_START:

@@ -726,7 +722,6 @@ static size_t __process_echoes(struct tty_struct *tty)
 				ldata->column++;
 				space--;
 				tail += 2;
-				nr -= 2;
 				break;
 
 			default:

@@ -748,7 +743,6 @@ static size_t __process_echoes(struct tty_struct *tty)
 				ldata->column += 2;
 				space -= 2;
 				tail += 2;
-				nr -= 2;
 			}
 
 			if (no_space_left)

@@ -766,7 +760,6 @@ static size_t __process_echoes(struct tty_struct *tty)
 				space -= 1;
 			}
 			tail += 1;
-			nr -= 1;
 		}
 	}

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:04 UTC #196 lore raw

PATCH v2 9/9 n_tty: Avoid false-sharing echo buffer indices

In reply to Peter Hurley

Separate the head & commit indices from the tail index to avoid cache-line contention (so called 'false-sharing') between concurrent threads.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

drivers/tty/n_tty.c +4 -5

index e689026..b9ccb49 100644
@@ -90,6 +90,8 @@ struct n_tty_data {
 	/* producer-published */
 	size_t read_head;
 	size_t canon_head;
+	size_t echo_head;
+	size_t echo_commit;
 	DECLARE_BITMAP(process_char_map, 256);
 
 	/* private to n_tty_receive_overrun (single-threaded) */

@@ -105,20 +107,17 @@ struct n_tty_data {
 	/* shared by producer and consumer */
 	char *read_buf;
 	DECLARE_BITMAP(read_flags, N_TTY_BUF_SIZE);
+	unsigned char *echo_buf;
 
 	int minimum_to_wake;
 
 	/* consumer-published */
 	size_t read_tail;
 
-	unsigned char *echo_buf;
-	size_t echo_head;
-	size_t echo_tail;
-	size_t echo_commit;
-
 	/* protected by output lock */
 	unsigned int column;
 	unsigned int canon_column;
+	size_t echo_tail;
 
 	struct mutex atomic_read_lock;
 	struct mutex output_lock;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #197 lore raw

v2 cover tty: streamline per-char receiving

In reply to Peter Hurley

** v2 changes **
- Rebase on v2 of 'mostly lockless tty echo'

This 4th of 4 patchsets implements efficient per-char receiving in the N_TTY line discipline. As mentioned in the initial patchset, 'lockless n_tty receive path', casual performance measurements show a 9~15x speedup in end-to-end copying.

Most of this patchset factors out modal conditions from the per-char i/o path. For example, regardless of if ISTRIP is set or not, the modal condition, I_ISTRIP(tty), was tested on every char received, despite that ISTRIP can only be set by termios changes.

The modes handled by separate code paths are:
1) 'real raw' mode (which was already handled separately)
2) raw mode
3) EXTPROC when ISTRIP and IUCLC are not selected
4) tty closing mode
5) LNEXT mode
6) ISTRIP or IUCLC or IPARMRK set
7) everything else

The code duplication is kept to a minimum by carefully factoring out shared slow paths (such as specially handled chars in canonical mode or receiving PARITY or OVERRUN flags); only the per-char loops themselves are duplicated.

Peter Hurley (20):
n_tty: Fix EOF push handling
n_tty: Remove alias ptrs in __receive_buf()
n_tty: Move buffers into n_tty_data
n_tty: Rename process_char_map to char_map
n_tty: Simplify __receive_buf loop count
n_tty: Factor 'real raw' receive_buf into standalone fn
n_tty: Factor signal char handling into separate fn
n_tty: Factor flagged char handling into separate fn
n_tty: Factor raw mode receive_buf() into separate fn
n_tty: Special case EXTPROC receive_buf() as raw mode
n_tty: Factor tty->closing receive_buf() into separate fn
n_tty: Factor standard per-char i/o into separate fn
n_tty: Eliminate char tests from IXANY restart test
n_tty: Split n_tty_receive_char()
n_tty: Factor ISTRIP and IUCLC receive_buf into separate fn
n_tty: Factor PARMRK from normal per-char i/o
n_tty: Remove overflow tests from receive_buf() path
n_tty: Un-inline single-use functions
n_tty: Factor LNEXT processing from per-char i/o path
tty: Remove extra wakeup from pty write() path

 drivers/tty/n_tty.c  | 617 +++++++++++++++++++++++++++++++--------------------
 drivers/tty/pty.c    |   4 +-
 drivers/tty/tty_io.c |   1 -
 include/linux/tty.h  |   1 -
 4 files changed, 381 insertions(+), 242 deletions(-)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #198 lore raw

PATCH v2 1/20 n_tty: Fix EOF push handling

In canonical mode, an EOF which is not the first character of the line causes read() to complete and return the number of characters read so far (commonly referred to as EOF push). However, if the previous read() returned because the user buffer was full _and_ the next character is an EOF not at the beginning of the line, read() must not return 0, thus mistakenly indicating the end-of-file condition.

The TTY_PUSH flag is used to indicate an EOF was received which is not at the beginning of the line. Because the EOF push condition is evaluated by a thread other than the read(), multiple EOF pushes can cause a premature end-of-file to be indicated.

Instead, discover the 'EOF push as first read character' condition from the read() thread itself, and restart the i/o loop if detected.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c  | 34 +++++++++++++++++-----------------
 drivers/tty/tty_io.c |  1 -
 include/linux/tty.h  |  1 -
 3 files changed, 17 insertions(+), 19 deletions(-)

drivers/tty/n_tty.c +17 ~~-17~~

index b9ccb49..55e4e38 100644
@@ -113,6 +113,7 @@ struct n_tty_data {
 
 	/* consumer-published */
 	size_t read_tail;
+	size_t line_start;
 
 	/* protected by output lock */
 	unsigned int column;

@@ -336,6 +337,7 @@ static void reset_buffer_flags(struct n_tty_data *ldata)
 {
 	ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
 	ldata->echo_head = ldata->echo_tail = ldata->echo_commit = 0;
+	ldata->line_start = 0;
 
 	ldata->erasing = 0;
 	bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);

@@ -1395,8 +1397,6 @@ send_signal:
 		if (c == EOF_CHAR(tty)) {
 			if (read_cnt(ldata) >= N_TTY_BUF_SIZE)
 				return;
-			if (ldata->canon_head != ldata->read_head)
-				set_bit(TTY_PUSH, &tty->flags);
 			c = __DISABLED_CHAR;
 			goto handle_newline;
 		}

@@ -1603,6 +1603,7 @@ static void n_tty_set_termios(struct tty_struct *tty, struct ktermios *old)
 		canon_change = (old->c_lflag ^ tty->termios.c_lflag) & ICANON;
 	if (canon_change) {
 		bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);
+		ldata->line_start = 0;
 		ldata->canon_head = ldata->read_tail;
 		ldata->erasing = 0;
 		ldata->lnext = 0;

@@ -1836,6 +1837,7 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	size_t eol;
 	size_t tail;
 	int ret, found = 0;
+	bool eof_push = 0;
 
 	/* N.B. avoid overrun if nr == 0 */
 	n = min(*nr, read_cnt(ldata));

@@ -1862,8 +1864,10 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	n = (found + eol + size) & (N_TTY_BUF_SIZE - 1);
 	c = n;
 
-	if (found && read_buf(ldata, eol) == __DISABLED_CHAR)
+	if (found && read_buf(ldata, eol) == __DISABLED_CHAR) {
 		n--;
+		eof_push = !n && ldata->read_tail != ldata->line_start;
+	}
 
 	n_tty_trace("%s: eol:%zu found:%d n:%zu c:%zu size:%zu more:%zu\n",
 		    __func__, eol, found, n, c, size, more);

@@ -1886,9 +1890,11 @@ static int canon_copy_from_read_buf(struct tty_struct *tty,
 	smp_mb__after_clear_bit();
 	ldata->read_tail += c;
 
-	if (found)
+	if (found) {
+		ldata->line_start = ldata->read_tail;
 		tty_audit_push(tty);
-	return 0;
+	}
+	return eof_push ? -EAGAIN : 0;
 }
 
 extern ssize_t redirected_tty_write(struct file *, const char __user *,

@@ -1963,12 +1969,10 @@ static ssize_t n_tty_read(struct tty_struct *tty, struct file *file,
 	int c;
 	int minimum, time;
 	ssize_t retval = 0;
-	ssize_t size;
 	long timeout;
 	unsigned long flags;
 	int packet;
 
-do_it_again:
 	c = job_control(tty, file);
 	if (c < 0)
 		return c;

@@ -2075,7 +2079,10 @@ do_it_again:
 
 		if (ldata->icanon && !L_EXTPROC(tty)) {
 			retval = canon_copy_from_read_buf(tty, &b, &nr);
-			if (retval)
+			if (retval == -EAGAIN) {
+				retval = 0;
+				continue;
+			} else if (retval)
 				break;
 		} else {
 			int uncopied;

@@ -2103,15 +2110,8 @@ do_it_again:
 		ldata->minimum_to_wake = minimum;
 
 	__set_current_state(TASK_RUNNING);
-	size = b - buf;
-	if (size) {
-		retval = size;
-		if (nr)
-			clear_bit(TTY_PUSH, &tty->flags);
-	} else if (test_and_clear_bit(TTY_PUSH, &tty->flags)) {
-		up_read(&tty->termios_rwsem);
-		goto do_it_again;
-	}
+	if (b - buf)
+		retval = b - buf;
 
 	n_tty_set_room(tty);
 	up_read(&tty->termios_rwsem);

drivers/tty/tty_io.c +0 -1

index 7dbea58..b83c5c5 100644
@@ -664,7 +664,6 @@ static void __tty_hangup(struct tty_struct *tty, int exit_session)
 
 	spin_lock_irq(&tty->ctrl_lock);
 	clear_bit(TTY_THROTTLED, &tty->flags);
-	clear_bit(TTY_PUSH, &tty->flags);
 	clear_bit(TTY_DO_WRITE_WAKEUP, &tty->flags);
 	put_pid(tty->session);
 	put_pid(tty->pgrp);

include/linux/tty.h +0 -1

index 5fd5d6f..554b732 100644
@@ -304,7 +304,6 @@ struct tty_file_private {
 #define TTY_EXCLUSIVE 		3	/* Exclusive open mode */
 #define TTY_DEBUG 		4	/* Debugging */
 #define TTY_DO_WRITE_WAKEUP 	5	/* Call write_wakeup after queuing new */
-#define TTY_PUSH 		6	/* n_tty private */
 #define TTY_CLOSING 		7	/* ->close() in progress */
 #define TTY_LDISC_OPEN	 	11	/* Line discipline is open */
 #define TTY_PTY_LOCK 		16	/* pty private */

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #199 lore raw

PATCH v2 2/20 n_tty: Remove alias ptrs in __receive_buf()

In reply to Peter Hurley

The char and flag buffer local alias pointers, p and f, are unnecessary; remove them.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

drivers/tty/n_tty.c +6 -7

index 55e4e38..b8851b6 100644
@@ -1484,8 +1484,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	const unsigned char *p;
-	char *f, flags = TTY_NORMAL;
+	char	flags = TTY_NORMAL;
 	char	buf[64];
 
 	if (ldata->real_raw) {

@@ -1507,19 +1506,19 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	} else {
 		int i;
 
-		for (i = count, p = cp, f = fp; i; i--, p++) {
-			if (f)
-				flags = *f++;
+		for (i = count; i; i--, cp++) {
+			if (fp)
+				flags = *fp++;
 			switch (flags) {
 			case TTY_NORMAL:
-				n_tty_receive_char(tty, *p);
+				n_tty_receive_char(tty, *cp);
 				break;
 			case TTY_BREAK:
 				n_tty_receive_break(tty);
 				break;
 			case TTY_PARITY:
 			case TTY_FRAME:
-				n_tty_receive_parity_error(tty, *p);
+				n_tty_receive_parity_error(tty, *cp);
 				break;
 			case TTY_OVERRUN:
 				n_tty_receive_overrun(tty);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #200 lore raw

PATCH v2 3/20 n_tty: Move buffers into n_tty_data

In reply to Peter Hurley

Reduce pointer reloading and improve locality-of-reference; allocate read_buf and echo_buf within struct n_tty_data.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 25 +++++++++----------------
 1 file changed, 9 insertions(+), 16 deletions(-)

drivers/tty/n_tty.c +9 ~~-16~~

index b8851b6..3daf9f0 100644
@@ -105,9 +105,9 @@ struct n_tty_data {
 	unsigned char lnext:1, erasing:1, raw:1, real_raw:1, icanon:1;
 
 	/* shared by producer and consumer */
-	char *read_buf;
+	char read_buf[N_TTY_BUF_SIZE];
 	DECLARE_BITMAP(read_flags, N_TTY_BUF_SIZE);
-	unsigned char *echo_buf;
+	unsigned char echo_buf[N_TTY_BUF_SIZE];
 
 	int minimum_to_wake;

@@ -1694,9 +1694,7 @@ static void n_tty_close(struct tty_struct *tty)
 	if (tty->link)
 		n_tty_packet_mode_flush(tty);
 
-	kfree(ldata->read_buf);
-	kfree(ldata->echo_buf);
-	kfree(ldata);
+	vfree(ldata);
 	tty->disc_data = NULL;
 }

@@ -1714,7 +1712,8 @@ static int n_tty_open(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata;
 
-	ldata = kzalloc(sizeof(*ldata), GFP_KERNEL);
+	/* Currently a malloc failure here can panic */
+	ldata = vmalloc(sizeof(*ldata));
 	if (!ldata)
 		goto err;

@@ -1722,16 +1721,14 @@ static int n_tty_open(struct tty_struct *tty)
 	mutex_init(&ldata->atomic_read_lock);
 	mutex_init(&ldata->output_lock);
 
-	/* These are ugly. Currently a malloc failure here can panic */
-	ldata->read_buf = kzalloc(N_TTY_BUF_SIZE, GFP_KERNEL);
-	ldata->echo_buf = kzalloc(N_TTY_BUF_SIZE, GFP_KERNEL);
-	if (!ldata->read_buf || !ldata->echo_buf)
-		goto err_free_bufs;
-
 	tty->disc_data = ldata;
 	reset_buffer_flags(tty->disc_data);
 	ldata->column = 0;
+	ldata->canon_column = 0;
 	ldata->minimum_to_wake = 1;
+	ldata->num_overrun = 0;
+	ldata->no_room = 0;
+	ldata->lnext = 0;
 	tty->closing = 0;
 	/* indicate buffer work may resume */
 	clear_bit(TTY_LDISC_HALTED, &tty->flags);

@@ -1739,10 +1736,6 @@ static int n_tty_open(struct tty_struct *tty)
 	tty_unthrottle(tty);
 
 	return 0;
-err_free_bufs:
-	kfree(ldata->read_buf);
-	kfree(ldata->echo_buf);
-	kfree(ldata);
 err:
 	return -ENOMEM;
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #201 lore raw

PATCH v2 4/20 n_tty: Rename process_char_map to char_map

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 43 ++++++++++++++++++++-----------------------
 1 file changed, 20 insertions(+), 23 deletions(-)

drivers/tty/n_tty.c +20 ~~-23~~

index 3daf9f0..caf37d3 100644
@@ -92,7 +92,7 @@ struct n_tty_data {
 	size_t canon_head;
 	size_t echo_head;
 	size_t echo_commit;
-	DECLARE_BITMAP(process_char_map, 256);
+	DECLARE_BITMAP(char_map, 256);
 
 	/* private to n_tty_receive_overrun (single-threaded) */
 	unsigned long overrun_time;

@@ -1277,7 +1277,7 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	 * handle specially, do shortcut processing to speed things
 	 * up.
 	 */
-	if (!test_bit(c, ldata->process_char_map) || ldata->lnext) {
+	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
 		ldata->lnext = 0;
 		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
 		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {

@@ -1617,41 +1617,38 @@ static void n_tty_set_termios(struct tty_struct *tty, struct ktermios *old)
 	    I_ICRNL(tty) || I_INLCR(tty) || L_ICANON(tty) ||
 	    I_IXON(tty) || L_ISIG(tty) || L_ECHO(tty) ||
 	    I_PARMRK(tty)) {
-		bitmap_zero(ldata->process_char_map, 256);
+		bitmap_zero(ldata->char_map, 256);
 
 		if (I_IGNCR(tty) || I_ICRNL(tty))
-			set_bit('\r', ldata->process_char_map);
+			set_bit('\r', ldata->char_map);
 		if (I_INLCR(tty))
-			set_bit('\n', ldata->process_char_map);
+			set_bit('\n', ldata->char_map);
 
 		if (L_ICANON(tty)) {
-			set_bit(ERASE_CHAR(tty), ldata->process_char_map);
-			set_bit(KILL_CHAR(tty), ldata->process_char_map);
-			set_bit(EOF_CHAR(tty), ldata->process_char_map);
-			set_bit('\n', ldata->process_char_map);
-			set_bit(EOL_CHAR(tty), ldata->process_char_map);
+			set_bit(ERASE_CHAR(tty), ldata->char_map);
+			set_bit(KILL_CHAR(tty), ldata->char_map);
+			set_bit(EOF_CHAR(tty), ldata->char_map);
+			set_bit('\n', ldata->char_map);
+			set_bit(EOL_CHAR(tty), ldata->char_map);
 			if (L_IEXTEN(tty)) {
-				set_bit(WERASE_CHAR(tty),
-					ldata->process_char_map);
-				set_bit(LNEXT_CHAR(tty),
-					ldata->process_char_map);
-				set_bit(EOL2_CHAR(tty),
-					ldata->process_char_map);
+				set_bit(WERASE_CHAR(tty), ldata->char_map);
+				set_bit(LNEXT_CHAR(tty), ldata->char_map);
+				set_bit(EOL2_CHAR(tty), ldata->char_map);
 				if (L_ECHO(tty))
 					set_bit(REPRINT_CHAR(tty),
-						ldata->process_char_map);
+						ldata->char_map);
 			}
 		}
 		if (I_IXON(tty)) {
-			set_bit(START_CHAR(tty), ldata->process_char_map);
-			set_bit(STOP_CHAR(tty), ldata->process_char_map);
+			set_bit(START_CHAR(tty), ldata->char_map);
+			set_bit(STOP_CHAR(tty), ldata->char_map);
 		}
 		if (L_ISIG(tty)) {
-			set_bit(INTR_CHAR(tty), ldata->process_char_map);
-			set_bit(QUIT_CHAR(tty), ldata->process_char_map);
-			set_bit(SUSP_CHAR(tty), ldata->process_char_map);
+			set_bit(INTR_CHAR(tty), ldata->char_map);
+			set_bit(QUIT_CHAR(tty), ldata->char_map);
+			set_bit(SUSP_CHAR(tty), ldata->char_map);
 		}
-		clear_bit(__DISABLED_CHAR, ldata->process_char_map);
+		clear_bit(__DISABLED_CHAR, ldata->char_map);
 		ldata->raw = 0;
 		ldata->real_raw = 0;
 	} else {

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #202 lore raw

PATCH v2 5/20 n_tty: Simplify __receive_buf loop count

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

drivers/tty/n_tty.c +3 -5

index caf37d3..f959dc9 100644
@@ -1504,21 +1504,19 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		memcpy(read_buf_addr(ldata, head), cp, n);
 		ldata->read_head += n;
 	} else {
-		int i;
-
-		for (i = count; i; i--, cp++) {
+		while (count--) {
 			if (fp)
 				flags = *fp++;
 			switch (flags) {
 			case TTY_NORMAL:
-				n_tty_receive_char(tty, *cp);
+				n_tty_receive_char(tty, *cp++);
 				break;
 			case TTY_BREAK:
 				n_tty_receive_break(tty);
 				break;
 			case TTY_PARITY:
 			case TTY_FRAME:
-				n_tty_receive_parity_error(tty, *cp);
+				n_tty_receive_parity_error(tty, *cp++);
 				break;
 			case TTY_OVERRUN:
 				n_tty_receive_overrun(tty);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #203 lore raw

PATCH v2 6/20 n_tty: Factor 'real raw' receive_buf into standalone fn

In reply to Peter Hurley

Convert to modal receive_buf() processing; factor real_raw receive_buf() into n_tty_receive_buf_real_raw().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 51 ++++++++++++++++++++++++++++++---------------------
 1 file changed, 30 insertions(+), 21 deletions(-)

drivers/tty/n_tty.c +30 ~~-21~~

index f959dc9..309d964 100644
@@ -242,7 +242,7 @@ static void n_tty_write_wakeup(struct tty_struct *tty)
 		kill_fasync(&tty->fasync, SIGIO, POLL_OUT);
 }
 
-static inline void n_tty_check_throttle(struct tty_struct *tty)
+static void n_tty_check_throttle(struct tty_struct *tty)
 {
 	if (tty->driver->type == TTY_DRIVER_TYPE_PTY)
 		return;

@@ -1480,6 +1480,28 @@ handle_newline:
  *		publishes read_head and canon_head
  */
 
+static void
+n_tty_receive_buf_real_raw(struct tty_struct *tty, const unsigned char *cp,
+			   char *fp, int count)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	size_t n, head;
+
+	head = ldata->read_head & (N_TTY_BUF_SIZE - 1);
+	n = N_TTY_BUF_SIZE - max(read_cnt(ldata), head);
+	n = min_t(size_t, count, n);
+	memcpy(read_buf_addr(ldata, head), cp, n);
+	ldata->read_head += n;
+	cp += n;
+	count -= n;
+
+	head = ldata->read_head & (N_TTY_BUF_SIZE - 1);
+	n = N_TTY_BUF_SIZE - max(read_cnt(ldata), head);
+	n = min_t(size_t, count, n);
+	memcpy(read_buf_addr(ldata, head), cp, n);
+	ldata->read_head += n;
+}
+
 static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {

@@ -1487,23 +1509,9 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	char	flags = TTY_NORMAL;
 	char	buf[64];
 
-	if (ldata->real_raw) {
-		size_t n, head;
-
-		head = ldata->read_head & (N_TTY_BUF_SIZE - 1);
-		n = N_TTY_BUF_SIZE - max(read_cnt(ldata), head);
-		n = min_t(size_t, count, n);
-		memcpy(read_buf_addr(ldata, head), cp, n);
-		ldata->read_head += n;
-		cp += n;
-		count -= n;
-
-		head = ldata->read_head & (N_TTY_BUF_SIZE - 1);
-		n = N_TTY_BUF_SIZE - max(read_cnt(ldata), head);
-		n = min_t(size_t, count, n);
-		memcpy(read_buf_addr(ldata, head), cp, n);
-		ldata->read_head += n;
-	} else {
+	if (ldata->real_raw)
+		n_tty_receive_buf_real_raw(tty, cp, fp, count);
+	else {
 		while (count--) {
 			if (fp)
 				flags = *fp++;

@@ -1539,8 +1547,6 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		if (waitqueue_active(&tty->read_wait))
 			wake_up_interruptible(&tty->read_wait);
 	}
-
-	n_tty_check_throttle(tty);
 }
 
 static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,

@@ -1548,6 +1554,7 @@ static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 {
 	down_read(&tty->termios_rwsem);
 	__receive_buf(tty, cp, fp, count);
+	n_tty_check_throttle(tty);
 	up_read(&tty->termios_rwsem);
 }

@@ -1563,8 +1570,10 @@ static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,
 	if (!room)
 		ldata->no_room = 1;
 	count = min(count, room);
-	if (count)
+	if (count) {
 		__receive_buf(tty, cp, fp, count);
+		n_tty_check_throttle(tty);
+	}
 
 	up_read(&tty->termios_rwsem);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #204 lore raw

PATCH v2 7/20 n_tty: Factor signal char handling into separate fn

In reply to Peter Hurley

Reduce the monolithic n_tty_receive_char() complexity; factor the handling of INTR_CHAR, QUIT_CHAR and SUSP_CHAR into n_tty_receive_signal_char().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 52 ++++++++++++++++++++++++++++------------------------
 1 file changed, 28 insertions(+), 24 deletions(-)

drivers/tty/n_tty.c +28 ~~-24~~

index 309d964..be5ef56 100644
@@ -1218,6 +1218,26 @@ static inline void n_tty_receive_parity_error(struct tty_struct *tty,
 	wake_up_interruptible(&tty->read_wait);
 }
 
+static void
+n_tty_receive_signal_char(struct tty_struct *tty, int signal, unsigned char c)
+{
+	if (!L_NOFLSH(tty)) {
+		/* flushing needs exclusive termios_rwsem */
+		up_read(&tty->termios_rwsem);
+		n_tty_flush_buffer(tty);
+		tty_driver_flush_buffer(tty);
+		down_read(&tty->termios_rwsem);
+	}
+	if (I_IXON(tty))
+		start_tty(tty);
+	if (L_ECHO(tty)) {
+		echo_char(c, tty);
+		commit_echoes(tty);
+	}
+	isig(signal, tty);
+	return;
+}
+
 /**
  *	n_tty_receive_char	-	perform processing
  *	@tty: terminal device

@@ -1313,30 +1333,14 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	}
 
 	if (L_ISIG(tty)) {
-		int signal;
-		signal = SIGINT;
-		if (c == INTR_CHAR(tty))
-			goto send_signal;
-		signal = SIGQUIT;
-		if (c == QUIT_CHAR(tty))
-			goto send_signal;
-		signal = SIGTSTP;
-		if (c == SUSP_CHAR(tty)) {
-send_signal:
-			if (!L_NOFLSH(tty)) {
-				/* flushing needs exclusive termios_rwsem */
-				up_read(&tty->termios_rwsem);
-				n_tty_flush_buffer(tty);
-				tty_driver_flush_buffer(tty);
-				down_read(&tty->termios_rwsem);
-			}
-			if (I_IXON(tty))
-				start_tty(tty);
-			if (L_ECHO(tty)) {
-				echo_char(c, tty);
-				commit_echoes(tty);
-			}
-			isig(signal, tty);
+		if (c == INTR_CHAR(tty)) {
+			n_tty_receive_signal_char(tty, SIGINT, c);
+			return;
+		} else if (c == QUIT_CHAR(tty)) {
+			n_tty_receive_signal_char(tty, SIGQUIT, c);
+			return;
+		} else if (c == SUSP_CHAR(tty)) {
+			n_tty_receive_signal_char(tty, SIGTSTP, c);
 			return;
 		}
 	}

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #205 lore raw

PATCH v2 8/20 n_tty: Factor flagged char handling into separate fn

In reply to Peter Hurley

Prepare for modal receive_buf() handling; factor handling for TTY_BREAK, TTY_PARITY, TTY_FRAME and TTY_OVERRUN into n_tty_receive_char_flagged().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 50 +++++++++++++++++++++++++++++---------------------
 1 file changed, 29 insertions(+), 21 deletions(-)

drivers/tty/n_tty.c +29 ~~-21~~

index be5ef56..b7c5ab3 100644
@@ -1467,6 +1467,29 @@ handle_newline:
 	put_tty_queue(c, ldata);
 }
 
+static void
+n_tty_receive_char_flagged(struct tty_struct *tty, unsigned char c, char flag)
+{
+	char buf[64];
+
+	switch (flag) {
+	case TTY_BREAK:
+		n_tty_receive_break(tty);
+		break;
+	case TTY_PARITY:
+	case TTY_FRAME:
+		n_tty_receive_parity_error(tty, c);
+		break;
+	case TTY_OVERRUN:
+		n_tty_receive_overrun(tty);
+		break;
+	default:
+		printk(KERN_ERR "%s: unknown flag %d\n",
+		       tty_name(tty, buf), flag);
+		break;
+	}
+}
+
 /**
  *	n_tty_receive_buf	-	data receive
  *	@tty: terminal device

@@ -1510,34 +1533,19 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	char	flags = TTY_NORMAL;
-	char	buf[64];
 
 	if (ldata->real_raw)
 		n_tty_receive_buf_real_raw(tty, cp, fp, count);
 	else {
+		char flag = TTY_NORMAL;
+
 		while (count--) {
 			if (fp)
-				flags = *fp++;
-			switch (flags) {
-			case TTY_NORMAL:
+				flag = *fp++;
+			if (likely(flag == TTY_NORMAL))
 				n_tty_receive_char(tty, *cp++);
-				break;
-			case TTY_BREAK:
-				n_tty_receive_break(tty);
-				break;
-			case TTY_PARITY:
-			case TTY_FRAME:
-				n_tty_receive_parity_error(tty, *cp++);
-				break;
-			case TTY_OVERRUN:
-				n_tty_receive_overrun(tty);
-				break;
-			default:
-				printk(KERN_ERR "%s: unknown flag %d\n",
-				       tty_name(tty, buf), flags);
-				break;
-			}
+			else
+				n_tty_receive_char_flagged(tty, *cp++, flag);
 		}
 
 		flush_echoes(tty);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #206 lore raw

PATCH v2 9/20 n_tty: Factor raw mode receive_buf() into separate fn

In reply to Peter Hurley

Convert to modal receive_buf() processing; factor raw mode per-char i/o into n_tty_receive_buf_raw().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

drivers/tty/n_tty.c +19 -5

index b7c5ab3..40cc0a5 100644
@@ -1258,11 +1258,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	if (ldata->raw) {
-		put_tty_queue(c, ldata);
-		return;
-	}
-
 	if (I_ISTRIP(tty))
 		c &= 0x7f;
 	if (I_IUCLC(tty) && L_IEXTEN(tty))

@@ -1529,6 +1524,23 @@ n_tty_receive_buf_real_raw(struct tty_struct *tty, const unsigned char *cp,
 	ldata->read_head += n;
 }
 
+static void
+n_tty_receive_buf_raw(struct tty_struct *tty, const unsigned char *cp,
+		      char *fp, int count)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	char flag = TTY_NORMAL;
+
+	while (count--) {
+		if (fp)
+			flag = *fp++;
+		if (likely(flag == TTY_NORMAL))
+			put_tty_queue(*cp++, ldata);
+		else
+			n_tty_receive_char_flagged(tty, *cp++, flag);
+	}
+}
+
 static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {

@@ -1536,6 +1548,8 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 
 	if (ldata->real_raw)
 		n_tty_receive_buf_real_raw(tty, cp, fp, count);
+	else if (ldata->raw)
+		n_tty_receive_buf_raw(tty, cp, fp, count);
 	else {
 		char flag = TTY_NORMAL;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:27 UTC #207 lore raw

PATCH v2 10/20 n_tty: Special case EXTPROC receive_buf() as raw mode

In reply to Peter Hurley

When EXTPROC is set without ISTRIP or IUCLC, processing is identical to raw mode; handle this receiving mode as a special-case of raw mode.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

drivers/tty/n_tty.c +2 -1

index 40cc0a5..75af7f5 100644
@@ -1545,10 +1545,11 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {
 	struct n_tty_data *ldata = tty->disc_data;
+	bool preops = I_ISTRIP(tty) || (I_IUCLC(tty) && L_IEXTEN(tty));
 
 	if (ldata->real_raw)
 		n_tty_receive_buf_real_raw(tty, cp, fp, count);
-	else if (ldata->raw)
+	else if (ldata->raw || (L_EXTPROC(tty) && !preops))
 		n_tty_receive_buf_raw(tty, cp, fp, count);
 	else {
 		char flag = TTY_NORMAL;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #208 lore raw

PATCH v2 11/20 n_tty: Factor tty->closing receive_buf() into separate fn

In reply to Peter Hurley

Convert to modal receive_buf() processing; factor receive char processing when tty->closing into n_tty_receive_buf_closing().

Note that EXTPROC when ISTRIP or IUCLC is set continues to be handled by n_tty_receive_char().

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 50 +++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 11 deletions(-)

drivers/tty/n_tty.c +39 ~~-11~~

index 75af7f5..099b02d 100644
@@ -1275,17 +1275,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 		process_echoes(tty);
 	}
 
-	if (tty->closing) {
-		if (I_IXON(tty)) {
-			if (c == START_CHAR(tty)) {
-				start_tty(tty);
-				process_echoes(tty);
-			} else if (c == STOP_CHAR(tty))
-				stop_tty(tty);
-		}
-		return;
-	}
-
 	/*
 	 * If the previous character was LNEXT, or we know that this
 	 * character is not one of the characters that we'll have to

@@ -1462,6 +1451,27 @@ handle_newline:
 	put_tty_queue(c, ldata);
 }
 
+static inline void
+n_tty_receive_char_closing(struct tty_struct *tty, unsigned char c)
+{
+	if (I_ISTRIP(tty))
+		c &= 0x7f;
+	if (I_IUCLC(tty) && L_IEXTEN(tty))
+		c = tolower(c);
+
+	if (I_IXON(tty)) {
+		if (c == STOP_CHAR(tty))
+			stop_tty(tty);
+		else if (c == START_CHAR(tty) ||
+			 (tty->stopped && !tty->flow_stopped && I_IXANY(tty) &&
+			  c != INTR_CHAR(tty) && c != QUIT_CHAR(tty) &&
+			  c != SUSP_CHAR(tty))) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+	}
+}
+
 static void
 n_tty_receive_char_flagged(struct tty_struct *tty, unsigned char c, char flag)
 {

@@ -1541,6 +1551,22 @@ n_tty_receive_buf_raw(struct tty_struct *tty, const unsigned char *cp,
 	}
 }
 
+static void
+n_tty_receive_buf_closing(struct tty_struct *tty, const unsigned char *cp,
+			  char *fp, int count)
+{
+	char flag = TTY_NORMAL;
+
+	while (count--) {
+		if (fp)
+			flag = *fp++;
+		if (likely(flag == TTY_NORMAL))
+			n_tty_receive_char_closing(tty, *cp++);
+		else
+			n_tty_receive_char_flagged(tty, *cp++, flag);
+	}
+}
+
 static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {

@@ -1551,6 +1577,8 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 		n_tty_receive_buf_real_raw(tty, cp, fp, count);
 	else if (ldata->raw || (L_EXTPROC(tty) && !preops))
 		n_tty_receive_buf_raw(tty, cp, fp, count);
+	else if (tty->closing && !L_EXTPROC(tty))
+		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
 		char flag = TTY_NORMAL;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #209 lore raw

PATCH v2 12/20 n_tty: Factor standard per-char i/o into separate fn

In reply to Peter Hurley

Simplify __receive_buf() into a dispatch function; perform per-char processing for all other modes not already handled.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)

drivers/tty/n_tty.c +18 ~~-10~~

index 099b02d..785def1 100644
@@ -1567,6 +1567,23 @@ n_tty_receive_buf_closing(struct tty_struct *tty, const unsigned char *cp,
 	}
 }
 
+static void
+n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
+			   char *fp, int count)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	char flag = TTY_NORMAL;
+
+	while (count--) {
+		if (fp)
+			flag = *fp++;
+		if (likely(flag == TTY_NORMAL))
+			n_tty_receive_char(tty, *cp++);
+		else
+			n_tty_receive_char_flagged(tty, *cp++, flag);
+	}
+}
+
 static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {

@@ -1580,16 +1597,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
-		char flag = TTY_NORMAL;
-
-		while (count--) {
-			if (fp)
-				flag = *fp++;
-			if (likely(flag == TTY_NORMAL))
-				n_tty_receive_char(tty, *cp++);
-			else
-				n_tty_receive_char_flagged(tty, *cp++, flag);
-		}
+		n_tty_receive_buf_standard(tty, cp, fp, count);
 
 		flush_echoes(tty);
 		if (tty->ops->flush_chars)

Signature · 1 line

1.8.1.2

Greg Kroah-Hartman

maintainer

2013-07-24 00:13 UTC #210 lore raw

Re: [PATCH v2 12/20] n_tty: Factor standard per-char i/o into separate fn

Quoted 7 lines · Peter Hurley

Simplify __receive_buf() into a dispatch function; perform per-char processing for all other modes not already handled.
Signed-off-by Peter Hurley <[email protected]>
---
 drivers/tty/n_tty.c | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)

This patch causes the warning:
drivers/tty/n_tty.c: In function ‘n_tty_receive_buf_standard’:
drivers/tty/n_tty.c:1575:21: warning: unused variable ‘ldata’ [-Wunused-variable]
to happen.

Care to fix it up, and refresh the rest of the series (something fails later on), and resend these so that I can apply them?

Everything before this was applied to my tree, you should have gotten emails about them.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-serial" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Peter Hurley

author

2013-07-24 00:49 UTC #211 lore raw

Re: [PATCH v2 12/20] n_tty: Factor standard per-char i/o into separate fn

Quoted 9 lines · Greg Kroah-Hartman

Quoted 7 lines · Peter Hurley
Simplify __receive_buf() into a dispatch function; perform per-char processing for all other modes not already handled.
Signed-off-by Peter Hurley <[email protected]>
---
  drivers/tty/n_tty.c | 28 ++++++++++++++++++----------
  1 file changed, 18 insertions(+), 10 deletions(-)

Quoted 7 lines

This patch causes the warning:
drivers/tty/n_tty.c: In function ‘n_tty_receive_buf_standard’:
drivers/tty/n_tty.c:1575:21: warning: unused variable ‘ldata’ [-Wunused-variable]
to happen.
Care to fix it up, and refresh the rest of the series (something fails later on), and resend these so that I can apply them?

Sorry about that. Working on it right now.

I think these errors came out of a reordering rebase I did by hand a while back. Obviously I need to add per-patch build testing to my workflow. Time to get an SSD :)

Regards,
Peter Hurley
--
To unsubscribe from this list: send the line "unsubscribe linux-serial" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Peter Hurley

author

2013-07-24 12:29 UTC #212 lore raw

PATCH v3 1/9 n_tty: Factor standard per-char i/o into separate fn

In reply to Greg Kroah-Hartman

Simplify __receive_buf() into a dispatch function; perform per-char processing for all other modes not already handled.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

drivers/tty/n_tty.c +17 ~~-10~~

index ed6c4e4..15d95e4 100644
@@ -1568,6 +1568,22 @@ n_tty_receive_buf_closing(struct tty_struct *tty, const unsigned char *cp,
 	}
 }
 
+static void
+n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
+			   char *fp, int count)
+{
+	char flag = TTY_NORMAL;
+
+	while (count--) {
+		if (fp)
+			flag = *fp++;
+		if (likely(flag == TTY_NORMAL))
+			n_tty_receive_char(tty, *cp++);
+		else
+			n_tty_receive_char_flagged(tty, *cp++, flag);
+	}
+}
+
 static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {

@@ -1581,16 +1597,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
-		char flag = TTY_NORMAL;
-
-		while (count--) {
-			if (fp)
-				flag = *fp++;
-			if (likely(flag == TTY_NORMAL))
-				n_tty_receive_char(tty, *cp++);
-			else
-				n_tty_receive_char_flagged(tty, *cp++, flag);
-		}
+		n_tty_receive_buf_standard(tty, cp, fp, count);
 
 		flush_echoes(tty);
 		if (tty->ops->flush_chars)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:29 UTC #213 lore raw

PATCH v3 2/9 n_tty: Eliminate char tests from IXANY restart test

Relocate the IXANY restart tty test to code paths where the the received char is not START_CHAR, STOP_CHAR, INTR_CHAR, QUIT_CHAR or SUSP_CHAR.

Fixes the condition when ISIG if off and one of INTR_CHAR, QUIT_CHAR or SUSP_CHAR does not restart i/o.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

drivers/tty/n_tty.c +12 -7

index 15d95e4..d3b4129 100644
@@ -1269,13 +1269,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 		return;
 	}
 
-	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-	    I_IXANY(tty) && c != START_CHAR(tty) && c != STOP_CHAR(tty) &&
-	    c != INTR_CHAR(tty) && c != QUIT_CHAR(tty) && c != SUSP_CHAR(tty)) {
-		start_tty(tty);
-		process_echoes(tty);
-	}
-
 	/*
 	 * If the previous character was LNEXT, or we know that this
 	 * character is not one of the characters that we'll have to

@@ -1284,6 +1277,13 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	 */
 	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
 		ldata->lnext = 0;
+
+		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
+		    I_IXANY(tty)) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+
 		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
 		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
 			/* beep if no space */

@@ -1330,6 +1330,11 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 		}
 	}
 
+	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) && I_IXANY(tty)) {
+		start_tty(tty);
+		process_echoes(tty);
+	}
+
 	if (c == '\r') {
 		if (I_IGNCR(tty))
 			return;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:30 UTC #214 lore raw

PATCH v3 3/9 n_tty: Split n_tty_receive_char()

In reply to Peter Hurley

Factor 'special' per-char processing into standalone fn, n_tty_receive_char_special(), which handles processing for chars marked in the char_map.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 103 ++++++++++++++++++++++++++++------------------------
 1 file changed, 56 insertions(+), 47 deletions(-)

drivers/tty/n_tty.c +56 ~~-47~~

index d3b4129..140b555 100644
@@ -1254,57 +1254,12 @@ n_tty_receive_signal_char(struct tty_struct *tty, int signal, unsigned char c)
  *		otherwise, publishes read_head via put_tty_queue()
  */
 
-static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+static void
+n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	if (I_ISTRIP(tty))
-		c &= 0x7f;
-	if (I_IUCLC(tty) && L_IEXTEN(tty))
-		c = tolower(c);
-
-	if (L_EXTPROC(tty)) {
-		put_tty_queue(c, ldata);
-		return;
-	}
-
-	/*
-	 * If the previous character was LNEXT, or we know that this
-	 * character is not one of the characters that we'll have to
-	 * handle specially, do shortcut processing to speed things
-	 * up.
-	 */
-	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
-		ldata->lnext = 0;
-
-		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-		    I_IXANY(tty)) {
-			start_tty(tty);
-			process_echoes(tty);
-		}
-
-		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
-			/* beep if no space */
-			if (L_ECHO(tty))
-				process_output('\a', tty);
-			return;
-		}
-		if (L_ECHO(tty)) {
-			finish_erasing(ldata);
-			/* Record the column of first canon char. */
-			if (ldata->canon_head == ldata->read_head)
-				echo_set_canon_col(ldata);
-			echo_char(c, tty);
-			commit_echoes(tty);
-		}
-		if (parmrk)
-			put_tty_queue(c, ldata);
-		put_tty_queue(c, ldata);
-		return;
-	}
-
 	if (I_IXON(tty)) {
 		if (c == START_CHAR(tty)) {
 			start_tty(tty);

@@ -1457,6 +1412,60 @@ handle_newline:
 	put_tty_queue(c, ldata);
 }
 
+static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	int parmrk;
+
+	if (I_ISTRIP(tty))
+		c &= 0x7f;
+	if (I_IUCLC(tty) && L_IEXTEN(tty))
+		c = tolower(c);
+
+	if (L_EXTPROC(tty)) {
+		put_tty_queue(c, ldata);
+		return;
+	}
+
+	/*
+	 * If the previous character was LNEXT, or we know that this
+	 * character is not one of the characters that we'll have to
+	 * handle specially, do shortcut processing to speed things
+	 * up.
+	 */
+	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
+		ldata->lnext = 0;
+
+		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
+		    I_IXANY(tty)) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+
+		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
+		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
+			/* beep if no space */
+			if (L_ECHO(tty))
+				process_output('\a', tty);
+			return;
+		}
+		if (L_ECHO(tty)) {
+			finish_erasing(ldata);
+			/* Record the column of first canon char. */
+			if (ldata->canon_head == ldata->read_head)
+				echo_set_canon_col(ldata);
+			echo_char(c, tty);
+			commit_echoes(tty);
+		}
+		if (parmrk)
+			put_tty_queue(c, ldata);
+		put_tty_queue(c, ldata);
+		return;
+	}
+
+	n_tty_receive_char_special(tty, c);
+}
+
 static inline void
 n_tty_receive_char_closing(struct tty_struct *tty, unsigned char c)
 {

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:29 UTC #215 lore raw

PATCH v3 4/9 n_tty: Factor ISTRIP and IUCLC receive_buf into separate fn

In reply to Peter Hurley

Convert to modal receive_buf processing; factor char receive processing for unusual termios settings out of normal per-char i/o path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)

drivers/tty/n_tty.c +32 ~~-12~~

index 140b555..99bbee4 100644
@@ -1417,16 +1417,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	if (I_ISTRIP(tty))
-		c &= 0x7f;
-	if (I_IUCLC(tty) && L_IEXTEN(tty))
-		c = tolower(c);
-
-	if (L_EXTPROC(tty)) {
-		put_tty_queue(c, ldata);
-		return;
-	}
-
 	/*
 	 * If the previous character was LNEXT, or we know that this
 	 * character is not one of the characters that we'll have to

@@ -1584,7 +1574,34 @@ n_tty_receive_buf_closing(struct tty_struct *tty, const unsigned char *cp,
 
 static void
 n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
-			   char *fp, int count)
+			  char *fp, int count)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	char flag = TTY_NORMAL;
+
+	while (count--) {
+		if (fp)
+			flag = *fp++;
+		if (likely(flag == TTY_NORMAL)) {
+			unsigned char c = *cp++;
+
+			if (I_ISTRIP(tty))
+				c &= 0x7f;
+			if (I_IUCLC(tty) && L_IEXTEN(tty))
+				c = tolower(c);
+			if (L_EXTPROC(tty)) {
+				put_tty_queue(c, ldata);
+				continue;
+			}
+			n_tty_receive_char(tty, c);
+		} else
+			n_tty_receive_char_flagged(tty, *cp++, flag);
+	}
+}
+
+static void
+n_tty_receive_buf_fast(struct tty_struct *tty, const unsigned char *cp,
+		       char *fp, int count)
 {
 	char flag = TTY_NORMAL;

@@ -1611,7 +1628,10 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
-		n_tty_receive_buf_standard(tty, cp, fp, count);
+		if (!preops)
+			n_tty_receive_buf_fast(tty, cp, fp, count);
+		else
+			n_tty_receive_buf_standard(tty, cp, fp, count);
 
 		flush_echoes(tty);
 		if (tty->ops->flush_chars)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:29 UTC #216 lore raw

PATCH v3 5/9 n_tty: Factor PARMRK from normal per-char i/o

In reply to Peter Hurley

Handle PARMRK processing on the slow per-char i/o path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 45 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)

drivers/tty/n_tty.c +43 -2

index 99bbee4..aed1e94 100644
@@ -1457,6 +1457,47 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 }
 
 static inline void
+n_tty_receive_char_fast(struct tty_struct *tty, unsigned char c)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+
+	/*
+	 * If the previous character was LNEXT, or we know that this
+	 * character is not one of the characters that we'll have to
+	 * handle specially, do shortcut processing to speed things
+	 * up.
+	 */
+	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
+		ldata->lnext = 0;
+
+		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
+		    I_IXANY(tty)) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+
+		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - 1)) {
+			/* beep if no space */
+			if (L_ECHO(tty))
+				process_output('\a', tty);
+			return;
+		}
+		if (L_ECHO(tty)) {
+			finish_erasing(ldata);
+			/* Record the column of first canon char. */
+			if (ldata->canon_head == ldata->read_head)
+				echo_set_canon_col(ldata);
+			echo_char(c, tty);
+			commit_echoes(tty);
+		}
+		put_tty_queue(c, ldata);
+		return;
+	}
+
+	n_tty_receive_char_special(tty, c);
+}
+
+static inline void
 n_tty_receive_char_closing(struct tty_struct *tty, unsigned char c)
 {
 	if (I_ISTRIP(tty))

@@ -1609,7 +1650,7 @@ n_tty_receive_buf_fast(struct tty_struct *tty, const unsigned char *cp,
 		if (fp)
 			flag = *fp++;
 		if (likely(flag == TTY_NORMAL))
-			n_tty_receive_char(tty, *cp++);
+			n_tty_receive_char_fast(tty, *cp++);
 		else
 			n_tty_receive_char_flagged(tty, *cp++, flag);
 	}

@@ -1628,7 +1669,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
-		if (!preops)
+		if (!preops && !I_PARMRK(tty))
 			n_tty_receive_buf_fast(tty, cp, fp, count);
 		else
 			n_tty_receive_buf_standard(tty, cp, fp, count);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:29 UTC #217 lore raw

PATCH v3 6/9 n_tty: Remove overflow tests from receive_buf() path

In reply to Peter Hurley

Always pre-figure the space available in the read_buf and limit the inbound receive request to that amount.

For compatibility reasons with the non-flow-controlled interface, n_tty_receive_buf() will continue filling read_buf until all data has been received or receive_room() returns 0.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 85 +++++++++++++++++++++++------------------------------
 1 file changed, 37 insertions(+), 48 deletions(-)

drivers/tty/n_tty.c +37 ~~-48~~

index aed1e94..0f33e0c 100644
@@ -315,12 +315,9 @@ static inline void n_tty_check_unthrottle(struct tty_struct *tty)
  *	not active.
  */
 
-static void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
+static inline void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
 {
-	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
-		*read_buf_addr(ldata, ldata->read_head) = c;
-		ldata->read_head++;
-	}
+	*read_buf_addr(ldata, ldata->read_head++) = c;
 }
 
 /**

@@ -1332,11 +1329,6 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 			return;
 		}
 		if (c == '\n') {
-			if (read_cnt(ldata) >= N_TTY_BUF_SIZE) {
-				if (L_ECHO(tty))
-					process_output('\a', tty);
-				return;
-			}
 			if (L_ECHO(tty) || L_ECHONL(tty)) {
 				echo_char_raw('\n', ldata);
 				commit_echoes(tty);

@@ -1344,8 +1336,6 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 			goto handle_newline;
 		}
 		if (c == EOF_CHAR(tty)) {
-			if (read_cnt(ldata) >= N_TTY_BUF_SIZE)
-				return;
 			c = __DISABLED_CHAR;
 			goto handle_newline;
 		}

@@ -1353,11 +1343,6 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 		    (c == EOL2_CHAR(tty) && L_IEXTEN(tty))) {
 			parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty))
 				 ? 1 : 0;
-			if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk)) {
-				if (L_ECHO(tty))
-					process_output('\a', tty);
-				return;
-			}
 			/*
 			 * XXX are EOL_CHAR and EOL2_CHAR echoed?!?
 			 */

@@ -1387,12 +1372,6 @@ handle_newline:
 	}
 
 	parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-	if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
-		/* beep if no space */
-		if (L_ECHO(tty))
-			process_output('\a', tty);
-		return;
-	}
 	if (L_ECHO(tty)) {
 		finish_erasing(ldata);
 		if (c == '\n')

@@ -1431,14 +1410,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 			start_tty(tty);
 			process_echoes(tty);
 		}
-
-		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
-			/* beep if no space */
-			if (L_ECHO(tty))
-				process_output('\a', tty);
-			return;
-		}
 		if (L_ECHO(tty)) {
 			finish_erasing(ldata);
 			/* Record the column of first canon char. */

@@ -1447,6 +1418,7 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 			echo_char(c, tty);
 			commit_echoes(tty);
 		}
+		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
 		if (parmrk)
 			put_tty_queue(c, ldata);
 		put_tty_queue(c, ldata);

@@ -1475,13 +1447,6 @@ n_tty_receive_char_fast(struct tty_struct *tty, unsigned char c)
 			start_tty(tty);
 			process_echoes(tty);
 		}
-
-		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - 1)) {
-			/* beep if no space */
-			if (L_ECHO(tty))
-				process_output('\a', tty);
-			return;
-		}
 		if (L_ECHO(tty)) {
 			finish_erasing(ldata);
 			/* Record the column of first canon char. */

@@ -1690,8 +1655,23 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			      char *fp, int count)
 {
+	int room, n;
+
 	down_read(&tty->termios_rwsem);
-	__receive_buf(tty, cp, fp, count);
+
+	while (1) {
+		room = receive_room(tty);
+		n = min(count, room);
+		if (!n)
+			break;
+		__receive_buf(tty, cp, fp, n);
+		cp += n;
+		if (fp)
+			fp += n;
+		count -= n;
+	}
+
+	tty->receive_room = room;
 	n_tty_check_throttle(tty);
 	up_read(&tty->termios_rwsem);
 }

@@ -1700,22 +1680,31 @@ static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,
 			      char *fp, int count)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	int room;
+	int room, n, rcvd = 0;
 
 	down_read(&tty->termios_rwsem);
 
-	tty->receive_room = room = receive_room(tty);
-	if (!room)
-		ldata->no_room = 1;
-	count = min(count, room);
-	if (count) {
-		__receive_buf(tty, cp, fp, count);
-		n_tty_check_throttle(tty);
+	while (1) {
+		room = receive_room(tty);
+		n = min(count, room);
+		if (!n) {
+			if (!room)
+				ldata->no_room = 1;
+			break;
+		}
+		__receive_buf(tty, cp, fp, n);
+		cp += n;
+		if (fp)
+			fp += n;
+		count -= n;
+		rcvd += n;
 	}
 
+	tty->receive_room = room;
+	n_tty_check_throttle(tty);
 	up_read(&tty->termios_rwsem);
 
-	return count;
+	return rcvd;
 }
 
 int is_ignored(int sig)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:29 UTC #218 lore raw

PATCH v3 7/9 n_tty: Un-inline single-use functions

In reply to Peter Hurley

gcc will likely inline these single-use functions anyway; remove inline modifier.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

drivers/tty/n_tty.c +5 -6

index 0f33e0c..f516247 100644
@@ -263,7 +263,7 @@ static void n_tty_check_throttle(struct tty_struct *tty)
 	__tty_set_flow_change(tty, 0);
 }
 
-static inline void n_tty_check_unthrottle(struct tty_struct *tty)
+static void n_tty_check_unthrottle(struct tty_struct *tty)
 {
 	if (tty->driver->type == TTY_DRIVER_TYPE_PTY &&
 	    tty->link->ldisc->ops->write_wakeup == n_tty_write_wakeup) {

@@ -1109,7 +1109,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
  *	Locking: ctrl_lock
  */
 
-static inline void isig(int sig, struct tty_struct *tty)
+static void isig(int sig, struct tty_struct *tty)
 {
 	struct pid *tty_pgrp = tty_get_pgrp(tty);
 	if (tty_pgrp) {

@@ -1132,7 +1132,7 @@ static inline void isig(int sig, struct tty_struct *tty)
  *	Note: may get exclusive termios_rwsem if flushing input buffer
  */
 
-static inline void n_tty_receive_break(struct tty_struct *tty)
+static void n_tty_receive_break(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;

@@ -1170,7 +1170,7 @@ static inline void n_tty_receive_break(struct tty_struct *tty)
  *	private.
  */
 
-static inline void n_tty_receive_overrun(struct tty_struct *tty)
+static void n_tty_receive_overrun(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	char buf[64];

@@ -1198,8 +1198,7 @@ static inline void n_tty_receive_overrun(struct tty_struct *tty)
  *		caller holds non-exclusive termios_rwsem
  *		publishes read_head via put_tty_queue()
  */
-static inline void n_tty_receive_parity_error(struct tty_struct *tty,
-					      unsigned char c)
+static void n_tty_receive_parity_error(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:30 UTC #219 lore raw

PATCH v3 8/9 n_tty: Factor LNEXT processing from per-char i/o path

In reply to Peter Hurley

LNEXT processing accounts for ~15% of total cpu time in end-to-end tty i/o; factor the lnext test/clear from the per-char i/o path.

Instead, attempt to immediately handle the literal next char if not at the end of this received buffer; otherwise, handle the first char of the next received buffer as the literal next char, then continue with normal i/o.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 166 +++++++++++++++++++++++++++++-----------------------
 1 file changed, 94 insertions(+), 72 deletions(-)

drivers/tty/n_tty.c +94 ~~-72~~

index f516247..d8cf96a 100644
@@ -1248,9 +1248,11 @@ n_tty_receive_signal_char(struct tty_struct *tty, int signal, unsigned char c)
  *		caller holds non-exclusive termios_rwsem
  *		publishes canon_head if canonical mode is active
  *		otherwise, publishes read_head via put_tty_queue()
+ *
+ *	Returns 1 if LNEXT was received, else returns 0
  */
 
-static void
+static int
 n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;

@@ -1260,24 +1262,24 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 		if (c == START_CHAR(tty)) {
 			start_tty(tty);
 			commit_echoes(tty);
-			return;
+			return 0;
 		}
 		if (c == STOP_CHAR(tty)) {
 			stop_tty(tty);
-			return;
+			return 0;
 		}
 	}
 
 	if (L_ISIG(tty)) {
 		if (c == INTR_CHAR(tty)) {
 			n_tty_receive_signal_char(tty, SIGINT, c);
-			return;
+			return 0;
 		} else if (c == QUIT_CHAR(tty)) {
 			n_tty_receive_signal_char(tty, SIGQUIT, c);
-			return;
+			return 0;
 		} else if (c == SUSP_CHAR(tty)) {
 			n_tty_receive_signal_char(tty, SIGTSTP, c);
-			return;
+			return 0;
 		}
 	}

@@ -1288,7 +1290,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 
 	if (c == '\r') {
 		if (I_IGNCR(tty))
-			return;
+			return 0;
 		if (I_ICRNL(tty))
 			c = '\n';
 	} else if (c == '\n' && I_INLCR(tty))

@@ -1299,7 +1301,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 		    (c == WERASE_CHAR(tty) && L_IEXTEN(tty))) {
 			eraser(c, tty);
 			commit_echoes(tty);
-			return;
+			return 0;
 		}
 		if (c == LNEXT_CHAR(tty) && L_IEXTEN(tty)) {
 			ldata->lnext = 1;

@@ -1311,10 +1313,9 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 					commit_echoes(tty);
 				}
 			}
-			return;
+			return 1;
 		}
-		if (c == REPRINT_CHAR(tty) && L_ECHO(tty) &&
-		    L_IEXTEN(tty)) {
+		if (c == REPRINT_CHAR(tty) && L_ECHO(tty) && L_IEXTEN(tty)) {
 			size_t tail = ldata->canon_head;
 
 			finish_erasing(ldata);

@@ -1325,7 +1326,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 				tail++;
 			}
 			commit_echoes(tty);
-			return;
+			return 0;
 		}
 		if (c == '\n') {
 			if (L_ECHO(tty) || L_ECHONL(tty)) {

@@ -1366,7 +1367,7 @@ handle_newline:
 			kill_fasync(&tty->fasync, SIGIO, POLL_IN);
 			if (waitqueue_active(&tty->read_wait))
 				wake_up_interruptible(&tty->read_wait);
-			return;
+			return 0;
 		}
 	}

@@ -1388,43 +1389,36 @@ handle_newline:
 		put_tty_queue(c, ldata);
 
 	put_tty_queue(c, ldata);
+	return 0;
 }
 
-static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+static inline void
+n_tty_receive_char_inline(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	/*
-	 * If the previous character was LNEXT, or we know that this
-	 * character is not one of the characters that we'll have to
-	 * handle specially, do shortcut processing to speed things
-	 * up.
-	 */
-	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
-		ldata->lnext = 0;
-
-		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-		    I_IXANY(tty)) {
-			start_tty(tty);
-			process_echoes(tty);
-		}
-		if (L_ECHO(tty)) {
-			finish_erasing(ldata);
-			/* Record the column of first canon char. */
-			if (ldata->canon_head == ldata->read_head)
-				echo_set_canon_col(ldata);
-			echo_char(c, tty);
-			commit_echoes(tty);
-		}
-		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (parmrk)
-			put_tty_queue(c, ldata);
-		put_tty_queue(c, ldata);
-		return;
+	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) && I_IXANY(tty)) {
+		start_tty(tty);
+		process_echoes(tty);
 	}
+	if (L_ECHO(tty)) {
+		finish_erasing(ldata);
+		/* Record the column of first canon char. */
+		if (ldata->canon_head == ldata->read_head)
+			echo_set_canon_col(ldata);
+		echo_char(c, tty);
+		commit_echoes(tty);
+	}
+	parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
+	if (parmrk)
+		put_tty_queue(c, ldata);
+	put_tty_queue(c, ldata);
+}
 
-	n_tty_receive_char_special(tty, c);
+static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+{
+	n_tty_receive_char_inline(tty, c);
 }
 
 static inline void

@@ -1432,33 +1426,19 @@ n_tty_receive_char_fast(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 
-	/*
-	 * If the previous character was LNEXT, or we know that this
-	 * character is not one of the characters that we'll have to
-	 * handle specially, do shortcut processing to speed things
-	 * up.
-	 */
-	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
-		ldata->lnext = 0;
-
-		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-		    I_IXANY(tty)) {
-			start_tty(tty);
-			process_echoes(tty);
-		}
-		if (L_ECHO(tty)) {
-			finish_erasing(ldata);
-			/* Record the column of first canon char. */
-			if (ldata->canon_head == ldata->read_head)
-				echo_set_canon_col(ldata);
-			echo_char(c, tty);
-			commit_echoes(tty);
-		}
-		put_tty_queue(c, ldata);
-		return;
+	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) && I_IXANY(tty)) {
+		start_tty(tty);
+		process_echoes(tty);
 	}
-
-	n_tty_receive_char_special(tty, c);
+	if (L_ECHO(tty)) {
+		finish_erasing(ldata);
+		/* Record the column of first canon char. */
+		if (ldata->canon_head == ldata->read_head)
+			echo_set_canon_col(ldata);
+		echo_char(c, tty);
+		commit_echoes(tty);
+	}
+	put_tty_queue(c, ldata);
 }
 
 static inline void

@@ -1505,6 +1485,22 @@ n_tty_receive_char_flagged(struct tty_struct *tty, unsigned char c, char flag)
 	}
 }
 
+static void
+n_tty_receive_char_lnext(struct tty_struct *tty, unsigned char c, char flag)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+
+	ldata->lnext = 0;
+	if (likely(flag == TTY_NORMAL)) {
+		if (I_ISTRIP(tty))
+			c &= 0x7f;
+		if (I_IUCLC(tty) && L_IEXTEN(tty))
+			c = tolower(c);
+		n_tty_receive_char(tty, c);
+	} else
+		n_tty_receive_char_flagged(tty, c, flag);
+}
+
 /**
  *	n_tty_receive_buf	-	data receive
  *	@tty: terminal device

@@ -1598,7 +1594,14 @@ n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
 				put_tty_queue(c, ldata);
 				continue;
 			}
-			n_tty_receive_char(tty, c);
+			if (!test_bit(c, ldata->char_map))
+				n_tty_receive_char_inline(tty, c);
+			else if (n_tty_receive_char_special(tty, c) && count) {
+				if (fp)
+					flag = *fp++;
+				n_tty_receive_char_lnext(tty, *cp++, flag);
+				count--;
+			}
 		} else
 			n_tty_receive_char_flagged(tty, *cp++, flag);
 	}

@@ -1608,14 +1611,24 @@ static void
 n_tty_receive_buf_fast(struct tty_struct *tty, const unsigned char *cp,
 		       char *fp, int count)
 {
+	struct n_tty_data *ldata = tty->disc_data;
 	char flag = TTY_NORMAL;
 
 	while (count--) {
 		if (fp)
 			flag = *fp++;
-		if (likely(flag == TTY_NORMAL))
-			n_tty_receive_char_fast(tty, *cp++);
-		else
+		if (likely(flag == TTY_NORMAL)) {
+			unsigned char c = *cp++;
+
+			if (!test_bit(c, ldata->char_map))
+				n_tty_receive_char_fast(tty, c);
+			else if (n_tty_receive_char_special(tty, c) && count) {
+				if (fp)
+					flag = *fp++;
+				n_tty_receive_char_lnext(tty, *cp++, flag);
+				count--;
+			}
+		} else
 			n_tty_receive_char_flagged(tty, *cp++, flag);
 	}
 }

@@ -1633,6 +1646,15 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
+		if (ldata->lnext) {
+			char flag = TTY_NORMAL;
+
+			if (fp)
+				flag = *fp++;
+			n_tty_receive_char_lnext(tty, *cp++, flag);
+			count--;
+		}
+
 		if (!preops && !I_PARMRK(tty))
 			n_tty_receive_buf_fast(tty, cp, fp, count);
 		else

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-24 12:30 UTC #220 lore raw

PATCH v3 9/9 tty: Remove extra wakeup from pty write() path

In reply to Peter Hurley

Acquiring the write_wait queue spin lock now accounts for the largest slice of cpu time on the tty write path. Two factors contribute to this situation; a overly-pessimistic line discipline write loop which _always_ sets up a wait loop even if i/o will immediately succeed, and on ptys, a wakeup storm from reads and writes.

Writer wakeup does not need to be performed by the pty driver. Firstly, since the actual i/o is performed within the write, the line discipline write loop will continue while space remains in the flip buffers. Secondly, when space becomes avail in the line discipline receive buffer (and thus also in the flip buffers), the pty unthrottle re-wakes the writer (non-flow-controlled line disciplines unconditionally unthrottle the driver when data is received). Thus, existing in-kernel i/o is guaranteed to advance. Finally, writer wakeup occurs at the conclusion of the line discipline write (in tty_write_unlock()). This guarantees that any user-space write waiters are woken to continue additional i/o.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/pty.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

drivers/tty/pty.c +1 -3

index b38a28b..b940127 100644
@@ -121,10 +121,8 @@ static int pty_write(struct tty_struct *tty, const unsigned char *buf, int c)
 		/* Stuff the data into the input queue of the other end */
 		c = tty_insert_flip_string(to->port, buf, c);
 		/* And shovel */
-		if (c) {
+		if (c)
 			tty_flip_buffer_push(to->port);
-			tty_wakeup(tty);
-		}
 	}
 	return c;
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #221 lore raw

PATCH v2 13/20 n_tty: Eliminate char tests from IXANY restart test

In reply to Peter Hurley

Relocate the IXANY restart tty test to code paths where the the received char is not START_CHAR, STOP_CHAR, INTR_CHAR, QUIT_CHAR or SUSP_CHAR.

Fixes the condition when ISIG if off and one of INTR_CHAR, QUIT_CHAR or SUSP_CHAR does not restart i/o.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

drivers/tty/n_tty.c +12 -7

index 785def1..1f21c97 100644
@@ -1268,13 +1268,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 		return;
 	}
 
-	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-	    I_IXANY(tty) && c != START_CHAR(tty) && c != STOP_CHAR(tty) &&
-	    c != INTR_CHAR(tty) && c != QUIT_CHAR(tty) && c != SUSP_CHAR(tty)) {
-		start_tty(tty);
-		process_echoes(tty);
-	}
-
 	/*
 	 * If the previous character was LNEXT, or we know that this
 	 * character is not one of the characters that we'll have to

@@ -1283,6 +1276,13 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	 */
 	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
 		ldata->lnext = 0;
+
+		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
+		    I_IXANY(tty)) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+
 		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
 		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
 			/* beep if no space */

@@ -1329,6 +1329,11 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 		}
 	}
 
+	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) && I_IXANY(tty)) {
+		start_tty(tty);
+		process_echoes(tty);
+	}
+
 	if (c == '\r') {
 		if (I_IGNCR(tty))
 			return;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:27 UTC #222 lore raw

PATCH v2 14/20 n_tty: Split n_tty_receive_char()

In reply to Peter Hurley

Factor 'special' per-char processing into standalone fn, n_tty_receive_char_special(), which handles processing for chars marked in the char_map.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 103 ++++++++++++++++++++++++++++------------------------
 1 file changed, 56 insertions(+), 47 deletions(-)

drivers/tty/n_tty.c +56 ~~-47~~

index 1f21c97..3e96828 100644
@@ -1253,57 +1253,12 @@ n_tty_receive_signal_char(struct tty_struct *tty, int signal, unsigned char c)
  *		otherwise, publishes read_head via put_tty_queue()
  */
 
-static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+static void
+n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	if (I_ISTRIP(tty))
-		c &= 0x7f;
-	if (I_IUCLC(tty) && L_IEXTEN(tty))
-		c = tolower(c);
-
-	if (L_EXTPROC(tty)) {
-		put_tty_queue(c, ldata);
-		return;
-	}
-
-	/*
-	 * If the previous character was LNEXT, or we know that this
-	 * character is not one of the characters that we'll have to
-	 * handle specially, do shortcut processing to speed things
-	 * up.
-	 */
-	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
-		ldata->lnext = 0;
-
-		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-		    I_IXANY(tty)) {
-			start_tty(tty);
-			process_echoes(tty);
-		}
-
-		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
-			/* beep if no space */
-			if (L_ECHO(tty))
-				process_output('\a', tty);
-			return;
-		}
-		if (L_ECHO(tty)) {
-			finish_erasing(ldata);
-			/* Record the column of first canon char. */
-			if (ldata->canon_head == ldata->read_head)
-				echo_set_canon_col(ldata);
-			echo_char(c, tty);
-			commit_echoes(tty);
-		}
-		if (parmrk)
-			put_tty_queue(c, ldata);
-		put_tty_queue(c, ldata);
-		return;
-	}
-
 	if (I_IXON(tty)) {
 		if (c == START_CHAR(tty)) {
 			start_tty(tty);

@@ -1456,6 +1411,60 @@ handle_newline:
 	put_tty_queue(c, ldata);
 }
 
+static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+	int parmrk;
+
+	if (I_ISTRIP(tty))
+		c &= 0x7f;
+	if (I_IUCLC(tty) && L_IEXTEN(tty))
+		c = tolower(c);
+
+	if (L_EXTPROC(tty)) {
+		put_tty_queue(c, ldata);
+		return;
+	}
+
+	/*
+	 * If the previous character was LNEXT, or we know that this
+	 * character is not one of the characters that we'll have to
+	 * handle specially, do shortcut processing to speed things
+	 * up.
+	 */
+	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
+		ldata->lnext = 0;
+
+		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
+		    I_IXANY(tty)) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+
+		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
+		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
+			/* beep if no space */
+			if (L_ECHO(tty))
+				process_output('\a', tty);
+			return;
+		}
+		if (L_ECHO(tty)) {
+			finish_erasing(ldata);
+			/* Record the column of first canon char. */
+			if (ldata->canon_head == ldata->read_head)
+				echo_set_canon_col(ldata);
+			echo_char(c, tty);
+			commit_echoes(tty);
+		}
+		if (parmrk)
+			put_tty_queue(c, ldata);
+		put_tty_queue(c, ldata);
+		return;
+	}
+
+	n_tty_receive_char_special(tty, c);
+}
+
 static inline void
 n_tty_receive_char_closing(struct tty_struct *tty, unsigned char c)
 {

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #223 lore raw

PATCH v2 15/20 n_tty: Factor ISTRIP and IUCLC receive_buf into separate fn

In reply to Peter Hurley

Convert to modal receive_buf processing; factor char receive processing for unusual termios settings out of normal per-char i/o path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 44 +++++++++++++++++++++++++++++++-------------
 1 file changed, 31 insertions(+), 13 deletions(-)

drivers/tty/n_tty.c +31 ~~-13~~

index 3e96828..17cb70e 100644
@@ -1416,16 +1416,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	if (I_ISTRIP(tty))
-		c &= 0x7f;
-	if (I_IUCLC(tty) && L_IEXTEN(tty))
-		c = tolower(c);
-
-	if (L_EXTPROC(tty)) {
-		put_tty_queue(c, ldata);
-		return;
-	}
-
 	/*
 	 * If the previous character was LNEXT, or we know that this
 	 * character is not one of the characters that we'll have to

@@ -1583,9 +1573,34 @@ n_tty_receive_buf_closing(struct tty_struct *tty, const unsigned char *cp,
 
 static void
 n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
-			   char *fp, int count)
+			  char *fp, int count)
+{
+	char flag = TTY_NORMAL;
+
+	while (count--) {
+		if (fp)
+			flag = *fp++;
+		if (likely(flag == TTY_NORMAL)) {
+			unsigned char c = *cp++;
+
+			if (I_ISTRIP(tty))
+				c &= 0x7f;
+			if (I_IUCLC(tty) && L_IEXTEN(tty))
+				c = tolower(c);
+			if (L_EXTPROC(tty)) {
+				put_tty_queue(c, ldata);
+				continue;
+			}
+			n_tty_receive_char(tty, c);
+		} else
+			n_tty_receive_char_flagged(tty, *cp++, flag);
+	}
+}
+
+static void
+n_tty_receive_buf_fast(struct tty_struct *tty, const unsigned char *cp,
+		       char *fp, int count)
 {
-	struct n_tty_data *ldata = tty->disc_data;
 	char flag = TTY_NORMAL;
 
 	while (count--) {

@@ -1611,7 +1626,10 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
-		n_tty_receive_buf_standard(tty, cp, fp, count);
+		if (!preops)
+			n_tty_receive_buf_fast(tty, cp, fp, count);
+		else
+			n_tty_receive_buf_standard(tty, cp, fp, count);
 
 		flush_echoes(tty);
 		if (tty->ops->flush_chars)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #224 lore raw

PATCH v2 16/20 n_tty: Factor PARMRK from normal per-char i/o

In reply to Peter Hurley

Handle PARMRK processing on the slow per-char i/o path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 45 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)

drivers/tty/n_tty.c +43 -2

index 17cb70e..40feeee 100644
@@ -1456,6 +1456,47 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 }
 
 static inline void
+n_tty_receive_char_fast(struct tty_struct *tty, unsigned char c)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+
+	/*
+	 * If the previous character was LNEXT, or we know that this
+	 * character is not one of the characters that we'll have to
+	 * handle specially, do shortcut processing to speed things
+	 * up.
+	 */
+	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
+		ldata->lnext = 0;
+
+		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
+		    I_IXANY(tty)) {
+			start_tty(tty);
+			process_echoes(tty);
+		}
+
+		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - 1)) {
+			/* beep if no space */
+			if (L_ECHO(tty))
+				process_output('\a', tty);
+			return;
+		}
+		if (L_ECHO(tty)) {
+			finish_erasing(ldata);
+			/* Record the column of first canon char. */
+			if (ldata->canon_head == ldata->read_head)
+				echo_set_canon_col(ldata);
+			echo_char(c, tty);
+			commit_echoes(tty);
+		}
+		put_tty_queue(c, ldata);
+		return;
+	}
+
+	n_tty_receive_char_special(tty, c);
+}
+
+static inline void
 n_tty_receive_char_closing(struct tty_struct *tty, unsigned char c)
 {
 	if (I_ISTRIP(tty))

@@ -1607,7 +1648,7 @@ n_tty_receive_buf_fast(struct tty_struct *tty, const unsigned char *cp,
 		if (fp)
 			flag = *fp++;
 		if (likely(flag == TTY_NORMAL))
-			n_tty_receive_char(tty, *cp++);
+			n_tty_receive_char_fast(tty, *cp++);
 		else
 			n_tty_receive_char_flagged(tty, *cp++, flag);
 	}

@@ -1626,7 +1667,7 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
-		if (!preops)
+		if (!preops && !I_PARMRK(tty))
 			n_tty_receive_buf_fast(tty, cp, fp, count);
 		else
 			n_tty_receive_buf_standard(tty, cp, fp, count);

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:27 UTC #225 lore raw

PATCH v2 17/20 n_tty: Remove overflow tests from receive_buf() path

In reply to Peter Hurley

Always pre-figure the space available in the read_buf and limit the inbound receive request to that amount.

For compatibility reasons with the non-flow-controlled interface, n_tty_receive_buf() will continue filling read_buf until all data has been received or receive_room() returns 0.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 85 +++++++++++++++++++++++------------------------------
 1 file changed, 37 insertions(+), 48 deletions(-)

drivers/tty/n_tty.c +37 ~~-48~~

index 40feeee..633cfcf 100644
@@ -314,12 +314,9 @@ static inline void n_tty_check_unthrottle(struct tty_struct *tty)
  *	not active.
  */
 
-static void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
+static inline void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
 {
-	if (read_cnt(ldata) < N_TTY_BUF_SIZE) {
-		*read_buf_addr(ldata, ldata->read_head) = c;
-		ldata->read_head++;
-	}
+	*read_buf_addr(ldata, ldata->read_head++) = c;
 }
 
 /**

@@ -1331,11 +1328,6 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 			return;
 		}
 		if (c == '\n') {
-			if (read_cnt(ldata) >= N_TTY_BUF_SIZE) {
-				if (L_ECHO(tty))
-					process_output('\a', tty);
-				return;
-			}
 			if (L_ECHO(tty) || L_ECHONL(tty)) {
 				echo_char_raw('\n', ldata);
 				commit_echoes(tty);

@@ -1343,8 +1335,6 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 			goto handle_newline;
 		}
 		if (c == EOF_CHAR(tty)) {
-			if (read_cnt(ldata) >= N_TTY_BUF_SIZE)
-				return;
 			c = __DISABLED_CHAR;
 			goto handle_newline;
 		}

@@ -1352,11 +1342,6 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 		    (c == EOL2_CHAR(tty) && L_IEXTEN(tty))) {
 			parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty))
 				 ? 1 : 0;
-			if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk)) {
-				if (L_ECHO(tty))
-					process_output('\a', tty);
-				return;
-			}
 			/*
 			 * XXX are EOL_CHAR and EOL2_CHAR echoed?!?
 			 */

@@ -1386,12 +1371,6 @@ handle_newline:
 	}
 
 	parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-	if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
-		/* beep if no space */
-		if (L_ECHO(tty))
-			process_output('\a', tty);
-		return;
-	}
 	if (L_ECHO(tty)) {
 		finish_erasing(ldata);
 		if (c == '\n')

@@ -1430,14 +1409,6 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 			start_tty(tty);
 			process_echoes(tty);
 		}
-
-		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - parmrk - 1)) {
-			/* beep if no space */
-			if (L_ECHO(tty))
-				process_output('\a', tty);
-			return;
-		}
 		if (L_ECHO(tty)) {
 			finish_erasing(ldata);
 			/* Record the column of first canon char. */

@@ -1446,6 +1417,7 @@ static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
 			echo_char(c, tty);
 			commit_echoes(tty);
 		}
+		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
 		if (parmrk)
 			put_tty_queue(c, ldata);
 		put_tty_queue(c, ldata);

@@ -1474,13 +1446,6 @@ n_tty_receive_char_fast(struct tty_struct *tty, unsigned char c)
 			start_tty(tty);
 			process_echoes(tty);
 		}
-
-		if (read_cnt(ldata) >= (N_TTY_BUF_SIZE - 1)) {
-			/* beep if no space */
-			if (L_ECHO(tty))
-				process_output('\a', tty);
-			return;
-		}
 		if (L_ECHO(tty)) {
 			finish_erasing(ldata);
 			/* Record the column of first canon char. */

@@ -1688,8 +1653,23 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 static void n_tty_receive_buf(struct tty_struct *tty, const unsigned char *cp,
 			      char *fp, int count)
 {
+	int room, n;
+
 	down_read(&tty->termios_rwsem);
-	__receive_buf(tty, cp, fp, count);
+
+	while (1) {
+		room = receive_room(tty);
+		n = min(count, room);
+		if (!n)
+			break;
+		__receive_buf(tty, cp, fp, n);
+		cp += n;
+		if (fp)
+			fp += n;
+		count -= n;
+	}
+
+	tty->receive_room = room;
 	n_tty_check_throttle(tty);
 	up_read(&tty->termios_rwsem);
 }

@@ -1698,22 +1678,31 @@ static int n_tty_receive_buf2(struct tty_struct *tty, const unsigned char *cp,
 			      char *fp, int count)
 {
 	struct n_tty_data *ldata = tty->disc_data;
-	int room;
+	int room, n, rcvd = 0;
 
 	down_read(&tty->termios_rwsem);
 
-	tty->receive_room = room = receive_room(tty);
-	if (!room)
-		ldata->no_room = 1;
-	count = min(count, room);
-	if (count) {
-		__receive_buf(tty, cp, fp, count);
-		n_tty_check_throttle(tty);
+	while (1) {
+		room = receive_room(tty);
+		n = min(count, room);
+		if (!n) {
+			if (!room)
+				ldata->no_room = 1;
+			break;
+		}
+		__receive_buf(tty, cp, fp, n);
+		cp += n;
+		if (fp)
+			fp += n;
+		count -= n;
+		rcvd += n;
 	}
 
+	tty->receive_room = room;
+	n_tty_check_throttle(tty);
 	up_read(&tty->termios_rwsem);
 
-	return count;
+	return rcvd;
 }
 
 int is_ignored(int sig)

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:27 UTC #226 lore raw

PATCH v2 18/20 n_tty: Un-inline single-use functions

In reply to Peter Hurley

gcc will likely inline these single-use functions anyway; remove inline modifier.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

drivers/tty/n_tty.c +5 -6

index 633cfcf..5e06aa1 100644
@@ -263,7 +263,7 @@ static void n_tty_check_throttle(struct tty_struct *tty)
 	__tty_set_flow_change(tty, 0);
 }
 
-static inline void n_tty_check_unthrottle(struct tty_struct *tty)
+static void n_tty_check_unthrottle(struct tty_struct *tty)
 {
 	if (tty->driver->type == TTY_DRIVER_TYPE_PTY) {
 		if (chars_in_buffer(tty) > TTY_THRESHOLD_UNTHROTTLE)

@@ -1108,7 +1108,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
  *	Locking: ctrl_lock
  */
 
-static inline void isig(int sig, struct tty_struct *tty)
+static void isig(int sig, struct tty_struct *tty)
 {
 	struct pid *tty_pgrp = tty_get_pgrp(tty);
 	if (tty_pgrp) {

@@ -1131,7 +1131,7 @@ static inline void isig(int sig, struct tty_struct *tty)
  *	Note: may get exclusive termios_rwsem if flushing input buffer
  */
 
-static inline void n_tty_receive_break(struct tty_struct *tty)
+static void n_tty_receive_break(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;

@@ -1169,7 +1169,7 @@ static inline void n_tty_receive_break(struct tty_struct *tty)
  *	private.
  */
 
-static inline void n_tty_receive_overrun(struct tty_struct *tty)
+static void n_tty_receive_overrun(struct tty_struct *tty)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	char buf[64];

@@ -1197,8 +1197,7 @@ static inline void n_tty_receive_overrun(struct tty_struct *tty)
  *		caller holds non-exclusive termios_rwsem
  *		publishes read_head via put_tty_queue()
  */
-static inline void n_tty_receive_parity_error(struct tty_struct *tty,
-					      unsigned char c)
+static void n_tty_receive_parity_error(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #227 lore raw

PATCH v2 19/20 n_tty: Factor LNEXT processing from per-char i/o path

In reply to Peter Hurley

LNEXT processing accounts for ~15% of total cpu time in end-to-end tty i/o; factor the lnext test/clear from the per-char i/o path.

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/n_tty.c | 167 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 95 insertions(+), 72 deletions(-)

drivers/tty/n_tty.c +95 ~~-72~~

index 5e06aa1..640fb35 100644
@@ -1247,9 +1247,11 @@ n_tty_receive_signal_char(struct tty_struct *tty, int signal, unsigned char c)
  *		caller holds non-exclusive termios_rwsem
  *		publishes canon_head if canonical mode is active
  *		otherwise, publishes read_head via put_tty_queue()
+ *
+ *	Returns 1 if LNEXT was received, else returns 0
  */
 
-static void
+static int
 n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;

@@ -1259,24 +1261,24 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 		if (c == START_CHAR(tty)) {
 			start_tty(tty);
 			commit_echoes(tty);
-			return;
+			return 0;
 		}
 		if (c == STOP_CHAR(tty)) {
 			stop_tty(tty);
-			return;
+			return 0;
 		}
 	}
 
 	if (L_ISIG(tty)) {
 		if (c == INTR_CHAR(tty)) {
 			n_tty_receive_signal_char(tty, SIGINT, c);
-			return;
+			return 0;
 		} else if (c == QUIT_CHAR(tty)) {
 			n_tty_receive_signal_char(tty, SIGQUIT, c);
-			return;
+			return 0;
 		} else if (c == SUSP_CHAR(tty)) {
 			n_tty_receive_signal_char(tty, SIGTSTP, c);
-			return;
+			return 0;
 		}
 	}

@@ -1287,7 +1289,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 
 	if (c == '\r') {
 		if (I_IGNCR(tty))
-			return;
+			return 0;
 		if (I_ICRNL(tty))
 			c = '\n';
 	} else if (c == '\n' && I_INLCR(tty))

@@ -1298,7 +1300,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 		    (c == WERASE_CHAR(tty) && L_IEXTEN(tty))) {
 			eraser(c, tty);
 			commit_echoes(tty);
-			return;
+			return 0;
 		}
 		if (c == LNEXT_CHAR(tty) && L_IEXTEN(tty)) {
 			ldata->lnext = 1;

@@ -1310,10 +1312,9 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 					commit_echoes(tty);
 				}
 			}
-			return;
+			return 1;
 		}
-		if (c == REPRINT_CHAR(tty) && L_ECHO(tty) &&
-		    L_IEXTEN(tty)) {
+		if (c == REPRINT_CHAR(tty) && L_ECHO(tty) && L_IEXTEN(tty)) {
 			size_t tail = ldata->canon_head;
 
 			finish_erasing(ldata);

@@ -1324,7 +1325,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
 				tail++;
 			}
 			commit_echoes(tty);
-			return;
+			return 0;
 		}
 		if (c == '\n') {
 			if (L_ECHO(tty) || L_ECHONL(tty)) {

@@ -1365,7 +1366,7 @@ handle_newline:
 			kill_fasync(&tty->fasync, SIGIO, POLL_IN);
 			if (waitqueue_active(&tty->read_wait))
 				wake_up_interruptible(&tty->read_wait);
-			return;
+			return 0;
 		}
 	}

@@ -1387,43 +1388,36 @@ handle_newline:
 		put_tty_queue(c, ldata);
 
 	put_tty_queue(c, ldata);
+	return 0;
 }
 
-static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+static inline void
+n_tty_receive_char_inline(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 	int parmrk;
 
-	/*
-	 * If the previous character was LNEXT, or we know that this
-	 * character is not one of the characters that we'll have to
-	 * handle specially, do shortcut processing to speed things
-	 * up.
-	 */
-	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
-		ldata->lnext = 0;
-
-		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-		    I_IXANY(tty)) {
-			start_tty(tty);
-			process_echoes(tty);
-		}
-		if (L_ECHO(tty)) {
-			finish_erasing(ldata);
-			/* Record the column of first canon char. */
-			if (ldata->canon_head == ldata->read_head)
-				echo_set_canon_col(ldata);
-			echo_char(c, tty);
-			commit_echoes(tty);
-		}
-		parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
-		if (parmrk)
-			put_tty_queue(c, ldata);
-		put_tty_queue(c, ldata);
-		return;
+	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) && I_IXANY(tty)) {
+		start_tty(tty);
+		process_echoes(tty);
 	}
+	if (L_ECHO(tty)) {
+		finish_erasing(ldata);
+		/* Record the column of first canon char. */
+		if (ldata->canon_head == ldata->read_head)
+			echo_set_canon_col(ldata);
+		echo_char(c, tty);
+		commit_echoes(tty);
+	}
+	parmrk = (c == (unsigned char) '\377' && I_PARMRK(tty)) ? 1 : 0;
+	if (parmrk)
+		put_tty_queue(c, ldata);
+	put_tty_queue(c, ldata);
+}
 
-	n_tty_receive_char_special(tty, c);
+static inline void n_tty_receive_char(struct tty_struct *tty, unsigned char c)
+{
+	n_tty_receive_char_inline(tty, c);
 }
 
 static inline void

@@ -1431,33 +1425,19 @@ n_tty_receive_char_fast(struct tty_struct *tty, unsigned char c)
 {
 	struct n_tty_data *ldata = tty->disc_data;
 
-	/*
-	 * If the previous character was LNEXT, or we know that this
-	 * character is not one of the characters that we'll have to
-	 * handle specially, do shortcut processing to speed things
-	 * up.
-	 */
-	if (!test_bit(c, ldata->char_map) || ldata->lnext) {
-		ldata->lnext = 0;
-
-		if (tty->stopped && !tty->flow_stopped && I_IXON(tty) &&
-		    I_IXANY(tty)) {
-			start_tty(tty);
-			process_echoes(tty);
-		}
-		if (L_ECHO(tty)) {
-			finish_erasing(ldata);
-			/* Record the column of first canon char. */
-			if (ldata->canon_head == ldata->read_head)
-				echo_set_canon_col(ldata);
-			echo_char(c, tty);
-			commit_echoes(tty);
-		}
-		put_tty_queue(c, ldata);
-		return;
+	if (tty->stopped && !tty->flow_stopped && I_IXON(tty) && I_IXANY(tty)) {
+		start_tty(tty);
+		process_echoes(tty);
 	}
-
-	n_tty_receive_char_special(tty, c);
+	if (L_ECHO(tty)) {
+		finish_erasing(ldata);
+		/* Record the column of first canon char. */
+		if (ldata->canon_head == ldata->read_head)
+			echo_set_canon_col(ldata);
+		echo_char(c, tty);
+		commit_echoes(tty);
+	}
+	put_tty_queue(c, ldata);
 }
 
 static inline void

@@ -1504,6 +1484,22 @@ n_tty_receive_char_flagged(struct tty_struct *tty, unsigned char c, char flag)
 	}
 }
 
+static void
+n_tty_receive_char_lnext(struct tty_struct *tty, unsigned char c, char flag)
+{
+	struct n_tty_data *ldata = tty->disc_data;
+
+	ldata->lnext = 0;
+	if (likely(flag == TTY_NORMAL)) {
+		if (I_ISTRIP(tty))
+			c &= 0x7f;
+		if (I_IUCLC(tty) && L_IEXTEN(tty))
+			c = tolower(c);
+		n_tty_receive_char(tty, c);
+	} else
+		n_tty_receive_char_flagged(tty, c, flag);
+}
+
 /**
  *	n_tty_receive_buf	-	data receive
  *	@tty: terminal device

@@ -1580,6 +1576,7 @@ static void
 n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
 			  char *fp, int count)
 {
+	struct n_tty_data *ldata = tty->disc_data;
 	char flag = TTY_NORMAL;
 
 	while (count--) {

@@ -1596,7 +1593,14 @@ n_tty_receive_buf_standard(struct tty_struct *tty, const unsigned char *cp,
 				put_tty_queue(c, ldata);
 				continue;
 			}
-			n_tty_receive_char(tty, c);
+			if (!test_bit(c, ldata->char_map))
+				n_tty_receive_char_inline(tty, c);
+			else if (n_tty_receive_char_special(tty, c) && count) {
+				if (fp)
+					flag = *fp++;
+				n_tty_receive_char_lnext(tty, *cp++, flag);
+				count--;
+			}
 		} else
 			n_tty_receive_char_flagged(tty, *cp++, flag);
 	}

@@ -1606,14 +1610,24 @@ static void
 n_tty_receive_buf_fast(struct tty_struct *tty, const unsigned char *cp,
 		       char *fp, int count)
 {
+	struct n_tty_data *ldata = tty->disc_data;
 	char flag = TTY_NORMAL;
 
 	while (count--) {
 		if (fp)
 			flag = *fp++;
-		if (likely(flag == TTY_NORMAL))
-			n_tty_receive_char_fast(tty, *cp++);
-		else
+		if (likely(flag == TTY_NORMAL)) {
+			unsigned char c = *cp++;
+
+			if (!test_bit(c, ldata->char_map))
+				n_tty_receive_char_fast(tty, c);
+			else if (n_tty_receive_char_special(tty, c) && count) {
+				if (fp)
+					flag = *fp++;
+				n_tty_receive_char_lnext(tty, *cp++, flag);
+				count--;
+			}
+		} else
 			n_tty_receive_char_flagged(tty, *cp++, flag);
 	}
 }

@@ -1631,6 +1645,15 @@ static void __receive_buf(struct tty_struct *tty, const unsigned char *cp,
 	else if (tty->closing && !L_EXTPROC(tty))
 		n_tty_receive_buf_closing(tty, cp, fp, count);
 	else {
+		if (ldata->lnext) {
+			char flag = TTY_NORMAL;
+
+			if (fp)
+				flag = *fp++;
+			n_tty_receive_char_lnext(tty, *cp++, flag);
+			count--;
+		}
+
 		if (!preops && !I_PARMRK(tty))
 			n_tty_receive_buf_fast(tty, cp, fp, count);
 		else

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-06-15 14:21 UTC #228 lore raw

PATCH v2 20/20 tty: Remove extra wakeup from pty write() path

In reply to Peter Hurley

Signed-off-by Peter Hurley <[email protected]>

---
 drivers/tty/pty.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

drivers/tty/pty.c +1 -3

index 0634dd9..b9bc5be 100644
@@ -121,10 +121,8 @@ static int pty_write(struct tty_struct *tty, const unsigned char *buf, int c)
 		/* Stuff the data into the input queue of the other end */
 		c = tty_insert_flip_string(to->port, buf, c);
 		/* And shovel */
-		if (c) {
+		if (c)
 			tty_flip_buffer_push(to->port);
-			tty_wakeup(tty);
-		}
 	}
 	return c;
 }

Signature · 1 line

1.8.1.2

Peter Hurley

author

2013-07-20 17:00 UTC #229 lore raw

Re: [PATCH v2 20/20] tty: Remove extra wakeup from pty write() path

Quoted 6 lines · Peter Hurley

Acquiring the write_wait queue spin lock now accounts for the largest slice of cpu time on the tty write path. Two factors contribute to this situation; a overly-pessimistic line discipline write loop which _always_ sets up a wait loop even if i/o will immediately succeed, and on ptys, a wakeup storm from reads and writes.

Quoted 11 lines

Writer wakeup does not need to be performed by the pty driver. Firstly, since the actual i/o is performed within the write, the line discipline write loop will continue while space remains in the flip buffers. Secondly, when space becomes avail in the line discipline receive buffer (and thus also in the flip buffers), the pty unthrottle re-wakes the writer (non-flow-controlled line disciplines unconditionally unthrottle the driver when data is received). Thus, existing in-kernel i/o is guaranteed to advance. Finally, writer wakeup occurs at the conclusion of the line discipline write (in tty_write_unlock()). This guarantees that any user-space write waiters are woken to continue additional i/o.

Greg,

I thought I should let you know I'm tracking down a bug/regression related to this patch.

In certain unusual pty/ldisc configurations, i/o fails to make forward progress. I still stand by my commit message above, so I'm in the process of instrumenting the i/o path so I can uncover the cause of the failure.

I would still recommend applying this patch to tty-next, as it resolves a much more critical bug discussed here [1].

Doing a write_wakeup() from a driver .write() method is a no-no; recursion is possible and results in a thread stack overrun.

Regards,
Peter Hurley

[1]
https://lkml.org/lkml/2013/7/1/308

Quoted 22 lines

Signed-off-by Peter Hurley <[email protected]>

---
  drivers/tty/pty.c | 4 +---
  1 file changed, 1 insertion(+), 3 deletions(-)

drivers/tty/pty.c +1 -3

index 0634dd9..b9bc5be 100644
@@ -121,10 +121,8 @@ static int pty_write(struct tty_struct *tty, const unsigned char *buf, int c)
  		/* Stuff the data into the input queue of the other end */
  		c = tty_insert_flip_string(to->port, buf, c);
  		/* And shovel */
-		if (c) {
+		if (c)
  			tty_flip_buffer_push(to->port);
-			tty_wakeup(tty);
-		}
  	}
  	return c;
  }

Peter Hurley

author

2013-07-23 12:57 UTC #230 lore raw

Re: [PATCH v2 20/20] tty: Remove extra wakeup from pty write() path

Quoted 21 lines · Peter Hurley

Quoted 6 lines · Peter Hurley
Acquiring the write_wait queue spin lock now accounts for the largest slice of cpu time on the tty write path. Two factors contribute to this situation; a overly-pessimistic line discipline write loop which _always_ sets up a wait loop even if i/o will immediately succeed, and on ptys, a wakeup storm from reads and writes.
Quoted 11 lines
Writer wakeup does not need to be performed by the pty driver. Firstly, since the actual i/o is performed within the write, the line discipline write loop will continue while space remains in the flip buffers. Secondly, when space becomes avail in the line discipline receive buffer (and thus also in the flip buffers), the pty unthrottle re-wakes the writer (non-flow-controlled line disciplines unconditionally unthrottle the driver when data is received). Thus, existing in-kernel i/o is guaranteed to advance. Finally, writer wakeup occurs at the conclusion of the line discipline write (in tty_write_unlock()). This guarantees that any user-space write waiters are woken to continue additional i/o.
Greg,

Quoted 7 lines

I thought I should let you know I'm tracking down a bug/regression related to this patch.
In certain unusual pty/ldisc configurations, i/o fails to make forward progress. I still stand by my commit message above, so I'm in the process of instrumenting the i/o path so I can uncover the cause of the failure.

Mystery solved.

[PATCH v4 23/24] n_tty: Special case pty flow control from the lockless n_tty receive path series introduced a regression in which i/o failed to advance.

This only occurred when one end of a pty pair was set to an ldisc other than N_TTY. The special case optimization which that patch introduces failed to address that configuration.

I've sent a v5 of that patch to resolve the regression.

Regards,
Peter Hurley

Greg Kroah-Hartman

maintainer

2013-07-23 15:01 UTC #231 lore raw

Re: [PATCH v2 20/20] tty: Remove extra wakeup from pty write() path

Quoted 36 lines · Peter Hurley

Quoted 21 lines · Peter Hurley
Quoted 6 lines · Peter Hurley
Acquiring the write_wait queue spin lock now accounts for the largest slice of cpu time on the tty write path. Two factors contribute to this situation; a overly-pessimistic line discipline write loop which _always_ sets up a wait loop even if i/o will immediately succeed, and on ptys, a wakeup storm from reads and writes.
Quoted 11 lines
Writer wakeup does not need to be performed by the pty driver. Firstly, since the actual i/o is performed within the write, the line discipline write loop will continue while space remains in the flip buffers. Secondly, when space becomes avail in the line discipline receive buffer (and thus also in the flip buffers), the pty unthrottle re-wakes the writer (non-flow-controlled line disciplines unconditionally unthrottle the driver when data is received). Thus, existing in-kernel i/o is guaranteed to advance. Finally, writer wakeup occurs at the conclusion of the line discipline write (in tty_write_unlock()). This guarantees that any user-space write waiters are woken to continue additional i/o.
Greg,
Quoted 7 lines
I thought I should let you know I'm tracking down a bug/regression related to this patch.
In certain unusual pty/ldisc configurations, i/o fails to make forward progress. I still stand by my commit message above, so I'm in the process of instrumenting the i/o path so I can uncover the cause of the failure.
Mystery solved.
[PATCH v4 23/24] n_tty: Special case pty flow control from the lockless n_tty receive path series introduced a regression in which i/o failed to advance.

Quoted 5 lines

This only occurred when one end of a pty pair was set to an ldisc other than N_TTY. The special case optimization which that patch introduces failed to address that configuration.
I've sent a v5 of that patch to resolve the regression.

Thanks for that, I'll work to queue all of your tty patches up today, as they have been sitting out of the tree for too long :)

greg k-h

Greg Kroah-Hartman

maintainer Applied

2013-07-24 00:04 UTC #232 lore raw

Re: [PATCH v2 0/9] mostly lockless tty echo

In reply to Peter Hurley

Quoted 19 lines · Peter Hurley

** v2 changes **
- Rebased on v2 of 'lockless tty flip buffers'
This 3rd of 4 patchsets implements a mostly-lockless tty echo output.
Because echoing is performed solely by the single-threaded ldisc receive_buf() method, most of the lockless requirements are already in-place. The main existing complications were;
1) Echoing could overrun itself. A fixed-size buffer is used to record the operations necessary when outputting the echoes; the most recent echo data is preserved.
2) An attempt to push unprocessed echoes is made by the n_tty_write method (on a different thread) before attempting write output.
The overrun condition is solved by outputting the echoes in blocks and ensuring that at least that much space is available each time an echo operation is committed. At the conclusion of each flip buffer received, any remaining unprocessed echoes are output.

Quoted 5 lines

This block output method is particularly effective when there is no reader (the tty is output-only) and termios is misconfigured with echo enabled.
The concurrent access by the n_tty_write() method is already excluded by the existing output_lock mutex.

Applied, thanks.

greg k-h

Greg Kroah-Hartman

maintainer

2013-07-23 23:53 UTC #233 lore raw

Re: [PATCH v2 00/16] lockless tty flip buffers

In reply to Peter Hurley

Quoted 28 lines · Peter Hurley

** v2 changes **
- Rebased on v4 of 'lockless n_tty receive path'
This 2nd of 4 patchsets implements lockless receive from the tty driver. By lockless, I'm referring to the 'lock' spin lock formerly used to serialize access to the flip buffer list.
Since the driver-side flip buffer usage is already single-threaded and line discipline receiving is already single-threaded, implementing a lockless flip buffer list was the primary hurdle. [The only 2 flip buffer consumers, flush_to_ldisc() and tty_buffer_flush() were already mutually exclusive and this exclusion remains although the mechanism is changed.]
Since the flip buffer consumers, flush_to_ldisc() and tty_buffer_flush(), already leave the last-consumed flip buffer on the list, and since the existing flip buffer api is already divided into an add/commit interface, most of the requirement for a lockless algorithm was already in-place. The main differences are;
1) the initial state of the flip buffer list points head and tail to a 0-sized sentinel flip buffer. This eliminates head & tail NULL testing and assigning the head ptr from the driver-side thread. This sentinel is 'consumed' on the first iteration of ldisc receiving and does not require special-case logic.
2) the free list uses the atomic singly-linked llist interface. While this guarantees safe concurrent usage by both producer and consumer, it's not optimal. Both producer and consumer unnecessarily contend over the free list head ptr; a better approach would be to maintain

Quoted 8 lines

an unconsumed buffer in the same way the flip buffer list works. Light testing has shown this contention accounts for roughly 5% of total cpu time in end-to-end copying.
3) The mutual exclusion between consumers is reimplemented as a mutex; this eliminates the need to drop the lock across the ldisc receive_buf() method. This mutual exclusion is extended to a public interface which the vt driver now uses to safely utilize the ldisc receive_buf() interface when pasting a selection.

All applied, thanks.

greg k-h

Greg Kroah-Hartman

maintainer

2013-06-17 20:01 UTC #234 lore raw

In reply to Peter Hurley

Quoted 10 lines · Peter Hurley

Greg,
As before, this patchset is dependent on the 'ldsem patchset'. The reason is that this series abandons tty->receive_room as a flow control mechanism (because that requires locking), and the TIOCSETD ioctl _without ldsem_ uses tty->receive_room to shutoff i/o.
It is also dependent on 'n_tty fixes' which I recently resent to you.

Thanks for rebasing all of this, and resending it. It's late in the 3.11 development cycle, so I'm going to wait unto 3.11-rc1 is out to queue it up for 3.12, to get it lots of testing in linux-next. So don't worry, they aren't lost, just waiting a few more weeks before I can apply them.

thanks,

greg k-h

Peter Hurley

author

2013-06-17 20:32 UTC #235 lore raw

Quoted 9 lines · Greg Kroah-Hartman

Quoted 8 lines · Peter Hurley
Greg,
As before, this patchset is dependent on the 'ldsem patchset'. The reason is that this series abandons tty->receive_room as a flow control mechanism (because that requires locking), and the TIOCSETD ioctl _without ldsem_ uses tty->receive_room to shutoff i/o.

Quoted 6 lines

Quoted 2 lines
It is also dependent on 'n_tty fixes' which I recently resent to you.
Thanks for rebasing all of this, and resending it. It's late in the 3.11 development cycle, so I'm going to wait unto 3.11-rc1 is out to queue it up for 3.12, to get it lots of testing in linux-next.

Thanks Greg. I actually meant to suggest these be held until after the merge window but I got distracted by the VT1 deallocation bug report.

Regards,
Peter Hurley

Greg Kroah-Hartman

maintainer

2013-07-23 23:44 UTC #236 lore raw

Quoted 13 lines · Peter Hurley

Quoted 12 lines · Greg Kroah-Hartman
Quoted 10 lines · Peter Hurley
Greg,
As before, this patchset is dependent on the 'ldsem patchset'. The reason is that this series abandons tty->receive_room as a flow control mechanism (because that requires locking), and the TIOCSETD ioctl _without ldsem_ uses tty->receive_room to shutoff i/o.
It is also dependent on 'n_tty fixes' which I recently resent to you.

Quoted 7 lines

Quoted 3 lines
Thanks for rebasing all of this, and resending it. It's late in the 3.11 development cycle, so I'm going to wait unto 3.11-rc1 is out to queue it up for 3.12, to get it lots of testing in linux-next.
Thanks Greg. I actually meant to suggest these be held until after the merge window but I got distracted by the VT1 deallocation bug report.

Now applied, thanks.

greg k-h