API to flush rx fifo?

I see the uart_ops.flush_buffer method which is used to flush the UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

Quoted 4 lines · Grant Edwards

I see the uart_ops.flush_buffer method which is used to flush the UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

If you're seeing stale i/o, it's more likely due to the flip buffers not being flushed

Quoted 7 lines · Peter Hurley

Quoted 4 lines · Grant Edwards

I see the uart_ops.flush_buffer method which is used to flush the UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

Thanks. I couldn't see any mechanism to do that, and I thought I must be missing something.

Quoted 2 lines

If you're seeing stale i/o, it's more likely due to the flip buffers not being flushed

Probably. There is a scenario where you can get old data because the rx fifo isn't flushed, but I suspect it's not what my customer is complaining about. FWIW, here's the scenario I'm worrying about:

1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver that handles that flow control in hardware[1].

2) Stop making read() calls on the tty device.

3) The buffers in the tty layer fill up, so the uart driver stops transferring data from the rx fifo to the tty layer.

4) The rx fifo fills up, and the flow control stops the other end from sending data.

[all working OK up to this point, now you wait for an arbitrary amount of time]

5) tcflush(TCIFLUSH) is called.

[data in the tty layer gets flushed, but old data in the rx
fifo remains]

6) Now that there's room for more rx data in the tty layer, the uart driver resumes transferring (old) data from the rx fifo to the tty layer.

7) read() is called and returns data that was received an arbitrary amount of time before tcflush(TCIFLUSH) was called.

My "old" drivers that interfaced directly with the tty layer handled this scenario, but those drivers were becoming unmaintainable because of instability of the tty API over the range of kernel versions I support. So, I converted them over to be "uart drivers" for the common serial layer which has a much more stable API (and generally requires much less code). Now I've got a customer complaining about not being able to flush data, so I'm looking closer at how the tcflush() calls are handled.

[1] Because of a bug in the serial-driver layer's handling of the setting of Xon/Xoff characters by the settermios() call, it's not possible to correctly use Xon/Xoff support in a UART for the case where the user wants to use non-default Xon/Xoff characters.

Quoted 30 lines · Grant Edwards

Quoted 7 lines · Peter Hurley

Quoted 4 lines · Grant Edwards

I see the uart_ops.flush_buffer method which is used to flush the UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

Thanks. I couldn't see any mechanism to do that, and I thought I must be missing something.

Quoted 2 lines

If you're seeing stale i/o, it's more likely due to the flip buffers not being flushed

Probably. There is a scenario where you can get old data because the rx fifo isn't flushed, but I suspect it's not what my customer is complaining about. FWIW, here's the scenario I'm worrying about:

1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver that handles that flow control in hardware[1].

2) Stop making read() calls on the tty device.

3) The buffers in the tty layer fill up, so the uart driver stops transferring data from the rx fifo to the tty layer.

4) The rx fifo fills up, and the flow control stops the other end from sending data.

Quoted 7 lines

[all working OK up to this point, now you wait for an arbitrary amount of time]

5) tcflush(TCIFLUSH) is called.

[data in the tty layer gets flushed, but old data in the rx
fifo remains]

Yep. Your driver continues to push new data as it should, but that's getting buffered up in the flip buffers. So that is what the app is reading now that it has restarted read()s.

Your hardware rx fifo shouldn't have stale data in it because that should generate an overrun; ie., if the flip buffers cannot accept data because they're full then the next char pushed when space becomes available should be a NUL flagged with TTY_OVERRUN.

Would a trial patch help (something that you might have to carry yourself for a little while until I could do some historical research)?

Quoted 4 lines

[1] Because of a bug in the serial-driver layer's handling of the setting of Xon/Xoff characters by the settermios() call, it's not possible to correctly use Xon/Xoff support in a UART for the case where the user wants to use non-default Xon/Xoff characters.

Can you be more specific about why setting non-default START_CHAR() and STOP_CHAR() didn't work? I recently overhauled this code so if there's a problem here, we should probably fix it.

Quoted 39 lines · Peter Hurley

Quoted 30 lines · Grant Edwards

Quoted 7 lines · Peter Hurley

Quoted 4 lines · Grant Edwards

I see the uart_ops.flush_buffer method which is used to flush the UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

Thanks. I couldn't see any mechanism to do that, and I thought I must be missing something.

Quoted 2 lines

If you're seeing stale i/o, it's more likely due to the flip buffers not being flushed

Probably. There is a scenario where you can get old data because the rx fifo isn't flushed, but I suspect it's not what my customer is complaining about. FWIW, here's the scenario I'm worrying about:

1) Enable either RTS/CTS or Xon/Xoff flow control for a UART driver that handles that flow control in hardware[1].

2) Stop making read() calls on the tty device.

3) The buffers in the tty layer fill up, so the uart driver stops transferring data from the rx fifo to the tty layer.

4) The rx fifo fills up, and the flow control stops the other end from sending data.

Quoted 7 lines

[all working OK up to this point, now you wait for an arbitrary amount of time]

5) tcflush(TCIFLUSH) is called.

[data in the tty layer gets flushed, but old data in the rx
fifo remains]

Quoted 8 lines

Yep. Your driver continues to push new data as it should, but that's getting buffered up in the flip buffers. So that is what the app is reading now that it has restarted read()s.

Your hardware rx fifo shouldn't have stale data in it because that should generate an overrun; ie., if the flip buffers cannot accept data because they're full then the next char pushed when space becomes available should be a NUL flagged with TTY_OVERRUN.

If flow control is enabled, there should be no rx overruns -- that's what flow control is for. In the scenario above, flow control is enabled (and working). In order to allow the UART to handle flow control, the UART driver must stop reading data from the rx fifo when the tty layer is "full". The documentation for the serial core API specifically states that UARTs are allowed to implement flow control in hardware, and the only way that can be done is to alow the rx fifo to fill up when the application stops makeing read() calls and the tty layer fills up.

I think in newer kernels instead of explicitly checking for room in the tty layer before unloading the rx fifo, the UART is supposed to rely on the throttle/unthrottle callbacks, but the end result is the
same: when the tty layer gets "full", the UART driver stops reading data from the rx fifo, and the rx fifo fills up.

Quoted 15 lines · Grant Edwards

Quoted 4 lines

Your hardware rx fifo shouldn't have stale data in it because that should generate an overrun; ie., if the flip buffers cannot accept data because they're full then the next char pushed when space becomes available should be a NUL flagged with TTY_OVERRUN.

If flow control is enabled, there should be no rx overruns -- that's what flow control is for. In the scenario above, flow control is enabled (and working). In order to allow the UART to handle flow control, the UART driver must stop reading data from the rx fifo when the tty layer is "full". The documentation for the serial core API specifically states that UARTs are allowed to implement flow control in hardware, and the only way that can be done is to alow the rx fifo to fill up when the application stops makeing read() calls and the tty layer fills up.

Quoted 5 lines

I think in newer kernels instead of explicitly checking for room in the tty layer before unloading the rx fifo, the UART is supposed to rely on the throttle/unthrottle callbacks, but the end result is the
same: when the tty layer gets "full", the UART driver stops reading data from the rx fifo, and the rx fifo fills up.

AFAIK, only USB serial stops reading the rx fifo on throttle;

the serial core and other tty drivers continue to empty the rx fifo -- throttle only shuts off the transmitter on the other end.

Without handling throttle/unthrottle, how are you determining that the tty layer is "full"? Return code from tty_insert_flip_xxxx()?

Quoted 23 lines · Peter Hurley

Quoted 20 lines · Grant Edwards

Quoted 4 lines

Your hardware rx fifo shouldn't have stale data in it because that should generate an overrun; ie., if the flip buffers cannot accept data because they're full then the next char pushed when space becomes available should be a NUL flagged with TTY_OVERRUN.

If flow control is enabled, there should be no rx overruns -- that's what flow control is for. In the scenario above, flow control is enabled (and working). In order to allow the UART to handle flow control, the UART driver must stop reading data from the rx fifo when the tty layer is "full". The documentation for the serial core API specifically states that UARTs are allowed to implement flow control in hardware, and the only way that can be done is to alow the rx fifo to fill up when the application stops makeing read() calls and the tty layer fills up.

I think in newer kernels instead of explicitly checking for room in the tty layer before unloading the rx fifo, the UART is supposed to rely on the throttle/unthrottle callbacks, but the end result is the
same: when the tty layer gets "full", the UART driver stops reading data from the rx fifo, and the rx fifo fills up.

Quoted 1 line

AFAIK, only USB serial stops reading the rx fifo on throttle;

All the drivers I maintain do that. It's the only way to get flow control to work. For UART with large FIFOs (e.g. 1KB) -- espcially those attached via USB or Ethernet -- flow control driven by code in serial_core just doesn't work right: you've got to let the UART handle it.

Quoted 5 lines

the serial core and other tty drivers continue to empty the rx fifo -- throttle only shuts off the transmitter on the other end.

Without handling throttle/unthrottle, how are you determining that the tty layer is "full"? Return code from tty_insert_flip_xxxx()?

I check tty->receive_room.

What are you supposed to do for kernel
versions that don't have the throttle()/unthrottle() callbacks?

Quoted 5 lines

All the drivers I maintain do that. It's the only way to get flow control to work. For UART with large FIFOs (e.g. 1KB) -- espcially those attached via USB or Ethernet -- flow control driven by code in serial_core just doesn't work right: you've got to let the UART handle it.

I had a similar situation with the firewire serial driver (which fakes serial i/o over the firewire bus @ 250~400Mb/s). The existing throttle mechanism is too ponderous to shut-off the transmitter before the receiver overflows the flip buffers.

Quoted 4 lines

Quoted 2 lines

Without handling throttle/unthrottle, how are you determining that the tty layer is "full"? Return code from tty_insert_flip_xxxx()?

I check tty->receive_room.

That will be going away as a means of flow control because it's not thread-safe (if you backscan this list, my 'lockless n_tty receive path' patchset only keeps tty->receive_room for the non-flow controlled line disciplines).

Quoted 2 lines

What are you supposed to do for kernel versions that don't have the throttle()/unthrottle() callbacks?

Which versions specifically do you mean?

Quoted 10 lines · Peter Hurley

Quoted 5 lines

All the drivers I maintain do that. It's the only way to get flow control to work. For UART with large FIFOs (e.g. 1KB) -- espcially those attached via USB or Ethernet -- flow control driven by code in serial_core just doesn't work right: you've got to let the UART handle it.

I had a similar situation with the firewire serial driver (which fakes serial i/o over the firewire bus @ 250~400Mb/s). The existing throttle mechanism is too ponderous to shut-off the transmitter before the receiver overflows the flip buffers.

Any time you combine high baud rates, large FIFOs, and latency, you run into that problem.

Quoted 9 lines

Quoted 4 lines

Quoted 2 lines

Without handling throttle/unthrottle, how are you determining that the tty layer is "full"? Return code from tty_insert_flip_xxxx()?

I check tty->receive_room.

That will be going away as a means of flow control because it's not thread-safe (if you backscan this list, my 'lockless n_tty receive path' patchset only keeps tty->receive_room for the non-flow controlled line disciplines).

Good to know.

Will tty_prepare_flip_string_flags() continue to return a "room" value? If so, then I could also switch over to using that instead of looking at tty->receive_room. [The advantage being simpler backwards compatibility.]

The tty_prepare_flip_string_flags() approach also uses a lot less CPU time than the uart_insert_char() approach.

I'll probably add throttle()/unthrottle() callbacks that set/clear an internal flag that tells the driver whether or not to read data from the rx fifo. That should less overhead than polling the tty layer by calling tty_prepare_flip_string_flags(). But, it doesn't help the situation for kernels before 3.8.

Quoted 4 lines

Quoted 2 lines

What are you supposed to do for kernel versions that don't have the throttle()/unthrottle() callbacks?

Which versions specifically do you mean?

My serial_core UART drivers support 2.6.25 and newer. The tech support guys would like support further back, but I've given up trying for anything older than that. Before our recent change-over to serial_core drivers (a few months back) I supported 2.6.18 and later. Tech support would have liked to go back to 2.6.12, which is still in use by some customers who've tested their systems with some ancient version of RH and don't want to change it. One customer just moved from 2.4 to 2.6 about a year ago.

I believe that the throttle/unthrottle callbacks didn't show up until 3.8. I doubt we even have any customers running 3.8 yet. :)

Quoted 2 lines

the serial core and other tty drivers continue to empty the rx fifo -- throttle only shuts off the transmitter on the other end.

From another point of view: not reading the rx fifo is how one "shuts

Quoted 2 lines

Without handling throttle/unthrottle, how are you determining that the tty layer is "full"? Return code from tty_insert_flip_xxxx()?

I check tty->receive_room. What are you supposed to do for kernel versions that don't have the throttle()/unthrottle() callbacks?

Quoted 2 lines · Grant Edwards

Quoted 2 lines

the serial core and other tty drivers continue to empty the rx fifo -- throttle only shuts off the transmitter on the other end.

Let's say that the tty layer fills up, and calls the uart driver's throttle() callback. Now, instead of stopping the unloading of the rx FIFO (like my drivers do), we explicitly send an XOFF and continue to unload the rx FIFO. Will the tty layer accept the 1KB (or 4KB or 8KB) of data that's already in the rx FIFO without any loss/overrun? AFAIK, it can't. That's why my drivers (and the USB drivers) do what they do.

From another point of view: not reading the rx fifo is how one "shuts
off the transmitter on the other end" (eventually) when one is using the UART's flow control support -- at least that's how all the flow-control-supporting UARTs I've seen work. Stopping the other end just because the tty layer is full is also wasteful: some of my "UARTs" can buffer up to 9KB of receive data in the rx fifo[1] after the tty layer fills up. They'll tell the other end to stop when it needs to be stopped. Aside from that, having the flow state controlled from two places (serial_core and the UART itself) tends to be error-prone.

When you're dealing with high baud rates (e.g. 921K) and a bus like USB or Ethernet, where there may be tens or hundreds of milliseconds of latency between the serial_core and the physical UART chip, and FIFO sizes are measured in KB. In those situations there just isn't any way for either the serial_core or tty layer to know when the transmitter on the other end needs to be turned on or off. You have to let the UART handle it. All the tty/serial_core layer knows is whether or not it has room for more receive data, and that's not the same thing as knowing when the transmitter on other end of the serial cable needs to be turned on/off.

So, when I add the UART-driver callbacks for throttle/unthrottle, what they'll have to do is control whether we read data from the rx FIFO or not.

So that implies that for tcflush(TCIFLUSH) to reliably do what a user would expect, there needs to be an input_flush() callback so that the uart driver can be told to flush the rx fifo. [Yes, flushing the "rx fifo" in some of these configurations isn't a simple task that takes a negligible amount of time, but it can be done.]

Quoted 5 lines

Quoted 2 lines

Without handling throttle/unthrottle, how are you determining that the tty layer is "full"? Return code from tty_insert_flip_xxxx()?

I check tty->receive_room. What are you supposed to do for kernel versions that don't have the throttle()/unthrottle() callbacks?

Actually, in another driver I check the return value from tty_prepare_flip_string_flags(). For various (mostly historical) reaons, one driver uses tty_prepare_flip_string_flags() and the other uses uart_insert_char() [both followed later by a call to tty_flip_buffer_push()].

And in yet another driver (that talks to the tty layer instead of using the serial_core), I call the line discipline's receive_room() method -- but that's moot for this thread.

[1] For ethernet/usb attached UARTs the "rx fifo" includes (from serial_core's POV) not only the hardware fifo in the UART (up to 1KB), but also software buffers at either end of the network "connection" (up to 4KB at each end).

Quoted 4 lines · Grant Edwards

Quoted 3 lines · Peter Hurley

Quoted 2 lines · Grant Edwards

I see the uart_ops.flush_buffer method which is used to flush the UART's tx fifo (presumably when the user calls tcflush(TCOFLUSH)).

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

Hi Grant,

I know this is old but I had marked this for re-review in my mail, and after re-looking at this problem, I realized I gave you some bad info.

Since you were referring to uart_ops, I'll assume this is for your serial-core mini-driver(s).

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

How does the rx fifo get flushed when the user calls tcflush(TCIFLUSH)?

It doesn't.

The ioctl(TCIFLUSH) does in fact flush the tty buffers.

If you want to flush your hardware rx fifo, you can define the .ioctl() method in your struct uart_ops and handle TCIFLUSH there.

Note: be sure to return -ENOIOCTLCMD so that TCIFLUSH processing still gets seen by the ldisc!

Like this (apologies in advance for formatting; my mailer is dain-bramaged),

/* called with port->mutex held */
static int xxxxxx_ioctl(struct uart_port *port, unsigned int cmd,
unsigned long arg)

{
	if (cmd == TCFLSH) {
		switch (arg) {
		case TCIFLUSH:
		case TCIOFLUSH:
			spin_lock_irq(&port->lock);
			xxxxxx_empty_rx_fifo(port);
			spin_unlock_irq(&port->lock);
			/* fall-through */
		}
	}
	return -ENOIOCTLCMD;
}

Sorry, hope this isn't too late :(