Skip to contentSkip to conversation

Strange problem with tty layer

DISCUSSION Lennart Sorensen 2 participants 11 messages Jan 24, 2007 – Feb 17, 2007 Char and misc drivers

lore
Share & tools
lore
#0 lore raw

I am seeing a very strange problem which seems to be in the tty layer.

I am using an exar 17D154 based PCI card (like the digi neo style card) using the jsm driver. Kernel version 2.6.16.25.

My test involves connecting two ports together with a cross over cable and then sending a test pattern of characters. I am currently using 11520 characters at 230400bps (so 0.5s for the transfer).

Most of the time, it works perfectly, but once in a while, I never receiver the last few characters (between 1 and 65 or so characters it seems).

I have confirmed that the driver does in fact receive all the characters, and that they are correct, and that they are being passed to the tty layer using tty_insert_flip_string, and that it returns that all the characters have been passed to the tty layer. The user space application however still doesn't see the last few characters (when it fails).

The problem seems to occour every few hours of testing on a 266MHz Geode SC1200. When I change the clock to 133MHz, it happens every few minutes instead (so much more frequently). I suspect there is some race condition that allows the tty layer to not get around to processing all the data in the buffer, even when asked for data by the application (which is waiting on the serial port using select, with a 4s timeout).

Any suggestions on where to head next to debug this?

Sending a few more characters when the receive times out before getting the complete message does not cause the missing data to arive it seems, only completely closing and starting the test program again seems to recover it, although I may just not have been patient enough.

--
Len Sorensen

#1 lore raw
Quoted 7 lines · Lennart Sorensen

I have confirmed that the driver does in fact receive all the characters, and that they are correct, and that they are being passed to the tty layer using tty_insert_flip_string, and that it returns that all the characters have been passed to the tty layer. The user space application however still doesn't see the last few characters (when it fails).

Quoted 6 lines

The problem seems to occour every few hours of testing on a 266MHz Geode SC1200. When I change the clock to 133MHz, it happens every few minutes instead (so much more frequently). I suspect there is some race condition that allows the tty layer to not get around to processing all the data in the buffer, even when asked for data by the application (which is waiting on the serial port using select, with a 4s timeout).

In 2.6.16 the tty buffering pushes data to the line discipline without regard to tty->receive_room.
If the line discipline can't keep up, the data gets dropped. I observed this data loss at higher speeds when placing the system under heavy load.

2.6.18 added code to respect tty->receive_room.

This may or may not be your problem, but you should be able to check by adding a conditional printk to drivers/char/tty_io.c:flush_to_ldisc()

If tty->receive_room is less than the size of the buffer passed to disc->receive_buf() then you are losing data.

Signature · 2 lines
Paul Fulghum
Microgate Systems, Ltd.
#2 lore raw
Quoted 6 lines · Paul Fulghum

In 2.6.16 the tty buffering pushes data to the line discipline without regard to tty->receive_room.
If the line discipline can't keep up, the data gets dropped. I observed this data loss at higher speeds when placing the system under heavy load.

Quoted 8 lines

2.6.18 added code to respect tty->receive_room.

This may or may not be your problem, but you should be able to check by adding a conditional printk to drivers/char/tty_io.c:flush_to_ldisc()

If tty->receive_room is less than the size of the buffer passed to disc->receive_buf() then you are losing data.

Sounds plausible. Certainly higher cpu load makes the problem occour more often, and making the system slower made it worse too. I will take a look at the tty_io.c in 2.6.18 and compare it against 2.6.16.

--
Len Sorensen

#3 lore raw
In reply to Paul Fulghum
Quoted 6 lines · Paul Fulghum

In 2.6.16 the tty buffering pushes data to the line discipline without regard to tty->receive_room.
If the line discipline can't keep up, the data gets dropped. I observed this data loss at higher speeds when placing the system under heavy load.

Quoted 8 lines

2.6.18 added code to respect tty->receive_room.

This may or may not be your problem, but you should be able to check by adding a conditional printk to drivers/char/tty_io.c:flush_to_ldisc()

If tty->receive_room is less than the size of the buffer passed to disc->receive_buf() then you are losing data.

I am now trying this, which so far seem to help (I had a printk in there earlier and managed to trigger that).

drivers/char/tty_io.c +8 -0
@@ -2774,6 +2778,14 @@
 	spin_lock_irqsave(&tty->buf.lock, flags);
 	while((tbuf = tty->buf.head) != NULL) {
 		while ((count = tbuf->commit - tbuf->read) != 0) {
+			if (!tty->receive_room) {
+				schedule_delayed_work(&tty->buf.work, 1);
+				spin_unlock_irqrestore(&tty->buf.lock, flags);
+				goto out;
+			}
+			if (count > tty->receive_room) {
+				count = tty->receive_room;
+			}
 			char_buf = tbuf->char_buf_ptr + tbuf->read;
 			flag_buf = tbuf->flag_buf_ptr + tbuf->read;
 			tbuf->read += count;

This appeared to be (essentially) the key change in 2.6.18 related to the check of tty->receive_room.

I will now run a bunch more tests to see if it manages to keep it from having any more character losses.

Thank you for the suggestion of where to look.

--
Len Sorensen

#4 lore raw
Quoted 21 lines · Lennart Sorensen

I am now trying this, which so far seem to help (I had a printk in there earlier and managed to trigger that).

drivers/char/tty_io.c +8 -0
@@ -2774,6 +2778,14 @@
 	spin_lock_irqsave(&tty->buf.lock, flags);
 	while((tbuf = tty->buf.head) != NULL) {
 		while ((count = tbuf->commit - tbuf->read) != 0) {
+			if (!tty->receive_room) {
+				schedule_delayed_work(&tty->buf.work, 1);
+				spin_unlock_irqrestore(&tty->buf.lock, flags);
+				goto out;
+			}
+			if (count > tty->receive_room) {
+				count = tty->receive_room;
+			}
 			char_buf = tbuf->char_buf_ptr + tbuf->read;
 			flag_buf = tbuf->flag_buf_ptr + tbuf->read;
 			tbuf->read += count;
Quoted 7 lines

This appeared to be (essentially) the key change in 2.6.18 related to the check of tty->receive_room.

I will now run a bunch more tests to see if it manages to keep it from having any more character losses.

Thank you for the suggestion of where to look.

Well it turns out that didn't help. Neither does 2.6.18 (that one was the easiest newer one to try). It does seem as if the error rate is lower with 2.6.18 than with 2.6.16, so perhaps there was more than one place that could cause losses in the tty buffering. I had only 2 failures in 15 hours with 2.6.18, rather than a whole lot of failures with 2.6.16. I guess I will have to try 2.6.19 or even something newer.

--
Len Sorensen

#5 lore raw
Quoted 6 lines · Lennart Sorensen

Well it turns out that didn't help. Neither does 2.6.18 (that one was the easiest newer one to try). It does seem as if the error rate is lower with 2.6.18 than with 2.6.16, so perhaps there was more than one place that could cause losses in the tty buffering. I had only 2 failures in 15 hours with 2.6.18, rather than a whole lot of failures with 2.6.16. I guess I will have to try 2.6.19 or even something newer.

You can eliminate the tty buffering altogether by observing what gets passed to the line discipline.

I assume you are using the default line discipline N_TTY.

Look at what is passed to drivers/char/n_tty.c:n_tty_receive_buf() If all the data gets that far, then there is some issue with the line discipline or something further downstream. If not, then the problem is with the tty buffering (assuming you are correct that all data gets to the tty buffering code followed by a tty_flip_buffer_push call).

Signature · 2 lines
Paul Fulghum
Microgate Systems, Ltd.
#6 lore raw
Quoted 2 lines · Paul Fulghum

You can eliminate the tty buffering altogether by observing what gets passed to the line discipline.

I will have to find where in the code that is happening.

Quoted 8 lines

I assume you are using the default line discipline N_TTY.

Look at what is passed to drivers/char/n_tty.c:n_tty_receive_buf() If all the data gets that far, then there is some issue with the line discipline or something further downstream. If not, then the problem is with the tty buffering (assuming you are correct that all data gets to the tty buffering code followed by a tty_flip_buffer_push call).

I am not sure actually. I just open /dev/ttyn0 and /dev/ttyn1 and write to one, and read from the other. I didn't even know about the line diciplines actually. How do I tell which one I am using?

I have confirmed that all the data is being passed to tty_insert_flip_string() in jsm_input().

--
Len Sorensen

#7 lore raw
Quoted 3 lines · Lennart Sorensen

I am not sure actually. I just open /dev/ttyn0 and /dev/ttyn1 and write to one, and read from the other. I didn't even know about the line diciplines actually. How do I tell which one I am using?

ioctl(TIOCSETD/TIOCGETD) sets/returns an integer identifier that can be compared agains the N_XXX macros. If you are not explicitly setting this then is is probably the default N_TTY.

Also at the application level, look at tcsetattr() for setting the termios features. Look specifically at the c_cc[VTIME] and c_cc[VMIN] members of the termios structure. These settings control how much data must be available before returning data to a read(). Try VTIME=0 and VMIN=1.

Since your 'missing' data is always on the tail end, maybe VMIN is set to 64 or something.

Signature · 2 lines
Paul Fulghum
Microgate Systems, Ltd.
#8 lore raw
Quoted 3 lines · Paul Fulghum

ioctl(TIOCSETD/TIOCGETD) sets/returns an integer identifier that can be compared agains the N_XXX macros. If you are not explicitly setting this then is is probably the default N_TTY.

Yes it is N_TTY (value 0). I never set it.

Quoted 8 lines

Also at the application level, look at tcsetattr() for setting the termios features. Look specifically at the c_cc[VTIME] and c_cc[VMIN] members of the termios structure. These settings control how much data must be available before returning data to a read(). Try VTIME=0 and VMIN=1.

Since your 'missing' data is always on the tail end, maybe VMIN is set to 64 or something.

OK, the tty settings according to stty that I am using are:

# stty -F /dev/ttyn0 -a
speed 230400 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -ixon -ixoff -iuclc -ixany -imaxbel
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten -echo -echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

So min and time seems good. I am not explicitly setting those in my test program (unless one of the other settings implies it). I will add it explicitly just in case too.

--
Len Sorensen

#9 lore raw
Quoted 10 lines · Lennart Sorensen

OK, the tty settings according to stty that I am using are:

# stty -F /dev/ttyn0 -a
speed 230400 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
(quoted diff) +0 -4
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -ixon -ixoff -iuclc -ixany -imaxbel
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten -echo -echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke
Quoted 3 lines

So min and time seems good. I am not explicitly setting those in my test program (unless one of the other settings implies it). I will add it explicitly just in case too.

I have run some tests with 8 patches from Linus's 2.6 tree on top of the 2.6.16.25 along with a bit of debug code in the n_tty.c, which ran perfectly for 3 days. I now removed the debug code to see if it will still run perfectly with those 8 patches. The patches I applied in order are:

commit 70522e121a521aa09bd0f4e62e1aa68708b798e1
commit 41c28ff1635e71af072c4711ff5fadd5855d48e7
commit 1aef821a6b3aeca8c19d06aee012ed9db617d1e3
commit ee37df7877eeaa16d7761cce64854110a7c17ad9
commit e1a2509023785bd3199ac068ab80155aeba01265
commit 817d6d3bceaf34c99f5343820f9b9e6021f0655c
commit 2c3bb20f46709a0adfa7ea408013edbcab945d5a
commit 33b37a33c242542fac2980b8ccd90977388b7a8d

Hopefully it will stay working, and actually be a solution to my tty problem in 2.6.16.25.

2.6.18.2 (Debian 2.6.18-4) on the other hand was still broken, but I am not sure which if any of the above commits are not in that version (the date ranges made me think they all should be, but maybe I got the date of 2.6.18 wrong, or not all commits in the linux-2.6 tree are actually part of a given release. I am still figuring out git).

--
Len Sorensen

#10 lore raw
Quoted 15 lines · Lennart Sorensen

I have run some tests with 8 patches from Linus's 2.6 tree on top of the 2.6.16.25 along with a bit of debug code in the n_tty.c, which ran perfectly for 3 days. I now removed the debug code to see if it will still run perfectly with those 8 patches. The patches I applied in order are:

commit 70522e121a521aa09bd0f4e62e1aa68708b798e1
commit 41c28ff1635e71af072c4711ff5fadd5855d48e7
commit 1aef821a6b3aeca8c19d06aee012ed9db617d1e3
commit ee37df7877eeaa16d7761cce64854110a7c17ad9
commit e1a2509023785bd3199ac068ab80155aeba01265
commit 817d6d3bceaf34c99f5343820f9b9e6021f0655c
commit 2c3bb20f46709a0adfa7ea408013edbcab945d5a
commit 33b37a33c242542fac2980b8ccd90977388b7a8d

Quoted 8 lines

Hopefully it will stay working, and actually be a solution to my tty problem in 2.6.16.25.

2.6.18.2 (Debian 2.6.18-4) on the other hand was still broken, but I am not sure which if any of the above commits are not in that version (the date ranges made me think they all should be, but maybe I got the date of 2.6.18 wrong, or not all commits in the linux-2.6 tree are actually part of a given release. I am still figuring out git).

2.6.18 does work, it failed in different ways (overruns on the fifo) which I dealt with by ensuring disk io was synced before testing the serial port.

--
Len Sorensen