TCP memory pressure question

When a Linux machine has reached the tcp_mem limit, what will happen to
'write's on non-blocking sockets? Will they block until more TCP memory is available? Will they return an error code? ENOMEM?

If it varies by kernel version, details about different versions would be
extremely helpful. I'm most interested in late 2.4 kernels.

Thanks in advance.

DS

David Schwartz
<[email protected]>

from write(2) man page.

EAGAIN Non-blocking I/O has been selected using O_NONBLOCK and the write would block.

// Gianni Tedesco (gianni at ecsc dot co dot uk)
lynx --source www.scaramanga.co.uk/gianni-at-ecsc.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

So this would be a case where 'poll' or 'select' would return a write hit
for a socket but 'write' would return -1 and set errno to EAGAIN.

DS

Is this really true? It would livelock several servers I've worked on...

-- Jamie

We're now getting close to my motivation for asking this question.

If it does in fact return EAGAIN, then a poll/select loop program that is
trying to write could get into trouble if another process was responsible for large receive queues. It would spin uselessly burning its timeslice as more data comes in and it delays the execution of the other process that might be able to drain receive queues.

On the other hand, if send blocked, there would be a disaster if a
select/poll loop process were the only TCP user on the box. In this case, it would deadlock. The process is blocked waiting for memory but the only way to get that memory is for the process to get around to reading from other connections, which it can't do because it's blocked.

My motivation in assessing what happens is to develop a sane application
strategy to detect and handle this condition. If we could detect it, we could try not to do TCP writes and try to do reads to relieve the memory problem.

Some strategies I was considering were this:

1) For applications that probably aren't using most of the TCP memory: In
the select/poll loop, we keep track of whether we were able to do any work. If we never successfully wrote or read, but did get a hit on at least one socket, we sleep for a few milliseconds. (We hope that while we sleep other applications can free up more memory for us by draining their receive queues.)

2) For applications that probably are using all or a significant fraction of
the TCP memory: If a 'write' for a socket that we got a write hit on returns EAGAIN, for the rest of this poll/select cycle, we do not attempt any writes, only reads. (We hope that this will relieve the memory pressure scenario and allow the system to function normally as quickly as possible.)

3) For applications that have control over how much load they take: If
writes begin returning EAGAIN under suspicious circumstances, treat this as an overload condition and trigger your handling logic. Refuse expensive commands or refuse to handle new connections.

Any comments or suggestions are appreciated. I've found that when we hit TCP
memory pressure, many applications become very badly behaved.

DS

TCP

What about:

int WRITE(int handle, char *whereto, int len)

{
        int cnt=len;

        while(len>0)
        {
                int rc;

                rc = write(handle, whereto, len);

                if (rc == -1)
                {
                        if (errno == EINTR)
				{
					/* just try again */
				}
				else if (errno == EAGAIN)
				{
					/* give up time-slice */
					if (sched_yield() == -1)
					{
						/* BIG troubles */
                              	syslog(LOG_DEBUG, "WRITE(), during EAGAIN
handling: sched_yield failed! [%d - %s]", errno, strerror(errno));
                               	return -1;
					}
				}
				else
                        {
                                syslog(LOG_DEBUG, "WRITE(): io-error [%d -
%s]", errno, strerror(errno));
                                return -1;
                        }
                }
                else if (rc == 0)
                {
                        return 0;
                }
                else
                {
                        whereto += rc;
                        len -= rc;
                }
        }

        return cnt;
}

By yielding your time slice, you prevent yourself from performing any
'read's. So what's going to fix the memory pressure?

There are two cases where this could work:

1) For an application that uses very little TCP memory. By yielding, perhaps
some other process will come along, a more heavy TCP memory user, and fix the problem.

2) For an application that uses a thread-per-connection architecture. By
yielding, we give other threads a chance to run and hope that they will relieve the memory pressure.

But for a poll/select loop application that is heavily using TCP memory,
this makes things worse. By stopping our own execution, we delay the time when we'll do 'read's. So we extend the TCP memory pressure problem for even longer.

DS