Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(Note: bug originally posted on BugTraq, this response is cross-posted to the LKML because of the way the server hosting the X font server ground to a halt.)

Re-verified on Debian GNU/Linux running kernel 2.4.18-xfs with Mozilla 1.0.0 and XFree86 4.1.0 with the font server running on a separate server also running on Debian GNU/Linux with kernel 2.4.18-xfs and the X font server from XFree86 4.1.0.

The server running the X font server has 512MB RAM, 1GB swap, has one ext2 partition (/boot), one ReiserFS partition (Squid cache) and the rest are XFS partitions. It also runs the kernel-mode NFS server v1.0 and Samba 2.2.4, among other things, and is typically headless.

Here I had a workstation (that I thought 'hey, I can reboot this after I've felt how the problem works') that used a remote font server, which was company-wide. Yes, I'm the administrator so I killed myself. I've got a personal note to myself, as well. "Don't try stuff posted on BugTraq on anything connected to the network, dummy." ;)

Anyway, FWIW:

First I noticed that the Mozilla window stopped responding immediately. So I closed it and breathed deeply. Then I noticed the server hosting the X font server started thrashing really badly. /home is exported from the same server via kernel-mode NFS, so file access to /home started freezing on workstations. Samba access slowed down, but continued to work.

I was able to log on to the server with enough time to SIGKILL the xfs-daemon process. Unfortunately this wasn't good enough. The server started running up various processes stuck in "D" state, the OOM killer went on panic mode and started killing things left and right, mostly from what I notice apache and apache-ssl processes with messages like "Out of
Memory: Killed process xxxx (apache)". I was also able to do a free after killing xfs-daemon and noticed that there was a lot of free memory both physically and on swap.

Within less than ten minutes on this relatively lightly-loaded server, I could not log in to the machine locally, even as root (whose home directory is not NFS-exported) and load levels shot up to around 25, which is definitely abnormal. Existing logged-on processes also got stuck in whatever they were doing (ps ax, in particular is what I remember).

Attempted reboots locally via Ctrl-Alt-Delete and Magic SysRq failed because (1) I had disabled ctrl-alt-delete mapping in inittab "for security", and (2) I had not compiled Magic SysRq support on this particular server. More notes to self.

In my case the workstation was easy to stabilize. I was actually able to close all windows and exit xfce properly. It's the server running the X font server that ground to painful halt. I do not know how things would have been if I had the Magic SysRq enabled.

While I agree with BugTraq posts in response to this that applications like Mozilla which accept font-sizes from unknown sources should have some check to prevent such large sizes from crashing X and/or the X Font Server, I'm alarmed by (1) the way the X font server allows itself to be crashed like this, and (2) the way the entire Linux kernel seems to have been unable to handle the situation. While having a central company or department wide font server may not have been the best choice I made, this seems like a simple way to drive a stake through a system's heart.

Suggestions on how to work around this on multiple levels would definitely be appreciated. I'll be starting by removing the X font server from our file and authentication server onto some high-powered workstation, but I'm sure this won't be enough, and knowing that a user process like xfs-daemon can drag the Linux kernel down to knees is not very comforting. :(

--> Jijo

- --

Federico Sevilla III   :  <http://jijo.free.net.ph/>
Network Administrator  :  The Leather Collection, Inc.
GnuPG Key ID           :  0x93B746BE

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD4DBQE9B/kJ5rCBSJO3Rr4RAqYOAJ90o1C6RnU7Lgyu2mgP0XOwrL11yACWKIad ksWA3s4feBrlFFXCX/pGlg==
=E751
-----END PGP SIGNATURE-----

So turn on the features to conrol it. Set rlimits on the xfs server and enable non overcommit (-ac kernel)

Alan

Hi Alan,
(cc BugTraq and LKML)

I am using SGI's XFS, and I think they follow Marcelo's kernels for the 2.4 series, at the moment. Are there any plans of getting non overcommit from your tree into Marcelo's?

TIA. :)

--> Jijo

Federico Sevilla III   :  <http://jijo.free.net.ph/>
Network Administrator  :  The Leather Collection, Inc.
GnuPG Key ID           :  0x93B746BE

SGI's XFS != xfs server. SGI XFS == journalling filesystem. xfs server == font server for the X windowing system.

Argh. And we get hit by similarly-named projects. :(

To clarify: I wanted to know if there were plans of getting non overcommit from the AC tree into Marcelo's mainline 2.4 tree, because I use SGI's XFS (journalling filesystem), and thus use the -xfs kernel tree, which follows -marcelo and not -ac.

Apologies.

--> Jijo

Federico Sevilla III   :  <http://jijo.free.net.ph/>
Network Administrator  :  The Leather Collection, Inc.
GnuPG Key ID           :  0x93B746BE

The protection that you need is provided by "ulimit" on most Unixes. There are facilities to limit maximum real memory used, maximum virtual memory, maximum number of processes, etc. This specific bug in XFree is one of a general case of inescapable user process bugs. It resulted in an almost infinite size malloc() request. You can acheive the same effect in any userspace program by just putting malloc() inside an infinite loop.

If you allow users to run with unlimited memory permission, you are vulnerable. The XFree bug will hit more people than usual because it is common to put the ulimit on regular user logins and forget to place a limit on the automatically started processes. The default configuration from RedHat, SuSE, and others is to start XFree outside the login system. You can also place limits on these processes but you need to examine the startup scripts to install the limits in the right places.

This would then result in a different DoS. Whenever XFree hits the memory limit, the malloc's will fail, and XFree will decide what to do about it. Depending on the circumstances, XFree may shut down, thus killing all the X window dependent processes.

R Horn

On Thursday 13 June 2002 12:26, [email protected] wrote:

Exactly, any user without limits can arbitrarily "fork bomb" a system too. A shell script and newbie level programming talent is all you need... That whole class of DoS are hard to stop. You can do 100 things, like starve a system of file handles, open 50,000 listen ports, whatever. You can set limits, but there are not even standard APIs for limiting every conceivable exhaustible resource systems allocate.

It has always puzzled me that a process using lots of memory can bring down an entire (otherwise relatively idle) server to the extent that one cannot even get mingetty on a local terminal to respond to keypresses. I can confirm that the described situation is not just a one-off.

It is my experience that a single process using large amounts of memory causes the system to start swapping. Once it starts swapping, every process that does anything (apart from indefinite wait) goes into "I'm ready to do some processing, but my memory is swapped out" state, and the whole system collapses.

I don't know if Linux prioritises swap-ins as well as CPU time, but perhaps this would be a good way of stabilising this circumstance. One could arrange the dynamic process priorities so that the bad neighbour (read: exploited xfs-daemon) gets less than its fair share of physical RAM, therefore letting the rest of the system live. Perhaps one could limit the number of processes that are having their swap-in serviced at any one time, and make the ones that are being serviced the ones with the highest dynamic priority.

I don't know what exactly is causing the problem, but it is possible that pages are being paged in, the process runs on them for a quantum, and then paged out. In this case, it might help to dictate a minimum time a page can be "in", measured in the amount of work the owning process manages to do.

My other quibble is that the out-of-memory killer seems to choose processes fairly randomly. Ideally, it should kill the process that is causing the problem - which in my experience is always the biggest process in the system, or the process which has grown the fastest recently. However, it should probably be positively discouraged from killing things like apache (where most of the process space is shared), or gettys (which are firstly small, and secondly likely to be respawned by someone at the first opportunity).

I think that if you had not disabled ctrl-alt-delete, it wouldn't have done much good anyway, since usually that key combination just tells init to spawn a shutdown process, which has just as much chance of getting resources as any other process in the system.

I am in complete agreement with both points, but particularly that the kernel should be able to cope with the situation gracefully. I know one can set limits on processes, but the kernel should still be able to cope if we don't.

My suggestion would be to set a maximum core size for the xfs-daemon process. Given your setup, something like 400MB seems appropriate - maybe a little lower. Details for doing this seem to differ from linux to linux. Having done that, I would make sure xfs respawns when it dies - that way someone can't just DOS your whole network by asking for a large font. Finally, wait for the xfs developers to put in a font size limit, or patch the source yourself.

Apart from that, I wouldn't move xfs to a bigger server unless you have already had people complaining about its performance. Moving it to a bigger system just changes the constant in the equation - the attacker would only need to specify a 100000 point font instead of a 50000 point font to bring the system down.

I doubt any of my kernel suggestions have not already been thought about, but it was worth a try. Please can this problem be fixed soon?

Matthew

"If I wanted to kill a battleship, I'd use a s?!tload of Harpoons."
"NT is a lot cheaper."  -- Paul Tomblin & Petro

Well, with crrent versions you have to have 2 or 3 of them to realy make a server trtash. Something like:

tail /dev/zero

recovers with the oom killer

tail /dev/zero & tail /dev/zero & tail /dev/zero

will most likely trash for minutes.

Some kind of penalty could be introduced, because this is better to set up than ulimit. On a system with large amounts of users you simply cant set the ulimit of a single user to mainram/concurrent-users or mainram/concurrent-evil-users.

Greetings
Bernd

It really is nothing new. (Besides of XFree86's Type1 engine calling abort() when it does not like to do something ..)
...

ANY very big program with active working set larger than memory size has problems at Linux / Linux has problems handling it. Indeed the problem is _not_ new, nor trivial to solve efficiently. The ultimate situation is called "trashing", where a program to proceed a page of code needs to be moved into memory, and to make room for that, some other program page must be moved out.. What makes the issue more difficult is that the memory is used for lots and lots of different kinds of buffers and caches as well, and playing a balancing act on them all is quite difficult.

This is recurring topic at linux-kernel list, and has its own list called linux-mm (not at vger, though.)

Others may have different views on the issue, but I think that:

     2.0:        fairly tolerable OOM/trashing behaviour
     2.2:        got worse
     2.4.early:  rather terrible
     2.4.late:   improved somewhat, roughly in par with 2.2

...

/Matti Aarnio