Linux 2.4 usage statistics
Hi all,
I would be very interested in getting feedback about kernel 2.4 usage. Please, please, I do not want this thread to degenerate into a useless flamewar or "mine is bigger" discussion. For this reason, I would like that participants reply privately to me so that this mail does not turn into a thread.
I know 2.4 is still used in a lot of sensible areas, although far less than 2.6 nowadays. At least I know that several appliance makers/sellers still use it in new products, and even very old versions sometimes. What I'd like to estimate is the way it is used, if it is systematically patched with local add-ons, frequency of updates (if any), etc... so that I can adapt my process if needed. For instance, I don't release any -rc for stable releases because I'm convinced that nobody will ever test them, but I have no problem being proved wrong.
If I get enough replies to build up a statistic synthesis, I'll try to summarise it (eventhough 73% of statistics are all made up :-) ).
This "poll" is not meant to dictate what will get merged next (we're in feature freeze anyway), but it helps me know your usage better, to try to serve you better. Also, if a majority of people report the same problem or lacking feature for which a trivial fix is know, it can be studied.
Basically what I'd like to know is the following. You can use the proposed responses as a guide, but they're not mandatory. Also if possible and adapted, please report number of concerned units :
1) where do you use it ?
- PC turned into network equipment (router, LB, BGP route reflector, ...)
- security equipment (firewall, vpn, ids/ips, traffic analyzer, ...)
- proxy services (proxy, smtp relay, reverse-proxy, anti-virus/spam, ...)
- storage/directory server (NFS, LDAP, logging, ...)
- multi-function server (proxy/fw/nfs/mail/...)
- monitoring / remote access
- desktop/workstation
- laptop (if recent, what model ?)
- dedicated appliances (that you may be designing/selling)
- soho embedded systems (eg: wifi routers, ...)
- PDA ?
- other ?
2) how critical are the systems it runs on ?
- medical (people's health depend on them) ?
- finance (massive amounts of money pass through them) ?
- business-critical (you may go bankrupt if it fails too often) ?
- mission-critical (you may loose your job if it fails too often) ?
- security-critical (security gets degraded when it fails) ?
- users get angry at you when it fails (admin, or do this for a living) ?
- not that important (you may reboot when you decide to do so) ?
- no importance at all (eg: home usage, experimentation, ...) ?
3) how many users are you serving with it approximately, all systems included ? Or better, how many people will notice the failure at once if any ? Note: do not count web population as users, but use maximum concurrent users instead.
- < 10
- < 100
- < 1000
- < 10000
- more ?
4) what version are you running ?
- latest or close to that (eg: 2.4.36.x)
- latest was available last time you upgraded
- somewhat old mainline (2.4.32-2.4.34)
- quite old mainline (2.4.21..2.4.31)
- very old mainline (2.4.2, 2.4.17, ...)
- debian's 2.4.27
- rhel 3's 2.4.21
- my other standard distros's 2.4 kernel
- an embedded distro's heavily patched 2.4 (port to other archs, ...)
5) have you applied any patches ?
- provided with the distro
- security: pax/grsec/ow, capabilities, ...
- performance: epoll, variable_hz, jiffies_64, preempt, O(1), ...
- network: tcp md5, transparent proxy, netfilter tcp_window_tracking...
- drivers: very specific to your platform, self-added PCI-IDs, vendors'
- filesystems: updated jffs2+mtd, squashfs, cifs, fuse, ...
- security/stability fixes backported from later 2.4 versions
- other security/stability fixes or improvements
- drivers backported from 2.6
- other ?
6) how often do you update (may translate into avg uptime) ?
- every release
- every major release after a few dot-versions
- only when you see an important (to you) security fix
- same as well as with reliability fixes
- at least once a year
- only if you need to re-install
- after a lot of careful revalidation
- never
7) how do you test ?
- pre-production environment in equivalent conditions
- long-term reliability stress-tests (days to weeks)
- quick regression testing before production
- you test in production
- no test at all, blind upgrade
- other ?
8) why have you not upgraded to 2.6 yet (including Adrian's 2.6.16.X ?)
- not needed, all features OK and hardware still 100% compatible
- you have to change user-space tools
- you don't feel comfortable with 2.6's patch pace (but 2.6.16 is quite comparable to 2.4)
- you do not trust 2.6 enough yet ? (why ?)
- missing features you had in 2.4 mainline that are not in 2.6 anymore ?
- missing features you had in your 2.4 patches that do not exist in 2.6 ?
- drivers not existing anymore in 2.6 ?
- drivers not working anymore in 2.6 ?
- performance regression in 2.6 ?
- you find 2.6 more complicated ?
- other ?
9) when do you plan to switch to 2.6 ?
- it has already begun (and as a supplement, if you had to do so because 2.4 did not work anymore for you for other reasons than hardware support and features, please indicate which ones)
- very soon (<1 year)
- not before 1 year
- next installations
- next batch of major upgrades
- when your hardware is not compatible anymore
- when you can't build 2.4 with your distro's gcc
- when 2.6 supports feature/driver XXX
- when 2.6 is more reliable (example ?)
- when 2.6 development slows down
- the latest possible (why ???)
- never (why ??? and what would you switch to then ?)
10) any comments or suggestions about current release process, minor issues that you are facing each time such as external patches that do not apply, recurrent problems on all of your setups, etc... ?
Well, that's all. You do not need to respond precisely, and once again, I'm just trying to get the figure.
And please do not forget! Reply to me directly, do not pollute the list, that way you will refrain the random geeks from comparing their setups to other ones.
Thanks!
Willy
Just a note that some "how many" and "which version exactly" data is available from [1] which is generated from data users send through a cron job with data from nearly 5000 machines reported during the last 60 days.
When clicking on a version (e.g. "2.4.27") you can also see which of them are distribution kernels.
Kinda shocking that 14% of these machines are not running kernel 2.6 ...
Considering how this data is generated it obviously shows only part of the picture, and your survey will hopefully bring data what we could do for making kernel 2.6 more attractive (not meant against your work, but we should aim at bringing users to 2.6).
cu
Adrian
[1] http://counter.li.org/reports/systemstats.php
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedHi Adrian,
Quoted 4 lines · Adrian Bunk
Just a note that some "how many" and "which version exactly" data is available from [1] which is generated from data users send through a cron job with data from nearly 5000 machines reported during the last 60 days.
I've already seen this one sometime ago, but realized that you will never get stats from sensible systems there. For instance, none of my customers running linux in production would ever accept to permit an HTTP communication between any of their servers with another one on the need, and especially when it comes to reporting stats about *their* versions.
Also, I recently realized that several high-grade commercial products still ship with 2.4 in it. I even know about one which never said it used Linux, running on MVL 2.4.2 :-)
Of course those ones do not care a dime about recent versions. But it looks like the falling curve has reached a stabilization point, because such products would have had hundreds of opportunities to upgrade. Reason why I'm wondering and asking them directly.
Quoted 9 lines
When clicking on a version (e.g. "2.4.27") you can also see which of them are distribution kernels.
Kinda shocking that 14% of these machines are not running kernel 2.6 ...
Considering how this data is generated it obviously shows only part of the picture, and your survey will hopefully bring data what we could do for making kernel 2.6 more attractive (not meant against your work, but we should aim at bringing users to 2.6).
100% agreed, and it's the orientation of the survey. At least 2.6.16 could be a first step for those who need high code stability.
Speaking for my case, at Exosec we still use 2.4 a lot. Main reason is that we are used to apply a lot of patches. And maintaining a kernel which does nearly not change in 6 months is really a joy. I have already thought about moving to 2.6.16, but I would have had to port my patches, and was not satisfied by the crapp^Wold scheduler which caused real performance issues for my workload. Since I would have gained nothing in this operation, it was easier to stick to 2.4.
I'm waiting for other people's excuses now :-)
I'm really tempted by making a new attempt with 2.6.25, but let's let it settle down first.
Quoted 4 lines
cu
Adrian
Cheers,
Willy
Quoted 7 lines · Willy Tarreau
Speaking for my case, at Exosec we still use 2.4 a lot. Main reason is that we are used to apply a lot of patches. [...] I have already thought about moving to 2.6.16, but I would have had to port my patches, [...] Since I would have gained nothing in this operation, it
was easier to stick to 2.4.I'm waiting for other people's excuses now :-)
A variation of the theme, which I as maintainer of some drivers in 2.6 heard by way of occasional questions on subsystem mailinglists or off-list: Kernel customizations for embedded systems in combination with a custom userland. Moving all this software over to 2.6 was not considered worth the effort.
Another reason which I read here and there and don't know how real the issue is: 2.6's minimum possible footprint is said to be unable to compete with 2.4 for really resource restricted applications.
Could it actually be that 2.4 outnumbers 2.6 in embedded installations?
(I don't work with 2.4 myself and I don't know the domain of embedded systems.)
Signature · 3 lines
Stefan Richter -=====-==--- -=-- ===-= http://arcgraph.de/sr/
Quoted 5 lines · Stefan Richter
Another reason which I read here and there and don't know how real the
issue is: 2.6's minimum possible footprint is said to be unable to compete with 2.4 for really resource restricted applications.Could it actually be that 2.4 outnumbers 2.6 in embedded installations?
Possibly. However many embedded applications have real time constraints so are using 2.6 these days. Both vanilla and -rt patched 2.6 beat the pants off 2.4 in this area.
Lee
Hi Willy,
Quoted 2 lines · Willy Tarreau
I'm really tempted by making a new attempt with 2.6.25, but let's let it settle down first.
You might want to wait a bit more and pick a version which will end up in either SLES or RHEL. These tend to be maintained for longer than the other versions. For example SLES10 is based on kernel 2.6.16, which is still maintained by Adrian (or at least was maintained for a long time
- I don't know what the current status is.)
Signature · 1 line
Jean Delvare
Linux 2.4 usage statistics (results)
In reply to Willy TarreauHi all,
I've tried to summarize the usage reports I got for kernel 2.4.
[ Note: some of the responders asked to be notified when I send the
results, so I've Bcc'd the responders so that they can get the
results without being polluted by potential responses. ]
About 22 useful responses. It is not possible to report accurate numbers because most participants did not themselves have accurate numbers. I would say that based on the responses (and a few cases I know), 2.4 usage approximately follows this distribution :
- old recycled laptops at home, or PDA/thin clients - ~5%
They rely on whatever kernel managed to boot on them, then they never got updated anymore. Security generally not too much of an issue, no plan to update to newer 2.4, let alone 2.6. The hardware will die first. Sometimes a few patches were added to support one particular component or enable one feature specific to the use case. These boxes have the lowest level of criticity.
- desktop PCs, monitoring stations - about the same as previous ones, with an increased level of criticity. Generally users have not upgraded simply because "it works". Those are systems with known fixed hardware and usage pattern. Most often they serve as X11 terminals and/or browsers, and do not require many updates. They cause trouble to the user(s) when they die and no benefit is expected from an upgrade to 2.6. Future installation (same or replaced hardware) will generally be on 2.6.
- general purpose servers : regularly updated - about 50%. The pattern is most always the same. A lot of services are installed, among which WWW, Mail, DNS, Samba, NFS. The server never experiences any outage, and a moderate amount of people would be affected if a problem happened. Almost always up to date with latest 2.4. Reason for not upgrading to 2.6 generally is lack of need and time, as well as risks of regressions which would get a lot of users angry. Users per server are in the range of 10 to 1000. Generally some migration tests are in progress. Early attempts at 2.6 failed due to stability problems (GRE tunnels freezing, sparc images limited in size).
- application specific servers : about 20%. Often mission-critical. Generally deployed with latest 2.4, and almost never updated. Long stress-tests before production. Generally the number of users does not mean anything, it's the process the system is involved in which makes sense. Pre-press workflow applications for daily newspapers and weather broadcasting for TV channels were reported. Reliability is the #1 requirement, and generally recent tests in 2.6 have not been much satisfying. One reported DRBD 0.6 very reliable in 2.4, while DRBD 0.7 not as much in 2.6. Another one reported that the application needed to be ported to correctly work in 2.6. Interestingly, in all cases, the high level of criticity implies that 2.6 is still being evaluated, in the event that one day there is no choice (eg: due to hardware compatibility).
- routers/firewalls/VPN/IDS : about 10%. Longest uptimes up to 5-600 days in business environments, implying kernel not often updated (avg once a year). For personal use, it's where the updates are the most common (probably due to the quick reboot and low impact of a short outage). One user reported a number of these boxes deployed in finance/business/train environments with high level of criticity, receiving occasional updates during dead hours. Generally no upgrade to 2.6 is planned (though I think it is where it might be the least painful). One user reported missing devfs in 2.6, too big kernel size, and no clear upgrade path. We might help them switch by enumerating the small number of changes in such a networked environment. What may be the most obscure in business environment is the lack of sight of long term reliability. I personally know about several firewalls I have deployed 5 years ago in a finance environment which see 1 million unique clients a day. It has been discussed about migrating them to 2.6 but it seems like the only way to know how long they will survive is to test in production, which is generally not acceptable. So leaving them on 2.4 is often the solution.
- embedded systems : about 10% of the reports, probably more in terms of number of systems. They often expect very long uptimes (two of them reported rebooting less than once a year). This is either dictated by the fact that consumer electronics get bad reputation when failures or updates happen too often, or because the big customer does not want to update something which works. Reported usages include outdoor information displays for trains and busses, various consumer electronic devices including TV sets, monitoring systems and building security devices (eg door openers). Upgrade to 2.6 not needed nor planned at all for some, and planned for others to gain better hardware support. Some are still testing but experiencing regressions (eg: parport). In all cases, the build process has to be massively redesigned and this costs a lot of time and money. Not surprizing that some major network equipement manufacturer still ships device with and old 2.4.2 in them.
It is worth noting that several users still using 2.4 roll their own distro, which will need to be substantially updated to support 2.6. This seems to be one showstopper too, especially when the distro was designed for easy remote and centralized upgrades. In this case, what seems to happen is that the old version continues to live at existing locations, and a new version appears with support for 2.6 for newer systems. But from an organisational point of view, it's not always easy to manage two branches of a distro when only one has been enough for years.
Surprizingly, it appears that very few users patch their kernels. Those who do so have their own distros or build kernels for large amounts of machines (eg: consultants doing so for their customers). Many of the reported patches have found their way in mainline 2.6, but some have definitely vanished (eg:
devfs,linux-abi, kstat, umsdos, lvm). Among the patches, GRsecurity and PaX were reported several times. Some special-purpose patches are included in appliance products.
Vendor drivers are the only ones installed by most users who patch (eg:
3Ware 9xxx, sk98lin, bnx2). One user reported an annoying problem with USB for which I realized I've been having a patch in my own tree for years, to the point I completely forgot the problem existed. It is related to USB-storage, where unplugging then replugging a device assigns it a new SCSI drive name. I may merge it into 2.4.37.
Based on that and on the workflow people took the time to explain, I realize that the distinction between -pre and -rc is useless (ding! Linus if you read this, don't beat me). In fact, either people want absolute reliability and they pick one kernel from the stable 2.4.X.Y branch, or they want more recent updates and they simply do their shopping in the -master branch, which is fairly easy thanks to the Gitweb interface. But there are almost no testers in 2.4, just users. That means that I don't have to expect immediate feedback when posting a pre-release. And it has happened several times that I got a build error report several weeks after the release.
Also, since most people do not update more than 1/2 times a year, it's not very useful to have more than 1/2 new versions a year, expecially since we have the stable release. For this reason, I think I will issue stable releases a bit more often for users to quickly get their fixes, but progressively increase the delay between major releases. Those ones will only be issued with new PCI IDs, major driver updates, compiler support, etc...
Last, I think users would benefit from an inventory of functional changes between 2.4 and recent 2.6, with a reversible upgrade path, so that they can experiment with 2.6 on their production machine and immediately go back to 2.4 in case of trouble.
Well, I hope it was not too much boring!
Regards,
Willy