Skip to contentSkip to conversation

Topic: Removal of code that is still in use by users but there is a better code.

Ksummit-discuss Konrad Rzeszutek Wilk 14 participants 20 messages Jun 10, 2014 – Jun 19, 2014

lore
Share & tools
lore
#0 lore raw

Hey,

I would want to propose a topic on removing code in Linux that users are using - but they are doing it less and less and it mostly is tied in with older hardware. Specifically how to do this transition properly - and if we want to define some checklist /policy to do it via.

Specifically in the Xen virtualization world there is in pipeline code that is going to obsolete some of the existing pvops code - and also make lguest obsolete. It makes the Linux kernel run faster, less code to deal with, makes x86 folks happy, and requires newer hardware. This is known as PVH (ParaVirtualization Hardware).

So, from one hand - with newer hardware - we can remove some of the code. On the other hand - with older hardware (pre EPT/NPT capable) or low-powered ones - we would making their life difficult and slower (as now the hypervisor has to do the emulation, probably has some bugs, code bitrotten, etc).

In essence it boils down to removing code in X years that users do use.

There is a nice migration path, but I am sure the moment we rip out the code folks will come out of the woodwork, chase us down, and hit us with clubs. I do enjoy hiking and don't want to have to look behind my back as I am hiking.

What I want to propose is a topic to discuss what is the right way to do this? I presume other platforms have had similar issues in the past (or will be) and what is the best way of doing this. Are there any policies in place?

P.S.
I say in pipeline because it is still experimental, the ABI hasn't been bolted down, and we have tons of outstanding bugs before we let enterprise customers take a stab at it, etc.

P.S.S.
Peter (hpa) is going to hate that I put 'X' instead of the '5' number. We figured that in 5 years since we get this stable we can start the count-down timer - but the 'getting' stable seems to take longer than I am happy with (#@)@#$ bugs). Hence, X = stable + 5. If there are any

#1 lore raw
Quoted 8 lines · Konrad Rzeszutek Wilk

Hey,

I would want to propose a topic on removing code in Linux that users are using - but they are doing it less and less and it mostly is tied in with older hardware. Specifically how to do this transition properly - and if we want to define some checklist /policy to do it via.

I second this. Right now deprecation is entirely ad hoc... usually in the form "this hasn't compiled for X releases and noone noticed", which makes it hard to do *controlled* deprecation...

I realize this is a third rail kind of topic, but it puts us in really hard spots as a project, sometime.

-hpa

#2 lore raw
Quoted 10 lines · H. Peter Anvin
Quoted 8 lines · Konrad Rzeszutek Wilk

Hey,

I would want to propose a topic on removing code in Linux that users are using - but they are doing it less and less and it mostly is tied in with older hardware. Specifically how to do this transition properly - and if we want to define some checklist /policy to do it via.

Quoted 6 lines

I second this. Right now deprecation is entirely ad hoc... usually in the form "this hasn't compiled for X releases and noone noticed", which makes it hard to do *controlled* deprecation...

I realize this is a third rail kind of topic, but it puts us in really hard spots as a project, sometime.

I agree. I think we've had some recent examples of how to do this better, with remap_file_pages: old userspace has to *work* on new kernels, but it doesn't necessarily need to run fast, or integrate with any new features. We should deprecate more kernel bits in favor of minimal compatibility layers, some of which we can associate with old userspace and drop for new userspace.

- Josh Triplett

#3 lore raw
Quoted 8 lines · On 06/10/2014 05:36 PM, [email protected] wrote:

I agree. I think we've had some recent examples of how to do this better, with remap_file_pages: old userspace has to *work* on new kernels, but it doesn't necessarily need to run fast, or integrate with any new features. We should deprecate more kernel bits in favor of minimal compatibility layers, some of which we can associate with old userspace and drop for new userspace.

Uhm... that applies only to a fairly restricted aspect of deprecation. Much more is tied to specific hardware.

-hpa

#5 lore raw
In reply to H. Peter Anvin
Quoted 10 lines · H. Peter Anvin
Quoted 8 lines · Konrad Rzeszutek Wilk

Hey,

I would want to propose a topic on removing code in Linux that users are using - but they are doing it less and less and it mostly is tied in with older hardware. Specifically how to do this transition properly - and if we want to define some checklist /policy to do it via.

Quoted 4 lines

I second this. Right now deprecation is entirely ad hoc... usually in the form "this hasn't compiled for X releases and noone noticed", which makes it hard to do *controlled* deprecation...

That is a bit different to "users are using but less and less". Personally I would not mind leaving code in the kernel as long as it is used, but it would be great if we had some rule that a file/driver which did not compile for X releases (pick a preferred number for X - two years worth of releases ?) can be removed.

Guenter

#6 lore raw
Quoted 15 lines · Guenter Roeck
Quoted 10 lines · H. Peter Anvin
Quoted 8 lines · Konrad Rzeszutek Wilk

Hey,

I would want to propose a topic on removing code in Linux that users are using - but they are doing it less and less and it mostly is tied in with older hardware. Specifically how to do this transition properly - and if we want to define some checklist /policy to do it via.

Quoted 4 lines

I second this. Right now deprecation is entirely ad hoc... usually in the form "this hasn't compiled for X releases and noone noticed", which makes it hard to do *controlled* deprecation...

Quoted 5 lines

That is a bit different to "users are using but less and less". Personally I would not mind leaving code in the kernel as long as it is used, but it would be great if we had some rule that a file/driver which did not compile for X releases (pick a preferred number for X - two years worth of releases ?) can be removed.

I suspect most people would agree with the idea that something that hasn't compiled for X years can be removed, possibly with some negotiation about what X is.

But my impression is the proposal is to go farther than that, and figure out a way to remove obsolete but still-compilable code. If we all did our jobs perfectly, we would never add a change to Linux that broke compilation of anything. So if there's a file that doesn't compile anymore, I think of that as a result of a mistake somewhere along the line. We can use that mistake to deduce that nobody cares anymore, but it'd be a lot nicer to have a scheme that didn't depend on people making random mistakes.

Bjorn

#4 lore raw
In reply to H. Peter Anvin
Quoted 10 lines · H. Peter Anvin
Quoted 8 lines · On 06/10/2014 05:36 PM, [email protected] wrote:

I agree. I think we've had some recent examples of how to do this better, with remap_file_pages: old userspace has to *work* on new kernels, but it doesn't necessarily need to run fast, or integrate with any new features. We should deprecate more kernel bits in favor of minimal compatibility layers, some of which we can associate with old userspace and drop for new userspace.

Quoted 2 lines

Uhm... that applies only to a fairly restricted aspect of deprecation. Much more is tied to specific hardware.

Agreed; hardware support does not have the same degree of compatibility concerns that userspace interfaces do.

For instance, we could reasonably ask whether we can deprecate support for ancient filesystems and push that code into FUSE in userspace, given that most such filesystems only exist for forensics rather than performance. That would mitigate potential exploits in that code.

- Josh Triplett

#7 lore raw
In reply to Bjorn Helgaas

Hi Bjorn,

Quoted 7 lines · On Wed, 11 Jun 2014 13:43:18 -0600 Bjorn Helgaas <[email protected]> wrote:

all did our jobs perfectly, we would never add a change to Linux that broke compilation of anything. So if there's a file that doesn't compile anymore, I think of that as a result of a mistake somewhere along the line. We can use that mistake to deduce that nobody cares anymore, but it'd be a lot nicer to have a scheme that didn't depend on people making random mistakes.

We could start making nonrandom mistakes? ;-)

Signature · 2 lines
Cheers,
Stephen Rothwell                    [email protected]
#8 lore raw
Quoted 4 lines · Stephen Rothwell

Hi Bjorn,

Quoted 1 line · On Wed, 11 Jun 2014 13:43:18 -0600 Bjorn Helgaas <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines

all did our jobs perfectly, we would never add a change to Linux that broke compilation of anything. So if there's a file that doesn't compile anymore, I think of that as a result of a mistake somewhere along the line. We can use that mistake to deduce that nobody cares anymore, but it'd be a lot nicer to have a scheme that didn't depend on people making random mistakes.

We could start making nonrandom mistakes? ;-)

How about adding CONFIG_DELETED_THINGS. If you enable it, you get a message asking you to speak up if you actually need anything in there. CONFIG_DELETED_THINGS would default to n, and the new (and temporary) things under it would also default to n.

Think feature-removal-schedule, but with teeth.

--Andy

#9 lore raw
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Stephen Rothwell

Hi Bjorn,

Quoted 7 lines · On Wed, 11 Jun 2014 13:43:18 -0600 Bjorn Helgaas <[email protected]> wrote:

all did our jobs perfectly, we would never add a change to Linux that broke compilation of anything. So if there's a file that doesn't compile anymore, I think of that as a result of a mistake somewhere along the line. We can use that mistake to deduce that nobody cares anymore, but it'd be a lot nicer to have a scheme that didn't depend on people making random mistakes.

Quoted 8 lines
Quoted 1 line

We could start making nonrandom mistakes? ;-)

How about adding CONFIG_DELETED_THINGS. If you enable it, you get a message asking you to speak up if you actually need anything in there. CONFIG_DELETED_THINGS would default to n, and the new (and temporary) things under it would also default to n.

Think feature-removal-schedule, but with teeth.

This would eventually become like CONFIG_EXPERIMENTAL before somebody put it out of its misery: a pointless thing which everybody enables.

Could we just step back and ask what the burning need to do this (at least for drivers; I understand the ABI deprecation headache) is? Most driver code for obsolete things harmlessly compiles; why bother trying to hunt them down and shoot them when they're not really causing offence?

James

#10 lore raw
Quoted 2 lines · James Bottomley

This would eventually become like CONFIG_EXPERIMENTAL before somebody put it out of its misery: a pointless thing which everybody enables.

Probably so.

Quoted 5 lines

Could we just step back and ask what the burning need to do this (at least for drivers; I understand the ABI deprecation headache) is? Most driver code for obsolete things harmlessly compiles; why bother trying to hunt them down and shoot them when they're not really causing offence?

Every time a developer wants to change a core API, that developer needs to patch every driver that uses the API. Every old, unused, bitrotten driver we have in the tree is 100% wasted work, and often a substantial amount of work because it's really hard even to understand how those drivers are (mis)using the API being changed.

- R.

#11 lore raw
Quoted 13 lines · Roland Dreier
Quoted 2 lines · James Bottomley

This would eventually become like CONFIG_EXPERIMENTAL before somebody put it out of its misery: a pointless thing which everybody enables.

Probably so.

Quoted 5 lines

Could we just step back and ask what the burning need to do this (at least for drivers; I understand the ABI deprecation headache) is? Most driver code for obsolete things harmlessly compiles; why bother trying to hunt them down and shoot them when they're not really causing offence?

Quoted 5 lines

Every time a developer wants to change a core API, that developer needs to patch every driver that uses the API. Every old, unused, bitrotten driver we have in the tree is 100% wasted work, and often a substantial amount of work because it's really hard even to understand how those drivers are (mis)using the API being changed.

Well how often do we do that? It's not like it's the most common activity.

Even when we do it, mostly the changes are mechanical and no-one really notices. On the odd occasion we have to change the call properties of the API (usually because of locking or layering problems) it forces the question of whether the driver is in use or not. Mostly you don't change the API and put the driver under CONFIG_BROKEN to see if anyone turns up to claim it.

In fact, we could document the above and call it our driver deprecation process. I've done this occasionally in SCSI, but the fact it's so rare tends to make me think we don't really have that much of a problem.

James

#16 lore raw
In reply to Bjorn Helgaas
Quoted 26 lines · Bjorn Helgaas
Quoted 15 lines · Guenter Roeck
Quoted 10 lines · H. Peter Anvin
Quoted 8 lines · Konrad Rzeszutek Wilk

Hey,

I would want to propose a topic on removing code in Linux that users are using - but they are doing it less and less and it mostly is tied in with older hardware. Specifically how to do this transition properly - and if we want to define some checklist /policy to do it via.

Quoted 4 lines

I second this. Right now deprecation is entirely ad hoc... usually in the form "this hasn't compiled for X releases and noone noticed", which makes it hard to do *controlled* deprecation...

Quoted 5 lines

That is a bit different to "users are using but less and less". Personally I would not mind leaving code in the kernel as long as it is used, but it would be great if we had some rule that a file/driver which did not compile for X releases (pick a preferred number for X - two years worth of releases ?) can be removed.

I suspect most people would agree with the idea that something that hasn't compiled for X years can be removed, possibly with some negotiation about what X is.

Quoted 9 lines

But my impression is the proposal is to go farther than that, and figure out a way to remove obsolete but still-compilable code. If we all did our jobs perfectly, we would never add a change to Linux that broke compilation of anything. So if there's a file that doesn't compile anymore, I think of that as a result of a mistake somewhere along the line. We can use that mistake to deduce that nobody cares anymore, but it'd be a lot nicer to have a scheme that didn't depend on people making random mistakes.

I understand, but personally I don't have much problem with code as long as it compiles. I am more concerned with code that doesn't compile and no one cared for years.

Sure, there is a difference if that code actually consumes cycles, but I suspect that isn't likely enough to be bothered about. If there is such code, I would agree that it should be removed even if it does compile.

Guenter

#19 lore raw
In reply to Guenter Roeck
Quoted 5 lines · Guenter Roeck

That is a bit different to "users are using but less and less". Personally I would not mind leaving code in the kernel as long as it is used, but it would be great if we had some rule that a file/driver which did not compile for X releases (pick a preferred number for X - two years worth of releases ?) can be removed.

This is the problem, though: the code can easily compile well past the point of anyone using it, because well-meaning maintainers is keeping it from falling apart completely, often at substantial cost.

-hpa

#17 lore raw
In reply to Guenter Roeck
Quoted 4 lines · On Wed, 11 Jun 2014 16:22:16 -0700 Guenter Roeck <[email protected]> wrote:

I understand, but personally I don't have much problem with code as long as it compiles. I am more concerned with code that doesn't compile and no one cared for years.

Matters what your definition of "doesn't compile" is.

You may not have the right config option set to make it compile. randconfig builds trigger failed compiles quite often. Is that code worth removing?

As most people test allyesconfig and allmodconfig quite a bit, I'm not sure what code you are talking about. Basically everything that gets enabled does compile.

Of course, if you set something to CONFIG_BROKEN and no one complains for a few years, that's worth nuking.

-- Steve

#18 lore raw
Quoted 5 lines · Steven Rostedt
Quoted 3 lines · On Wed, 11 Jun 2014 16:22:16 -0700 Guenter Roeck <[email protected]> wrote:

I understand, but personally I don't have much problem with code as long as it compiles. I am more concerned with code that doesn't compile and no one cared for years.

And once it stops compiling, more issues will be introduced into that area, which no one notices due to the first compile failure.

Quoted 9 lines

Matters what your definition of "doesn't compile" is.

You may not have the right config option set to make it compile. randconfig builds trigger failed compiles quite often. Is that code worth removing?

As most people test allyesconfig and allmodconfig quite a bit, I'm not sure what code you are talking about. Basically everything that gets enabled does compile.

And once it compiles, it usually works!

I have the impression we're getting good (read: having good review) at doing logical changes, i.e. when code is integrated, it works (almost always :-). The tricky parts these days are configuration issues, i.e. code that fails to build for certain configurations, due to various reasons (forgot to handle a case in another #ifdef branch, code inside vs. outside #ifdef, different indirect
includes on different architectures, ...). Fortunately the latter can be caught using more CPU resources, in contrast with more reviewers for the former.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#12 lore raw
In reply to James Bottomley
Quoted 30 lines · James Bottomley
Quoted 13 lines · Roland Dreier
Quoted 2 lines · James Bottomley

This would eventually become like CONFIG_EXPERIMENTAL before somebody put it out of its misery: a pointless thing which everybody enables.

Probably so.

Quoted 5 lines

Could we just step back and ask what the burning need to do this (at least for drivers; I understand the ABI deprecation headache) is? Most driver code for obsolete things harmlessly compiles; why bother trying to hunt them down and shoot them when they're not really causing offence?

Quoted 5 lines

Every time a developer wants to change a core API, that developer needs to patch every driver that uses the API. Every old, unused, bitrotten driver we have in the tree is 100% wasted work, and often a substantial amount of work because it's really hard even to understand how those drivers are (mis)using the API being changed.

Well how often do we do that? It's not like it's the most common activity.

Even when we do it, mostly the changes are mechanical and no-one really notices. On the odd occasion we have to change the call properties of the API (usually because of locking or layering problems) it forces the question of whether the driver is in use or not. Mostly you don't change the API and put the driver under CONFIG_BROKEN to see if anyone turns up to claim it.

Quoted 3 lines

In fact, we could document the above and call it our driver deprecation process. I've done this occasionally in SCSI, but the fact it's so rare tends to make me think we don't really have that much of a problem.

I agree here.

Generally, if a kernel-wide change is being made for good reasons and something gets in the way, it is a fair question to ask if that thing is actually used by someone. If it is not used by anyone, removing it may be regarded as part of making the change.

Rafael

#13 lore raw
In reply to James Bottomley
Quoted 14 lines · James Bottomley
Quoted 13 lines · Roland Dreier
Quoted 2 lines · James Bottomley

This would eventually become like CONFIG_EXPERIMENTAL before somebody put it out of its misery: a pointless thing which everybody enables.

Probably so.

Quoted 5 lines

Could we just step back and ask what the burning need to do this (at least for drivers; I understand the ABI deprecation headache) is? Most driver code for obsolete things harmlessly compiles; why bother trying to hunt them down and shoot them when they're not really causing offence?

Quoted 8 lines
Quoted 5 lines

Every time a developer wants to change a core API, that developer needs to patch every driver that uses the API. Every old, unused, bitrotten driver we have in the tree is 100% wasted work, and often a substantial amount of work because it's really hard even to understand how those drivers are (mis)using the API being changed.

Well how often do we do that? It's not like it's the most common activity.

I think it happens often enough to be a valid concern. There are definitely wireless drivers (some which predate my tenure), that still compile but which I strongly doubt are functional or used. I've pushed some out of the tree from time to time, and I'll probably continue to do so. But in the meantime, it can be difficult to judge what is used or not and what simply doesn't work because of a series of subtle mistakes in patch bombs...

John

Signature · 2 lines
John W. Linville		Someday the world will need a hero, and you
[email protected]			might be all we have.  Be ready.
#14 lore raw
Quoted 24 lines · John W. Linville
Quoted 14 lines · James Bottomley
Quoted 13 lines · Roland Dreier
Quoted 2 lines · James Bottomley

This would eventually become like CONFIG_EXPERIMENTAL before somebody put it out of its misery: a pointless thing which everybody enables.

Probably so.

Quoted 5 lines

Could we just step back and ask what the burning need to do this (at least for drivers; I understand the ABI deprecation headache) is? Most driver code for obsolete things harmlessly compiles; why bother trying to hunt them down and shoot them when they're not really causing offence?

Quoted 8 lines
Quoted 5 lines

Every time a developer wants to change a core API, that developer needs to patch every driver that uses the API. Every old, unused, bitrotten driver we have in the tree is 100% wasted work, and often a substantial amount of work because it's really hard even to understand how those drivers are (mis)using the API being changed.

Well how often do we do that? It's not like it's the most common activity.

Quoted 7 lines

I think it happens often enough to be a valid concern. There are definitely wireless drivers (some which predate my tenure), that still compile but which I strongly doubt are functional or used. I've pushed some out of the tree from time to time, and I'll probably continue to do so. But in the meantime, it can be difficult to judge what is used or not and what simply doesn't work because of a series of subtle mistakes in patch bombs...

So the solution to this is to put the drivers into a freeze maintenance mode where you make no changes but what is absolutely required (and you can decide depending on the extensiveness of the required change whether you'd rather mark the driver as BROKEN than do the API change). It doesn't solve the problem entirely but it means that breakage has to come from outside rather than subtle mistakes in unnecessary patches.

James

#15 lore raw
In reply to James Bottomley
Quoted 10 lines · James Bottomley
Quoted 5 lines · Roland Dreier

Every time a developer wants to change a core API, that developer needs to patch every driver that uses the API. Every old, unused, bitrotten driver we have in the tree is 100% wasted work, and often a substantial amount of work because it's really hard even to understand how those drivers are (mis)using the API being changed.

Well how often do we do that? It's not like it's the most common activity.

Well I'm doing that right now in the GPIO subsystem where I'm refactoring all local irqchip implementations into the gpiolib.

So I attack-patch all suitable refactoring targets, indeed. And sometimes the answer comes back "I don't have that hardware anymore, cannot test it", even from the person listed as maintainer. (drivers/gpio/gpio-adnp.c, drivers/pinctrl/spear/*)

And then I get a strong urge to delete it, because a maintainer that can't maintain is not much of a maintainer...

Yours,
Linus Walleij