Skip to contentSkip to conversation

Unable to continue testing of 2.6.25

DISCUSSION Frans Pop 12 participants 31 messages Feb 17, 2008 – Feb 19, 2008

lore
Share & tools
lore

Frans Pop

author
#0 lore raw

Yesterday, after spending quite a few hours over the last days on bisecting some serious regressions and finding workarounds for them, I thought I could start using 2.6.25-rc2 as the new kernel for my desktop. Unfortunately I found that I cannot because it would make my other main activity - working on the Debian installation system - impossible.

For my work on the Debian Installer I heavily rely on emulators to run test installs and ATM my emulator of choice is VirtualBox (the fully open "ose" version). This requires the vboxdrv kernel module, but unfortunately:
vboxdrv: Unknown symbol change_page_attr

At first I traced this to:
e1271f68
x86: deprecate change_page_attr() for drivers
With the introduction of the new API, no driver or non-archcore code needs to use c-p-a anymore, so this patch also deprecates the EXPORT_SYMBOL of CPA (it's a horrible API after all).
which had:
-EXPORT_SYMBOL(change_page_attr); +EXPORT_UNUSED_SYMBOL(change_page_attr); /* to be removed in 2.6.27 */

Which seemed entirely reasonable but left me wondering about the error I got.

But then I found:
d1028a15
x86: make various pageattr.c functions static change_page_attr_add is only used in pageattr.c now, so we can make this function static.
change_page_attr() isn't used anywere at all anymore; this function is a really bad API anyway so just remove the bloat entirely.

Which removed the entire function (without even properly mentioning it in the shortlog).

OF COURSE it is up to the VirtualBox developers to adjust to the new interface (and based on past experience I expect they will with their next version). And it may very well be that they were totally braindead to use the function in the first place. I don't know and I really don't care. The important fact for me is that I can no longer use a piece of software that is essential to me and thereby lose the motivation to do any work on the kernel.

Lesson of the day: thinking only about in-kernel users of published API when doing restructuring is going to lose you testers who do MORE with their systems than just kernel development.

Please allow external users some decent period for transitioning. The initial plan to "remove the old function in 2.6.27" was entirely sensible. It's a pity it was not followed through.

Thanks,
FJP

P.S. Of course I _will_ follow up on issues that I've already reported. I will just not be doing any new testing.

#1 lore raw
Quoted 13 lines · Frans Pop

...
For my work on the Debian Installer I heavily rely on emulators to run test installs and ATM my emulator of choice is VirtualBox (the fully open "ose" version). This requires the vboxdrv kernel module, but unfortunately:
...
OF COURSE it is up to the VirtualBox developers to adjust to the new interface (and based on past experience I expect they will with their next version). And it may very well be that they were totally braindead to use the function in the first place. I don't know and I really don't care. The important fact for me is that I can no longer use a piece of software that is essential to me and thereby lose the motivation to do any work on the kernel.

Quoted 4 lines

Lesson of the day: thinking only about in-kernel users of published API when doing restructuring is going to lose you testers who do MORE with their systems than just kernel development.
...

I get your problem, but you are looking in the wrong direction for a solution.

The real problem is that the kernel seems to lack functionality you require for doing some work.

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

Quoted 3 lines

Thanks,
FJP
...

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#2 lore raw

Am 17.02.2008 14:16 schrieb Adrian Bunk:

Quoted 4 lines

The real problem is that the kernel seems to lack functionality you require for doing some work.

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

No, that's not the real problem. Even if the kernel didn't lack any required functionality and it could all be done today without VirtualBox, pulling the rug from underneath it like that leaves all those who are currently relying on it without the ability to continue testing newer kernels until they find the time to redesign their working environment. Sticking to the removal schedule could have avoided that.

HTH
T.

Signature · 4 lines
Tilman Schmidt                          E-Mail: [email protected]
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)
#3 lore raw
Quoted 7 lines · Tilman Schmidt

Am 17.02.2008 14:16 schrieb Adrian Bunk:

Quoted 5 lines

The real problem is that the kernel seems to lack functionality you require for doing some work.

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

Quoted 7 lines

No, that's not the real problem. Even if the kernel didn't lack any required functionality and it could all be done today without VirtualBox, pulling the rug from underneath it like that leaves all those who are currently relying on it without the ability to continue testing newer kernels until they find the time to redesign their working environment. Sticking to the removal schedule could have avoided that.

Frans said that he requires VirtualBox for his work on the Debian Installer.

If the kernel would offer everything Frans needs for his work on the Debian Installer he wouldn't have sent his email.

So let's fix the problem (kernel lacks functionality) and not the symptom (external module requires some export).

Quoted 2 lines

HTH
T.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#4 lore raw
Quoted 1 line · Adrian

So let's fix the problem (kernel lacks functionality)

That's the problem as understood by Adrian.

I hear another problem as well ...

Quoted 3 lines · Frans

Please allow external users some decent period for transitioning. The initial plan to "remove the old function in 2.6.27" was entirely sensible. It's a pity it was not followed through.

That seems plain enough to me. It's not just the lack of functionality, but that such lack apparently happened with too little warning.

If this is a fair representation of what happened, then seems to me that we could have left that EXPORT_UNUSED_SYMBOL(change_page_attr) in place a bit longer.

Or at least, if we really did have to make Frans life difficult like this, we could offer more appreciation for how things might look from his perspective.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.940.382.4214
#5 lore raw
Quoted 12 lines · On Sun, 17 Feb 2008 14:38:51 -0600 Paul Jackson <[email protected]> wrote:
Quoted 1 line · Adrian

So let's fix the problem (kernel lacks functionality)

That's the problem as understood by Adrian.

I hear another problem as well ...

Quoted 3 lines · Frans

Please allow external users some decent period for transitioning. The initial plan to "remove the old function in 2.6.27" was entirely sensible. It's a pity it was not followed through.

Quoted 8 lines

That seems plain enough to me. It's not just the lack of functionality, but that such lack apparently happened with too little warning.

If this is a fair representation of what happened, then seems to me that we could have left that EXPORT_UNUSED_SYMBOL(change_page_attr) in place a bit longer.

it's not a fair repersentation. Again.. this export was unkeepable due to the API being nasty and having to be fixed anyway ;(.

One of the problems was that the c-p-a api has to be followed by a cache flush function call. Sadly that does a TOTAL flush of the caches of all cpus in the system. As part of the -rc1 changes, it is now done only on the exact pages that need to be flushed (so you no longer flush 12Mb of caches when you only needed to flush 4Kb), but to achieve that, it was no longer an option to keep this as 2 separate function calls.
Add to this that some very fundemanteal bugs couldn't be fixed without the function underlying cpa changing prototype and behavior, so the function had to go.

Quoted 3 lines

Or at least, if we really did have to make Frans life difficult like this, we could offer more appreciation for how things might look from his perspective.

I understand where he's coming from; at the same time it's a very small change to virtualbox to fix this
and has been done already... in minutes. Frans should take that up with the virtual box support forum, I'm sure they have the patch available there. (it's mostly removing workarounds for cpa bugs and then just calling set_memory_x / set_memory_nx).

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
author
#6 lore raw
Quoted 24 lines · Arjan van de Ven
Quoted 12 lines · On Sun, 17 Feb 2008 14:38:51 -0600 Paul Jackson <[email protected]> wrote:
Quoted 1 line · Adrian

So let's fix the problem (kernel lacks functionality)

That's the problem as understood by Adrian.

I hear another problem as well ...

Quoted 3 lines · Frans

Please allow external users some decent period for transitioning. The initial plan to "remove the old function in 2.6.27" was entirely sensible. It's a pity it was not followed through.

Quoted 7 lines

That seems plain enough to me. It's not just the lack of functionality, but that such lack apparently happened with too little warning.

If this is a fair representation of what happened, then seems to me that we could have left that EXPORT_UNUSED_SYMBOL(change_page_attr) in place a bit longer.

it's not a fair repersentation. Again.. this export was unkeepable due to the API being nasty and having to be fixed anyway ;(.

Quoted 9 lines

One of the problems was that the c-p-a api has to be followed by a cache flush function call. Sadly that does a TOTAL flush of the caches of all cpus in the system. As part of the -rc1 changes, it is now done only on the exact pages that need to be flushed (so you no longer flush 12Mb of caches when you only needed to flush 4Kb), but to achieve that, it was no longer an option to keep this as 2 separate function calls. Add to this that some very fundemanteal bugs couldn't be fixed without the function underlying cpa changing prototype and behavior, so the function had to go.

That's a much better explanation than can be found in the changelog.

The changelog effectively says: this commits makes a few (new) functions static; oh, and by the way, let's delete the old function _because it is unused and ugly_.

Well, I've shown that the first is not true and the second by itself is absolutely not sufficient reason to remove an exported function that has long been part of the kernel.
I would probably have started this thread differently if the removal had been done in a separate commit and had included the explanation you give above.

Quoted 2 lines

I understand where he's coming from; at the same time it's a very small change to virtualbox to fix this and has been done already... in minutes.

The problem here is that it may be a simple change for you and other similarly gifted people, but it's something that's above _my_ skills.

Quoted 1 line

Frans should take that up with the virtual box support forum, I'm sure

I did actually manage to create such a patch for the breakage in the VirtualBox source because of 2.6.24, but those really were very minor issues (basically overlapping definitions). I posted that on their user list (to which I am subscribed), but I never saw any response to it.

I also don't think it is really realistic to expect Innotek to provide patches for unreleased kernel versions. Maybe some other user could provide me with the patch, but I'm not as confident as you are.

My point still is that I feel they should not have to. That it's also the kernel developers responsibility to ensure backwards compatibility or at least a grace period for conversion whenever possible.
And IMO that not only goes for user-space API, but also for interfaces used by out-of-tree kernel modules.

Is it really impossible in this case for example to rewrite the old function so that it becomes a wrapper around the new interface? If it is impossible, then so be it.

Quoted 1 line

they have the patch available there.

I haven't seen anything on the user mailing list yet...

Quoted 2 lines · Arjan van de Ven

I assume you've read the other mails right and went to the virtualbox form to get the small patch to make it work on .25 right?

If you can give me a link to that patch, I'd be grateful. As I said, I'm subscribed to their user list but have not seen any patch there. I've also googled for it, without result.
I've also just quickly checked their "VirtualBox on Linux" forum, but did not see any obvious existing post about it.

Cheers,
FJP

#7 lore raw
In reply to Tilman Schmidt

From: Tilman Schmidt <[email protected]>
Date: Sun, 17 Feb 2008 20:24:56 +0100

Quoted 6 lines

No, that's not the real problem. Even if the kernel didn't lack any required functionality and it could all be done today without VirtualBox, pulling the rug from underneath it like that leaves all those who are currently relying on it without the ability to continue testing newer kernels until they find the time to redesign their working environment.

No, it is VirtualBox's problem. Nobody outside of the kernel should be using that symbol, it can only be used for totally unsupportable things as far as upstream is concerned.

#8 lore raw
Quoted 11 lines · David Miller

From: Tilman Schmidt <[email protected]>
Date: Sun, 17 Feb 2008 20:24:56 +0100

Quoted 6 lines

No, that's not the real problem. Even if the kernel didn't lack any required functionality and it could all be done today without VirtualBox, pulling the rug from underneath it like that leaves all those who are currently relying on it without the ability to continue testing newer kernels until they find the time to redesign their working environment.

No, it is VirtualBox's problem.

Why's that beeping noise coming from my SEP field detector? :-)

Quoted 3 lines

Nobody outside of the kernel
should be using that symbol, it can only be used for totally unsupportable things as far as upstream is concerned.

Then why was it exported in the first place?

Still, we are talking about two completely different types of problem here: (a) the technical problem of why something stopped working and how it should be fixed and (b) the practical problem of users left standing in the rain while such fixing is taking place. While (a) might look like the only real problem to kernel developers, (b) often feels much more real to users. Unfortunately, discussing alternative solutions to (a) does little to solve (b).

The conventional way of addressing (b) is to avoid it altogether by giving sufficient advance warning before removing a feature, and sticking to the feature removal schedule. If (as apparently in this case) that isn't possible, then other ways to mitigate the problem need to be found. That may require switching off some SEP field generators, though. :-)

HTH
T.

Signature · 4 lines
Tilman Schmidt                    E-Mail: [email protected]
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)
#9 lore raw
In reply to David Miller
Quoted 10 lines · David Miller

From: Tilman Schmidt <[email protected]>
Date: Sun, 17 Feb 2008 20:24:56 +0100

Quoted 6 lines

No, that's not the real problem. Even if the kernel didn't lack any required functionality and it could all be done today without VirtualBox, pulling the rug from underneath it like that leaves all those who are currently relying on it without the ability to continue testing newer kernels until they find the time to redesign their working environment.

Quoted 5 lines

No, it is VirtualBox's problem. Nobody outside of the kernel should be using that symbol,

it can only be used for totally
unsupportable things as far as upstream is concerned.

Creating any uncacheable or write protected mappings in unsupportable? I suspect you would have a hard time actually justifying this. That's a relatively common and useful operation in device drivers.

The only special case I would agree with you is doing write-combined mappings using this which is indeed generally unsupported yet on x86 before the full PAT infrastructure goes in (but
I cannot imagine VirtualBox would use WC for anything)

Most likely if they just want to write protect something they should either use the new interface or just open code it using lookup_address() / change ptes / flush tlbs.

-Andi

author
#10 lore raw
In reply to Adrian Bunk
Quoted 2 lines · Adrian Bunk

The real problem is that the kernel seems to lack functionality you require for doing some work.

Not sure how you reached that conclusion.

Quoted 2 lines

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

Work on the installer does not so much hard depend on VirtualBox (or any other emulator), but can be done much more effectively and efficiently using one.

It allows me to run the installer inside the emulator without the need for a second computer (and using the same keyboard). It allows me to take snapshots just before stages I'm interested in and then add debugging or try changes. If what I tried does not work, I can just revert to the snapshot and try something else without having to run the full installation from scratch. It allows me to easily test RAID setups without having multiple physical disks. Etc.

The fact that VirtualBox does not (yet) work for me with 2.6.25 means that I'm unable to run 2.6.25 on my main desktop and do any real work. Rebooting my desktop whenever I want to use VirtualBox is just not a realistic option.
That in turn means that I cannot do any more testing of 2.6.25, because most of the testing I do consists of just using a new kernel and critically observing the behavior of my system. As I've caught a fair number of bugs that way for the past few releases, I'd say that's a useful contribution.

#11 lore raw
Quoted 21 lines · On Tue, 19 Feb 2008 22:55:01 +0100 Frans Pop <[email protected]> wrote:
Quoted 2 lines · Adrian Bunk

The real problem is that the kernel seems to lack functionality you require for doing some work.

Not sure how you reached that conclusion.

Quoted 2 lines

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

Work on the installer does not so much hard depend on VirtualBox (or any other emulator), but can be done much more effectively and efficiently using one.

It allows me to run the installer inside the emulator without the need for a second computer (and using the same keyboard). It allows me to take snapshots just before stages I'm interested in and then add debugging or try changes. If what I tried does not work, I can just revert to the snapshot and try something else without having to run the full installation from scratch. It allows me to easily test RAID setups without having multiple physical disks. Etc.

Quoted 5 lines

The fact that VirtualBox does not (yet) work for me with 2.6.25 means that I'm unable to run 2.6.25 on my main desktop and do any real work. Rebooting my desktop whenever I want to use VirtualBox is just not a realistic option.
That in turn means that I cannot do any more testing of 2.6.25,

I assume you've read the other mails right and went to the virtualbox form to get the small patch to make it work on .25 right?

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#12 lore raw
In reply to Frans Pop
Quoted 9 lines · Frans Pop
Quoted 2 lines · Adrian Bunk

The real problem is that the kernel seems to lack functionality you require for doing some work.

Not sure how you reached that conclusion.

Quoted 2 lines

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

Quoted 8 lines

The fact that VirtualBox does not (yet) work for me with 2.6.25 means that I'm unable to run 2.6.25 on my main desktop and do any real work. Rebooting my desktop whenever I want to use VirtualBox is just not a realistic option.
That in turn means that I cannot do any more testing of 2.6.25, because most of the testing I do consists of just using a new kernel and critically observing the behavior of my system. As I've caught a fair number of bugs that way for the past few releases, I'd say that's a useful contribution.

And it will be just as valuable when you stay with 2.6.24 finding bugs there.

And once VirtualBox has been updated to work with 2.6.25, you can test 2.6.25.

Harvey

#13 lore raw
In reply to Frans Pop
Quoted 6 lines · Frans Pop
Quoted 2 lines · Adrian Bunk

The real problem is that the kernel seems to lack functionality you require for doing some work.

Not sure how you reached that conclusion.

Quoted 7 lines
Quoted 2 lines

Why does your work on the Debian Installer depend on VirtualBox and can't be done with what the kernel already ships?

Work on the installer does not so much hard depend on VirtualBox (or any other emulator), but can be done much more effectively and efficiently using one.
...

The "or any other emulator" is exactly where my question is directed at.

Xen, KVM or even qemu come into my mind, but considering how loudly you complained about a temporary breakage for VirtualBox there must be a reason why your work on the Debian Installer can only be done effectively and efficiently with an emulator module that has AFAIK not been submitted for inclusion in the kernel.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
author
#14 lore raw
Quoted 7 lines · Adrian Bunk

The "or any other emulator" is exactly where my question is directed at.

Xen, KVM or even qemu come into my mind, but considering how loudly you complained about a temporary breakage for VirtualBox there must be a reason why your work on the Debian Installer can only be done effectively and efficiently with an emulator module that has AFAIK not been submitted for inclusion in the kernel.

- Xen is currently not supported by the kernel Debian Installer uses (though work is being done to change that.
- KVM AFAIK requires hardware support that I don't have.
- QEMU is completely useless because of its slow speed without the (also out of tree) kqemu module, which does not work when the host system is x86_64 [1]. Also, I very much prefer the VirtualBox user interface over what qemu has to offer.
- I've actually used VMWare for a long time (licenced), but stopped after the 5 series stopped working with current kernels and around that time VirtualBox became available as an alternative.

Hope that explains.

Cheers,
FJP

[1] http://bugs.debian.org/444160

#15 lore raw
In reply to Frans Pop
Quoted 33 lines · On Sun, 17 Feb 2008 10:25:30 +0100 Frans Pop <[email protected]> wrote:

Yesterday, after spending quite a few hours over the last days on bisecting some serious regressions and finding workarounds for them, I thought I could start using 2.6.25-rc2 as the new kernel for my desktop. Unfortunately I found that I cannot because it would make my other main activity - working on the Debian installation system - impossible.

For my work on the Debian Installer I heavily rely on emulators to run test installs and ATM my emulator of choice is VirtualBox (the fully open "ose" version). This requires the vboxdrv kernel module, but unfortunately: vboxdrv: Unknown symbol change_page_attr

At first I traced this to:
e1271f68
x86: deprecate change_page_attr() for drivers
With the introduction of the new API, no driver or non-archcore code
needs to use c-p-a anymore, so this patch also deprecates the EXPORT_SYMBOL of CPA (it's a horrible API after all). which had:

(quoted diff) +1 -1
-EXPORT_SYMBOL(change_page_attr);
+EXPORT_UNUSED_SYMBOL(change_page_attr); /* to be removed in 2.6.27 */

Which seemed entirely reasonable but left me wondering about the error I got.

But then I found:
d1028a15
x86: make various pageattr.c functions static change_page_attr_add is only used in pageattr.c now, so we can make this function static.
change_page_attr() isn't used anywere at all anymore; this function is a really bad API anyway so just remove the bloat entirely.

Quoted 3 lines

Which removed the entire function (without even properly mentioning it in the shortlog).

the initial plan was for a depreciation period. Sadly it was untenable since the API was changing entirely to fix bugs and add a really important feature
(the ability to clflush the exact range rather than wbinvd'ing the caches of all cpus in the system), at which point we had to pull the function right away.

As for virtualbox; a fix for that already exists, and it consists of removing 25 lines of workarounds for c-p-a bugs, and just using set_memory_x / set_memory_nx.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#16 lore raw
Quoted 4 lines · Arjan van de Ven

the initial plan was for a depreciation period. Sadly it was untenable since the API was changing entirely to fix bugs and add a really important feature
(the ability to clflush the exact range rather than wbinvd'ing the caches of all cpus in the system),

Just for the record: I posted full patches to implement clflush support some time ago without changing any exported API. So your claims that changing the API was needed to implement CLFLUSH are not correct.

Also I believe some assumptions behind the new API are faulty (in particular that the caller doesn't fully own the to be changed pages) and make it actually impossible to implement the cache attribute PTE changing operation fully correct according to the Intel x86 manual (which requires temporary unmap)

-Andi

#17 lore raw
Quoted 11 lines · On Mon, 18 Feb 2008 13:31:48 +0100 Andi Kleen <[email protected]> wrote:
Quoted 5 lines · Arjan van de Ven

the initial plan was for a depreciation period. Sadly it was untenable since the API was changing entirely to fix bugs and add a really important feature (the ability to clflush the exact range rather than wbinvd'ing the caches of all cpus in the system),

Just for the record: I posted full patches to implement clflush support some time ago without changing any exported API. So your claims that changing the API was needed to implement CLFLUSH are not correct.

yeah of course it is possible to make things "smart" by having hidden state. doesn't make it right.

Quoted 6 lines

Also I believe some assumptions behind the new API are faulty (in particular that the caller doesn't fully own the to be changed pages) and make it actually impossible to implement the cache attribute PTE changing operation fully correct according to the Intel x86 manual (which requires temporary unmap)

the Intel x86 manual explicitly only has a temporary unmap when going from a cached state to a write-combining state. Any other transition does not require an unmap. Which makes this not impossible, all a cached->WC transition needs to do is go via an intermediate UC state and the really expensive process from the manual is not needed.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#18 lore raw
Quoted 10 lines · Arjan van de Ven
Quoted 7 lines · On Mon, 18 Feb 2008 13:31:48 +0100 Andi Kleen <[email protected]> wrote:
Quoted 5 lines · Arjan van de Ven

the initial plan was for a depreciation period. Sadly it was untenable since the API was changing entirely to fix bugs and add a really important feature (the ability to clflush the exact range rather than wbinvd'ing the caches of all cpus in the system),

Quoted 7 lines
Quoted 4 lines

Just for the record: I posted full patches to implement clflush support some time ago without changing any exported API. So your claims that changing the API was needed to implement CLFLUSH are not correct.

yeah of course it is possible to make things "smart" by having hidden state. doesn't make it right.

Not sure how you can call global_flush_tlb() "hidden". It was a quite exposed interface.

Anyways there was also no principle reason in the old interface why the flush couldn't have been done immediately. The only reason it wasn't done was that Linus long ago asked for separate flushing to improve efficiency of large remaps.

Quoted 9 lines
Quoted 5 lines

Also I believe some assumptions behind the new API are faulty (in particular that the caller doesn't fully own the to be changed pages) and make it actually impossible to implement the cache attribute PTE changing operation fully correct according to the Intel x86 manual (which requires temporary unmap)

the Intel x86 manual explicitly only has a temporary unmap when going from a cached state to a write-combining state. Any other transition does not require an unmap. Which makes this not impossible, all a cached->WC transition needs

True, that's a possible workaround for this deficiency of the new API.

Quoted 2 lines

to do is go via an intermediate UC state and the really expensive process from the manual is not needed.

Ok then you're proposing to use a even more expensive operation just to patch this over. I guess that will work as long as we assume none of the callers cares too much about performance, but trying to describe it as an improvement is quite a stretch.

-Andi

#19 lore raw
Quoted 6 lines · On Mon, 18 Feb 2008 18:11:59 +0100 Andi Kleen <[email protected]> wrote:
Quoted 2 lines

to do is go via an intermediate UC state and the really expensive process from the manual is not needed.

Ok then you're proposing to use a even more expensive operation just to patch this over. I guess that will work as long as we assume none of the callers cares too much about performance,

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why the folks who care about performance (the graphics guys) stopped using it. WC is slow, and on modern cpus leads to really bad performance. I'm really half tempted to just ignore WC entirely and suggest that we don't even implement it in the kernel. Yes it's really that bad.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#20 lore raw
Quoted 1 line

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why

Interesting.

Quoted 1 line

the folks who care about performance (the graphics guys) stopped using it.

I didn't know this. What do they do instead?

I understand that WC was added originally because AGP was really slow at IO towards the CPU. You mean on PCI-E it is fast enough now that standard cached WB works well enough?

Quoted 3 lines

WC is slow, and on modern cpus leads to really bad performance. I'm really half tempted to just ignore WC entirely and suggest that we don't even implement it in the kernel. Yes it's really that bad.

At least the X server still uses it. In fact there are already some performance regression regarding this from differing kernel behavioun in the sysfs interfaces vs /dev/mem.

What would you recommend should the X server use instead? Always map standard WB? How about on older AGP systems?

-Andi

#21 lore raw
Quoted 9 lines · On Mon, 18 Feb 2008 19:52:28 +0100 Andi Kleen <[email protected]> wrote:
Quoted 2 lines

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why

Interesting.

Quoted 2 lines

the folks who care about performance (the graphics guys) stopped using it.

I didn't know this. What do they do instead?

they use a cached mapping and use clflush for the cases they want to be sure the GPU sees the data. This turns out to be faster than WC. (think "double to tripple digit percentage")

Quoted 3 lines

I understand that WC was added originally because AGP was really slow at IO towards the CPU. You mean on PCI-E it is fast enough now that standard cached WB works well enough?

since then the graphics programming paradigm has changed as well; WC is really bad for reading data no matter what; it focuses on group writes so that you don't get one transaction per write, but reads are extremely slow. And apparently in the current graphics systems (with "composite" and the like) all of this stuff doesn't get used anymore. Instead they use cached mappings, acceleration engines etc.

Quoted 6 lines
Quoted 4 lines

WC is slow, and on modern cpus leads to really bad performance. I'm really half tempted to just ignore WC entirely and suggest that we don't even implement it in the kernel. Yes it's really that bad.

At least the X server still uses it.

depends on which driver. The new generation Intel graphics ones don't. I doubt the proprietary ones do either, those are even more performance tuned. I understand that the radeon driver is doing or going to do something similar to the Intel driver as well.

Quoted 4 lines

What would you recommend should the X server use instead? Always map standard WB? How about on older AGP systems?

the X drivers need to do what is best; the ones with acceleration use WB nowadays. (For the Intel driver that is somewhat recent, so it could be that your distro doesn't do that yet).

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#23 lore raw
Quoted 11 lines · Ingo Molnar
Quoted 4 lines · Andi Kleen
Quoted 2 lines

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why

Interesting.

does this refresh your memory:

http://lkml.org/lkml/2008/1/10/99

I'm still far from convinced it covers all cases (see also Roland's example) Arjan might be right for some modern graphics hardware, but there is a lot more hardware out there than only this, old and new and non graphics and graphics that Arjan didn't cover.

If Arjan was 100% right then PAT would not be needed at all. I wonder why Venki/Suresh went through all the pain of resurrecting the old PAT patchkit then!? Surely there is more about this.

-Andi

#24 lore raw
In reply to Arjan van de Ven
Quoted 2 lines

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why the folks who care about performance (the graphics guys) stopped using it.

WC for main memory or WC for mmio spaces ?

#25 lore raw
Quoted 5 lines · On Mon, 18 Feb 2008 18:40:50 +0000 Alan Cox <[email protected]> wrote:
Quoted 3 lines

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why the folks who care about performance (the graphics guys) stopped using it.

WC for main memory or WC for mmio spaces ?

if you mean 'framebuffer' with mmio space, they stopped using it there as well afaik.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#26 lore raw
Quoted 2 lines

if you mean 'framebuffer' with mmio space, they stopped using it there as well afaik.

And also bursting commands to mmio fifos - eg the 3Dfx where it makes a *huge* difference.

Alan

#27 lore raw
In reply to Arjan van de Ven
Quoted 5 lines

I've yet to see a user who wants WC. Lets face it, WC *sucks*. This is why the folks who care about performance (the graphics guys) stopped using it. WC is slow, and on modern cpus leads to really bad performance. I'm really half tempted to just ignore WC entirely and suggest that we don't even implement it in the kernel. Yes it's really that bad.

I know of one case at least where WC is very useful. Some InfiniBand adapters allow small messages to be written directly into the adapter's PCI space BAR to lower latency (having the CPU write the message avoids doing something like build descriptor, ring doorbell register on adapter, adapter DMA message out of CPU memory). And mapping the PCI space with WC is a pretty big win -- for example for mlx4 hardware it gets MPI latency from ~1.8 usec to ~1.3 usec which is a big deal. I think most real users of mlx4 hardware are using a hacky out-of-tree patch to allow using PAT to set WC.

AFAIK mapping PCI memory WB is not allowed, so WC is really our only choice.

- R.

#28 lore raw
Quoted 2 lines · On Mon, 18 Feb 2008 10:53:42 -0800 Roland Dreier <[email protected]> wrote:

AFAIK mapping PCI memory WB is not allowed, so WC is really our only choice.

afaik that depends on the BAR being prefetchable or not.

(and by your argument, ioremap_cached() would not be useful, and since that was, until 2.6.25-rc1, the default behavior for ioremap(), would have caused massive problems)

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#29 lore raw
Quoted 4 lines
Quoted 2 lines

AFAIK mapping PCI memory WB is not allowed, so WC is really our only choice.

afaik that depends on the BAR being prefetchable or not.

In my case the BAR is prefetchable.

Quoted 2 lines

(and by your argument, ioremap_cached() would not be useful, and since that was, until 2.6.25-rc1, the default behavior for ioremap(), would have caused massive problems)

I'm not sure what ioremap_cached() would really do in my case, since the MTRRs for PCI memory are set to UC, so without monkeying with MTRR contents (which can't really be done safely) the only choices we have are leaving the mapping as UC or using PAT to get WC.

Also in my case I'm more concerned about latency of finishing a small write rather than througput. So I'm not sure that I would really want to do a write to a WB mapping followed by CLFLUSH anyway.

- R.

#30 lore raw
Quoted 6 lines · Roland Dreier
Quoted 4 lines
Quoted 2 lines

AFAIK mapping PCI memory WB is not allowed, so WC is really our only choice.

afaik that depends on the BAR being prefetchable or not.

In my case the BAR is prefetchable.

Even if the BAR is prefetchable, on some platforms mapping MMIO space as WB can cause bad results like machine check etc.

Quoted 7 lines
Quoted 2 lines

(and by your argument, ioremap_cached() would not be useful, and since that was, until 2.6.25-rc1, the default behavior for ioremap(), would have caused massive problems)

I'm not sure what ioremap_cached() would really do in my case, since the MTRRs for PCI memory are set to UC, so without monkeying with MTRR contents (which can't really be done safely) the only choices we have are leaving the mapping as UC or using PAT to get WC.

thanks,
suresh