Skip to contentSkip to conversation

algorithm for handling bad cachelines

RFC Ben Widawsky 4 participants 8 messages Mar 27, 2012 – Mar 28, 2012 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad. There is also a mechanism to remap the bad cachelines. The implementation details aren't quite clear to me yet, but I'd like to enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's probably going to get a lot of flames.

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read a module parameter with a path in the filesystem of the list of bad lines. It's not clear to me yet exactly what I need to store, but it should be a relatively simple list.
4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW loading do we have to finish remapping. If it can happen at any time while the card is running, I don't need the filesystem stuff, but I believe I need to remap the lines quite early in the device bootstrap.

The only alternative I have is a huge comma separated string for a module parameter, but I kind of like reading the file better.

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

Ben

#1 lore raw
Quoted 3 lines · Ben Widawsky

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

badblocks and badram (memmap) both seem to be similar situations. A comma-separated list works when the number is quite small, otherwise an ioctl. Of course that requires us to be able to reprogram the cache on the fly.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#2 lore raw
In reply to Ben Widawsky
Quoted 13 lines · Ben Widawsky

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad. There is also a mechanism to remap the bad cachelines. The implementation details aren't quite clear to me yet, but I'd like to enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's probably going to get a lot of flames.

Quoted 8 lines

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read a module parameter with a path in the filesystem of the list of bad lines. It's not clear to me yet exactly what I need to store, but it should be a relatively simple list.

.... path in filesystem is no-go for kernel interface. So bad cachelines need to go into the modele parameter itself. Or we add a sysfs interface and reset the gpu (because if my understanding is right, we can't disable cachelines once the gpu has used them).

Quoted 7 lines

4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW loading do we have to finish remapping. If it can happen at any time while the card is running, I don't need the filesystem stuff, but I believe I need to remap the lines quite early in the device bootstrap.

I believe so, too ;-)

Quoted 2 lines

The only alternative I have is a huge comma separated string for a module parameter, but I kind of like reading the file better.

Well, you can't read a file from the kernel because we might init the driver without any userspace present (when the driver is built-in).

Quoted 3 lines

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

-Daniel

Signature · 3 lines
Daniel Vetter
Mail: [email protected]
Mobile: +41 (0)79 365 57 48
#3 lore raw
Quoted 23 lines · On Tue, 27 Mar 2012 16:50:39 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 13 lines · Ben Widawsky

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad. There is also a mechanism to remap the bad cachelines. The implementation details aren't quite clear to me yet, but I'd like to enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's probably going to get a lot of flames.

Quoted 8 lines

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read a module parameter with a path in the filesystem of the list of bad lines. It's not clear to me yet exactly what I need to store, but it should be a relatively simple list.

Quoted 4 lines

.... path in filesystem is no-go for kernel interface. So bad cachelines need to go into the modele parameter itself. Or we add a sysfs interface and reset the gpu (because if my understanding is right, we can't disable cachelines once the gpu has used them).

I think we have to assume the list could get quite long. So long in fact, I imagine the user may often want to reset it and try his/her luck again with some lines.

Could you elaborate more on why it's a no-go? The module parameter setting itself is limited to root. I was trying to clearly understand exactly why this can't be done, and some of the lore behind why file access in the kernel is such a bad thing (assuming the files being accessed are set at module load time). I wouldn't want to go the route of loading an arbitrary path - which seems like a terrible idea; though it works for firmware blobs, and I half thought we could load this like a firmware blob.

Anyway, assuming a gpu reset is sufficient to remap (docs only clearly state reset works for disabling, iirc) then I would like to do that. What is the appropriate interface for that? The dev node? Sysfs?

Quoted 10 lines
Quoted 9 lines

4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW loading do we have to finish remapping. If it can happen at any time while the card is running, I don't need the filesystem stuff, but I believe I need to remap the lines quite early in the device bootstrap.

Quoted 7 lines

I believe so, too ;-)

Quoted 2 lines

The only alternative I have is a huge comma separated string for a module parameter, but I kind of like reading the file better.

Well, you can't read a file from the kernel because we might init the driver without any userspace present (when the driver is built-in).

Userspace should still be present in this case, right? The kernel command line should suffice, I think.

Quoted 5 lines
Quoted 4 lines

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

-Daniel

Thanks

#4 lore raw
Quoted 44 lines · Ben Widawsky
Quoted 23 lines · On Tue, 27 Mar 2012 16:50:39 +0200 Daniel Vetter <[email protected]> wrote:
Quoted 13 lines · Ben Widawsky

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad. There is also a mechanism to remap the bad cachelines. The implementation details aren't quite clear to me yet, but I'd like to enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's probably going to get a lot of flames.

Quoted 8 lines

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read a module parameter with a path in the filesystem of the list of bad lines. It's not clear to me yet exactly what I need to store, but it should be a relatively simple list.

Quoted 4 lines

.... path in filesystem is no-go for kernel interface. So bad cachelines need to go into the modele parameter itself. Or we add a sysfs interface and reset the gpu (because if my understanding is right, we can't disable cachelines once the gpu has used them).

I think we have to assume the list could get quite long. So long in fact, I imagine the user may often want to reset it and try his/her luck again with some lines.

Could you elaborate more on why it's a no-go? The module parameter setting itself is limited to root. I was trying to clearly understand exactly why this can't be done, and some of the lore behind why file access in the kernel is such a bad thing (assuming the files being accessed are set at module load time). I wouldn't want to go the route of loading an arbitrary path - which seems like a terrible idea; though it works for firmware blobs, and I half thought we could load this like a firmware blob.

Quoted 3 lines

Anyway, assuming a gpu reset is sufficient to remap (docs only clearly state reset works for disabling, iirc) then I would like to do that. What is the appropriate interface for that? The dev node? Sysfs?

I personally prefer sysfs for this. Albeit you might have some issues with the one value per file limit ... I guess a list of hex values is ok though.

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines

4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW loading do we have to finish remapping. If it can happen at any time while the card is running, I don't need the filesystem stuff, but I believe I need to remap the lines quite early in the device bootstrap.

I believe so, too ;-)

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The only alternative I have is a huge comma separated string for a module parameter, but I kind of like reading the file better.

Well, you can't read a file from the kernel because we might init the driver without any userspace present (when the driver is built-in).

Userspace should still be present in this case, right? The kernel command line should suffice, I think.

Somewhen later on, but only after the hw is intialized. But if you're going the runtime interface route anyway, it doesn't matter. -Daniel

Signature · 3 lines
Daniel Vetter
Mail: [email protected]
Mobile: +41 (0)79 365 57 48
#5 lore raw
In reply to Ben Widawsky
Quoted 29 lines · On Tue, 27 Mar 2012 07:19:43 -0700 Ben Widawsky <[email protected]> wrote:

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad. There is also a mechanism to remap the bad cachelines. The implementation details aren't quite clear to me yet, but I'd like to enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's probably going to get a lot of flames.

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read a module parameter with a path in the filesystem of the list of bad lines. It's not clear to me yet exactly what I need to store, but it should be a relatively simple list.
4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW loading do we have to finish remapping. If it can happen at any time while the card is running, I don't need the filesystem stuff, but I believe I need to remap the lines quite early in the device bootstrap.

Quoted 6 lines

The only alternative I have is a huge comma separated string for a module parameter, but I kind of like reading the file better.

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

I think the main thing here is to make sure we handle the L3 parity check interrupts. I don't think "lines going bad" will be very common in practice (maybe if you really abuse your CPU by putting it in the freezer and then into an oven or something), so having a fancy interface for it probably isn't too important.

Also, the behavior should be configurable. For the vast majority of users, an L3 parity interrupt is no big deal, and resetting the GPU when we see one is more than we want. So it should probably be off by default and be controlled with a module parameter and maybe a config option.

As for the interface for feeding in bad lines (useful for testing if nothing else), I'd prefer an ioctl over a module parameter. Some as yet uncoded userspace service can load a set based on previously collected uevents at boot time before any real GPU stuff runs...

Reading a file at load time is definitely a non-starter; we have no idea whether the filesystem will be available to the module based on mount points, hiding, chroots, etc.

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw
Quoted 40 lines · On Wed, 28 Mar 2012 10:26:52 -0700 Jesse Barnes <[email protected]>…
Quoted 30 lines · On Tue, 27 Mar 2012 07:19:43 -0700 Ben Widawsky <[email protected]> wrote:

I wanted to run this by folks before I start doing any actual work.

This is primarily for GPGPU, or perhaps *really* accurate rendering requirements.

IVB+ has an interrupt to tell us when a cacheline seems to be going bad. There is also a mechanism to remap the bad cachelines. The implementation details aren't quite clear to me yet, but I'd like to enable this feature for userspace.

Here is my current plan, but it involves filesystem access, so it's probably going to get a lot of flames.

1. Handle cache line going bad interrupt.
<After n number of these interrupts to the same line,>
2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>
3. Read a module parameter with a path in the filesystem of the list of bad lines. It's not clear to me yet exactly what I need to store, but it should be a relatively simple list.
4. Parse list on driver load, and handle as necessary.
5. goto 1.

Probably the biggest unanswered question is exactly when in the HW loading do we have to finish remapping. If it can happen at any time while the card is running, I don't need the filesystem stuff, but I believe I need to remap the lines quite early in the device bootstrap.

Quoted 6 lines

The only alternative I have is a huge comma separated string for a module parameter, but I kind of like reading the file better.

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

Quoted 5 lines

I think the main thing here is to make sure we handle the L3 parity check interrupts. I don't think "lines going bad" will be very common in practice (maybe if you really abuse your CPU by putting it in the freezer and then into an oven or something), so having a fancy interface for it probably isn't too important.

I'd like to clarify my statement a bit. Thanks for pointing this out. I've yet to enable this interrupt and see what happens, so this is all speculative based on that document:

It seems like more lines may be susepitble to parity errors via cosmic rays. Since upon learning this information requires a GPU reset (to Andi's point which I'll address seperately), I think it's desirable for users to be able to choose early on whether or not the lines ever get enabled. For the graphics case however, learning the information every time is totally acceptible, and as you say below is probably a nop.

Quoted 11 lines

Also, the behavior should be configurable. For the vast majority of users, an L3 parity interrupt is no big deal, and resetting the GPU when we see one is more than we want. So it should probably be off by default and be controlled with a module parameter and maybe a config option.

As for the interface for feeding in bad lines (useful for testing if nothing else), I'd prefer an ioctl over a module parameter. Some as yet uncoded userspace service can load a set based on previously collected uevents at boot time before any real GPU stuff runs...

It's still not clear to me if we can remap after the GPU is fully up and running and handling the IOCTLs. If so, I believe that is definitely the consesus.

Quoted 5 lines

Reading a file at load time is definitely a non-starter; we have no idea whether the filesystem will be available to the module based on mount points, hiding, chroots, etc.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 7 lines · On Wed, 28 Mar 2012 02:59:26 -0700 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Ben Widawsky

1. Handle cache line going bad interrupt. <After n number of these interrupts to the same line,>

Never use global n without timeout for corrected errors, you would need a leaky bucket with a suitable timeout.

As I understand electrons (which is not very well) parity errors happen all the time and are transparently corrected by our HW. So I suppose 'n' is still interesting information, but your point is noted. It is probably better to let userspace decide that n value.

Take this with a grain of salt because the number of interrupts we get is speculative as I haven't actually tried to enable this.

Quoted 5 lines
Quoted 4 lines

2. send a uevent
2.5 reset the GPU (docs tell us to)
<On module load>

Quoted 8 lines

Persistent lists on disk usually suffer from all kinds of problems, e.g. you need to detect when the board or CPU has changed. Also when the problem is temporary you do not really want to save such information permanent.

Usually it's better to rediscover such state each time and handle it again. Then you also don't need the uevent or complicated user interfaces.

It seems nice to have information stored non-volatility. It doesn't have to be used by the user, but assuming they want to load the option to actually detect these events, it's probably also beneficial to give the known bad cachelines since this requires a GPU reset once detected. The reset both takes time, and may do more damage (that is based on past experience/products only and I hope IVB can magnificently recover from our bad GPU programming).

Quoted 8 lines
Quoted 4 lines

Any feedback is highly appreciated. I couldn't really find much precedent for doing this in other drivers, so pointers to similar things would also be highly welcome.

http://mcelog.org

-Andi

Thanks.