Skip to contentSkip to conversation

Better MCA recovery on IPF

RFC Hidetoshi Seto 10 participants 28 messages Oct 27, 2003 – Nov 10, 2003

lore
Share & tools
lore
#0 lore raw

I want to make contributions to the development of MCA Error Handling.

According to IPF Error Handling Guide, OS should have capability to recover from error.

There are three types of error, Corrected, Recoverable, and Fatal. They are reported to OS by MCA/CPEI/CMCI, and actions required to OS depend on the type of them. Relations between the type and the action are as follows;

- Corrected:
Do nothing.

- Recoverable:
Depends on the situation,
- Fix the error, continue interrupted thread.
- Terminate suffered threads.
- Just as Fatal, reboot.

- Fatal:
Reboot system immediately.

In all case, Linux should log error information based on SAL record. So, some programs in user land, like fault prediction logic or a daemon that reports error to remote site, could use these logs. And system administrator also could use these logs to keep their system healthy.

I have strong expectations for Linux to realize such recovery features. However, Linux is deficient in recovery codes, especially on recoverable MCA, at this moment. (I know your good job, Tony.)

I want to know what difficulty keep Linux as-is.

What do you think of error recovery on Linux?
What kind of functions, macros, structures should Linux have for recovery?

Best regards,

------

H.Seto <[email protected]>

#1 lore raw

Hi,

my name is Matthias Fouquet-Lapar, I'm working in SGI's SW platform group mainly on CPU exception and error handling.

As other members of this group, we're also looking into changing the Linux error handling to suit the needs of a reliable super-computer environment.

I think error handling needs to be extended to not only recover from errors and kill for example the concerned application. Increasing chip density will increase the soft error rate, so it also becomes important to determinate if a error is soft (caused for example by cosmic rays) or if it is a true HW component failure requiring a replacement.

There are also more complex error scenarios in multiple CPU environments when for example all CPUs access a cache line which has an error.

Traditionally we're verifying our error handling by error injection as well as running tests with real, broken HW components for verification and regression testing.

Obviously a lot of the error handling will be very platform dependant, but I think we should be able to come up with a common frame set. What do you think ?

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127
Quoted 43 lines

I want to make contributions to the development of MCA Error Handling.

According to IPF Error Handling Guide, OS should have capability to recover from error.

There are three types of error, Corrected, Recoverable, and Fatal. They are reported to OS by MCA/CPEI/CMCI, and actions required to OS depend on the type of them. Relations between the type and the action are as follows;

- Corrected:
Do nothing.

- Recoverable:
Depends on the situation,
- Fix the error, continue interrupted thread.
- Terminate suffered threads.
- Just as Fatal, reboot.

- Fatal:
Reboot system immediately.

In all case, Linux should log error information based on SAL record. So, some programs in user land, like fault prediction logic or a daemon that reports error to remote site, could use these logs. And system administrator also could use these logs to keep their system healthy.

I have strong expectations for Linux to realize such recovery features. However, Linux is deficient in recovery codes, especially on recoverable MCA, at this moment. (I know your good job, Tony.)

I want to know what difficulty keep Linux as-is.

What do you think of error recovery on Linux?
What kind of functions, macros, structures should Linux have for recovery?

Best regards,

------

H.Seto <[email protected]>

#2 lore raw
In reply to Hidetoshi Seto

Hi, Matthias.

Quoted 7 lines

I think error handling needs to be extended to not only recover from errors and kill for example the concerned application. Increasing chip density will increase the soft error rate, so it also becomes important to determinate if a error is soft (caused for example by cosmic rays) or if it is a true HW component failure requiring a replacement.

Surely, it is very important to specify where target error comes from. I do not want to carry out advice to replace the component, which working correctly.

Quoted 3 lines

Obviously a lot of the error handling will be very platform dependant, but I think we should be able to come up with a common frame set. What do you think ?

Of course, I agree with a common frame set. In the case of platform premising IPF, I think it is better to regard the Intel's Chipset as the de facto standard.

Thanks.

------

H.Seto <[email protected]>

#3 lore raw
In reply to Hidetoshi Seto
Quoted 3 lines · Hidetoshi Seto

In the case of platform premising IPF, I think it is better to regard the Intel's Chipset as the de facto standard.

hmm...given ia64 intel boxes I've played with have no error containment and softfail on everything, I'm not sure that's a good choice. Or has enough been published about the chipset to change those behaviors?

thanks,
grant

#4 lore raw
In reply to Hidetoshi Seto

Hi,

Quoted 4 lines

Of course, I agree with a common frame set. In the case of platform premising IPF, I think it is better to regard the Intel's Chipset as the de facto standard.

I think there should be an abstraction layer hiding the underlying HW implementation. I think handling for example a memory error by killing the affected user application, should work on any chipset and/or CPU architecture (if technically possible). We should not restrict ourselves to specific platforms, I think the general trend is that the error rate will go up because :

- faster off-chip frequencies
- lower supply voltages decreasing signal/noise ratio
- higher suspectibility to cosmis rays causing SEU (Single Event Upsets) due to smaller process. There are for example estimations that SEUs will increase by a factor of 100 when going from a .13um process to .9um

The only alternatives to burrying a system under 50 feet of solid rock to avoid cosmic rays and improvements in HW design (chipkill will help) is to improve error handling and recovery.

Today we have for example the ability that an application can deal with an unexpected event, such as a div by 0. In my eyes it would be possible that an application also could make provisions to handle memory (or cache errors) up to a certain extend, as long as the offending VA is known.

In other words, I would prefer the option for applications writers to have the option to recover within the application if is possible instead of having the application killed (or even the OS in the current state)

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127
#5 lore raw
In reply to Hidetoshi Seto
Quoted 9 lines · Matthias Fouquet-Lapar
Quoted 4 lines

Of course, I agree with a common frame set. In the case of platform premising IPF, I think it is better to regard the Intel's Chipset as the de facto standard.

I think there should be an abstraction layer hiding the underlying HW implementation. I think handling for example a memory error by killing the affected user application, should work on any chipset and/or CPU architecture (if technically possible).

We already have that interface, it is called a signal. The kernel code for handling these events has to be architecture dependent but, once the data has been gathered and the decision made about which user process to kill, we just send SEGV.

BTW, your email address includes your full hostname, instead of just [email protected].

#6 lore raw
In reply to Hidetoshi Seto
Quoted 4 lines

We already have that interface, it is called a signal. The kernel code for handling these events has to be architecture dependent but, once the data has been gathered and the decision made about which user process to kill, we just send SEGV.

:-) I think there is more to it, for example we should avoid re-allocating a page which has a hard error to make sure that the next user stumbles over this again. Maybe this information should also be kept accross a reboot for the same reason. Then you need some API to maintain this information, if for example a DIMM is replaced and it is not longer require to map out a page with an error.

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127
#7 lore raw
In reply to Hidetoshi Seto

Grant Grundler wrote:

Quoted 8 lines · Hidetoshi Seto
Quoted 3 lines

In the case of platform premising IPF, I think it is better to regard the Intel's Chipset as the de facto standard.

hmm...given ia64 intel boxes I've played with have no error containment and softfail on everything, I'm not sure that's a good choice. Or has enough been published about the chipset to change those behaviors?

There are some errors on ia64 that are recoverable, with the right SW (PAL,SAL,Linux) and chipset support.

There are some errors on ia64 that are not recoverable, but hopefully will be in newer cpu & chipset versions.

A Matthias points out, some of the recovery should abstracted out in linux to hide the underlying hardware implementation.

For example, in the case of an application hitting a memory uncorrectable on a multi-processor system, the MCA will be handled by PAL and SAL. If SAL can determine the failing HW physical address, it could pass that information up to linux. Linux could look at the physical address and figure out which application has that address mapped and kill the application, without crashing the system. Linux should also not allow that physical memory to be reused by any other process.

Part of that recovery is platform specific (HW, PAL, SAL) but part of it is platform independent (linux converting the physical address, shooting the app, page handling).

As for IPF being "the defacto standard", IPF is certainly the platform I'm interested in (hence posting to linux-ia64), but others will have their own preference. The platform independent parts of linux should have interfaces designed to work on any platform (duh). Actual implementation will likely be done on several different architectures.

Signature · 2 lines
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          [email protected]
#8 lore raw
In reply to Hidetoshi Seto
Quoted 8 lines

For example, in the case of an application hitting a memory uncorrectable on a multi-processor system, the MCA will be handled by PAL and SAL. If SAL can determine the failing HW physical address, it could pass that information up to linux. Linux could look at the physical address and figure out which application has that address mapped and kill the application, without crashing the system. Linux should also not allow that physical memory to be reused by any other process.

Hi,

I just wondered if a speculative load hitting a cache or memory error does cause an exception on IA64 ?

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127
#9 lore raw
In reply to Hidetoshi Seto

When I was at HP (a good number of years ago, we (HP and Intel) spent a lot of time trying to architect machine check behavior. Actually all of the things you guys have been discussing were considered. Because I have not been following up on this area in many years, I am not sure how much of the work we did actually made it to official architecture documents, although I do know that some of it did.

The main idea was that each layer of the machine check handling code will either be able to transparently (to that layer) recover the error, or pass the information up to the next layer (this information always included a flag that would be set if the error was considered non-recoverable by the lower layer, like for example a tag parity error on a dirty data cache line). The layers we defined and the order in which they were executed when a machine check abort occurred were PAL, SAL and the OS. I have seen some of this information (although I have not checked how complete it is) in chapter 4 of the SAL spec (Itanium Processor Family System Abstraction layer Specification) and section 13.3.i of the architecture spec (Intel Itanium Architecture Software Developers Manual, Volume 2: System Architecture). The SAL_GET_STATE_INFO call was to be central to getting all this information to the OS.

Bert Munoz

Quoted 59 lines

-----Original Message-----
From: Russ Anderson [mailto:[email protected]]
Sent: Monday, November 03, 2003 9:09 AM
To: [email protected]
Cc: [email protected]
Subject: Re: [RFC] Better MCA recovery on IPF

Grant Grundler wrote:

Quoted 5 lines · Hidetoshi Seto
Quoted 3 lines

In the case of platform premising IPF, I think it is better to regard the Intel's Chipset as the de facto standard.

hmm...given ia64 intel boxes I've played with have no error

containment

Quoted 3 lines

and softfail on everything, I'm not sure that's a good choice. Or has enough been published about the chipset to change those behaviors?

There are some errors on ia64 that are recoverable, with the right SW (PAL,SAL,Linux) and chipset support.

There are some errors on ia64 that are not recoverable, but hopefully will be in newer cpu & chipset versions.

A Matthias points out, some of the recovery should abstracted out in linux to hide the underlying hardware implementation.

For example, in the case of an application hitting a memory uncorrectable on a multi-processor system, the MCA will be handled by PAL and SAL. If SAL can determine the failing HW physical address, it could pass that information up to linux. Linux could look at the physical address and figure out which application has that address mapped and kill the application, without crashing the system. Linux should also not allow that physical memory to be reused by any other process.

Part of that recovery is platform specific (HW, PAL, SAL) but part of it is platform independent (linux converting the physical address, shooting the app, page handling).

As for IPF being "the defacto standard", IPF is certainly the platform I'm interested in (hence posting to linux-ia64), but others will have their own preference. The platform independent parts of linux should have interfaces designed to work on any platform (duh). Actual implementation will likely be done on several different architectures.

Signature · 7 lines
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          [email protected]
-
To unsubscribe from this list: send the line "unsubscribe 
linux-ia64" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#10 lore raw
In reply to Hidetoshi Seto
Quoted 7 lines

Hi,

I just wondered if a speculative load hitting a cache or memory error does cause an exception on IA64 ?

As far as I know, the answer is yes. The processor has no way of distinguishing speculative vs. non-speculative accesses on the buses or caches.

Bert Munoz

Quoted 14 lines

Thanks

Matthias Fouquet-Lapar Core Platform Software [email protected]
VNET 521-8213
Principal Engineer Silicon Graphics Home Office
(+33) 1 3047 4127

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
In reply to Hidetoshi Seto
Quoted 9 lines · Matthias Fouquet-Lapar
Quoted 8 lines

For example, in the case of an application hitting a memory uncorrectable on a multi-processor system, the MCA will be handled by PAL and SAL. If SAL can determine the failing HW physical address, it could pass that information up to linux. Linux could look at the physical address and figure out which application has that address mapped and kill the application, without crashing the system. Linux should also not allow that physical memory to be reused by any other process.

Quoted 4 lines

Hi,

I just wondered if a speculative load hitting a cache or memory error does cause an exception on IA64 ?

I dont think a speculative load should cause a problem - at least until code tries to consume the data by transfering it to a processor register.

As I understand the cpu architecture, an error that occurs reading data will result in a poisoned cache line being delivered to the cpu cache. The poisoned cache line can stay in the cache forever. No MCA error is reported until the data is actually consumed by tranfering the data from cache to a cpu register.

This requires some support from the chipset. Some chipsets dont fully support this error model.

Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
#12 lore raw
In reply to Hidetoshi Seto
Quoted 11 lines
Quoted 6 lines

Hi,

I just wondered if a speculative load hitting a cache or memory error does cause an exception on IA64 ?

I dont think a speculative load should cause a problem - at least until
code tries to consume the data by transfering it to a processor register.

If you are doing a read (which is what a speculative load will be generating), the error will be generated by whatever part of the logic that detects it. You cannot possible send poisoned data through a memory bus and a system bus (at least not the Intel system buses I am familiar with) without having some of the error checking logic (ECC or parity) complaining about it (this means generating an MCA).

Quoted 8 lines

As I understand the cpu architecture, an error that occurs reading data
will result in a poisoned cache line being delivered to the cpu cache.
The poisoned cache line can stay in the cache forever. No MCA error is reported until the data is actually consumed by tranfering the data from
cache to a cpu register.

The problem is that the cache error checking logic has no way of knowing that the data it is about to supply to some register is going to be used for a speculative operation. The cache logic is pretty far away (in processor terms) from the decoding logic.

Bert Munoz

Quoted 20 lines

This requires some support from the chipset. Some chipsets dont fully support this error model.

Signature · 11 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.


-
To unsubscribe from this list: send the line "unsubscribe 
linux-ia64" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#13 lore raw
In reply to Hidetoshi Seto
Quoted 12 lines · Alberto Munoz
Quoted 11 lines
Quoted 6 lines

Hi,

I just wondered if a speculative load hitting a cache or memory error does cause an exception on IA64 ?

I dont think a speculative load should cause a problem - at least until
code tries to consume the data by transfering it to a processor register.

Quoted 6 lines

If you are doing a read (which is what a speculative load will be generating), the error will be generated by whatever part of the logic that detects it. You cannot possible send poisoned data through a memory bus and a system bus (at least not the Intel system buses I am familiar with) without having some of the error checking logic (ECC or parity) complaining about it (this means generating an MCA).

As the poisoned data flows thru the BUSes, errors may be reported but these errors are not reported to the OS as uncorrected/fatal MCA errors. Depending on your chipset, errors are logged as platform errors.

There is a good paper by Tony Luck (Intel) that describes data poisoning as used in IA64. You can find it on google or at:

archive.linuxsymposium.org/ols2003/Proceedings/ All-Reprints/Reprint-Luck-OLS2003.pdf

See the section on "data poisoning".

Quoted 23 lines
Quoted 9 lines

As I understand the cpu architecture, an error that occurs reading data
will result in a poisoned cache line being delivered to the cpu cache.
The poisoned cache line can stay in the cache forever. No MCA error is reported until the data is actually consumed by tranfering the data from
cache to a cpu register.

The problem is that the cache error checking logic has no way of knowing that the data it is about to supply to some register is going to be used for a speculative operation. The cache logic is pretty far away (in processor terms) from the decoding logic.

Bert Munoz

Quoted 6 lines

This requires some support from the chipset. Some chipsets dont fully support this error model.

Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
#14 lore raw
In reply to Hidetoshi Seto

Because I was really curious as to how much this field may have changed since the last time I checked, I read fairly quickly through the paper you mention below.

As stated in section 5, second paragraph, of the document you reference below, poisoning does not apply to reads (except for delivering an MCA at any read attempt of the poisoned data). The main value for poisoning is to avoid delivering a machine check "out of context" when it would be caused by a write operation. The problem is that an execution context (or thread, or process) is allowed to retire write operations BEFORE the data has actually been safely stored in memory. For example, you can complete a write operation to the cache, and then have an error occur when the data is written from the cache to main memory. Unfortunately, when this error occurs, chances are that the original context that generated the write may no longer be executing. It is also possible that the written data will never be used again, in which case generating an MCA would be wasteful. Instead of generating an MCA, the hardware marks the data as poisoned (in an implementation specific way that allows the data to move through the memory hierarchy without generating MCAs).

I still believe that a failed speculative read (for example of poisoned data) will generate an MCA. Perhaps someone from Intel can confirm or deny?

Bert Munoz

Quoted 87 lines

-----Original Message-----
From: Jack Steiner [mailto:[email protected]]
Sent: Monday, November 03, 2003 11:29 AM
To: Alberto Munoz
Cc: Matthias Fouquet-Lapar; Russ Anderson; [email protected]
Subject: Re: [RFC] Better MCA recovery on IPF

Quoted 7 lines · Alberto Munoz
Quoted 7 lines
Quoted 6 lines

Hi,

I just wondered if a speculative load hitting a cache or memory error does cause an exception on IA64 ?

Quoted 7 lines
Quoted 4 lines

I dont think a speculative load should cause a problem - at least until
code tries to consume the data by transfering it to a processor register.

If you are doing a read (which is what a speculative load will be generating), the error will be generated by whatever part

of the logic that

Quoted 1 line

detects it. You cannot possible send poisoned data through

a memory bus and a

Quoted 1 line

system bus (at least not the Intel system buses I am

familiar with) without

Quoted 1 line

having some of the error checking logic (ECC or parity)

complaining about it

Quoted 1 line

(this means generating an MCA).

As the poisoned data flows thru the BUSes, errors may be reported but these errors
are not reported to the OS as uncorrected/fatal MCA errors. Depending on your
chipset, errors are logged as platform errors.

There is a good paper by Tony Luck (Intel) that describes data poisoning as used
in IA64. You can find it on google or at:

archive.linuxsymposium.org/ols2003/Proceedings/
All-Reprints/Reprint-Luck-OLS2003.pdf

See the section on "data poisoning".

Quoted 6 lines
Quoted 6 lines

As I understand the cpu architecture, an error that occurs reading data
will result in a poisoned cache line being delivered to the cpu cache.
The poisoned cache line can stay in the cache forever. No

MCA error is

Quoted 5 lines
Quoted 3 lines

reported until the data is actually consumed by tranfering the data from
cache to a cpu register.

The problem is that the cache error checking logic has no

way of knowing that

Quoted 1 line

the data it is about to supply to some register is going to

be used for a

Quoted 1 line

speculative operation. The cache logic is pretty far away

(in processor

Quoted 5 lines

terms) from the decoding logic.

Bert Munoz

Quoted 1 line

This requires some support from the chipset. Some

chipsets dont fully

Quoted 5 lines
Quoted 5 lines

support this error model.

Signature · 4 lines
Thanks

Jack Steiner ([email protected])          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.
#15 lore raw
In reply to Hidetoshi Seto
Quoted 8 lines · Keith Owens
Quoted 4 lines · Matthias Fouquet-Lapar

I think there should be an abstraction layer hiding the underlying HW implementation. I think handling for example a memory error by killing the affected user application, should work on any chipset and/or CPU architecture (if technically possible).

Quoted 4 lines

We already have that interface, it is called a signal. The kernel code for handling these events has to be architecture dependent but, once the data has been gathered and the decision made about which user process to kill, we just send SEGV.

The problem with SEGV is that there exist applications which do strange mmap/mprot tricks and catch and retry SEGVs to implement app-level paging-like behaviour. Two examples which already run on (some ports of) Linux are:

The Texas Persistent Store (open source)
http://www.iam.unibe.ch/~scg/Archive/Software/FreeDB/FreeDB.23.html

ObjectStore (commercial)
http://www.objectstore.net/products/objectstore/index.ssp

You'd have to use SIGBUS or some other signal, or add a new code to the sigcontext to allow those apps to handle the difference.

Greg.

Signature · 2 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.
#16 lore raw
In reply to Hidetoshi Seto
Quoted 3 lines

I still believe that a failed speculative read (for example of poisoned data) will generate an MCA. Perhaps someone from Intel can confirm or deny?

It depends on exactly what you mean by "speculative read", and even there it is not architecturally defined, so different implementations may behave differently ("Ask not the elves for advice for they will say both yes and no" - Tolkien).

"Speculative" reads from memory as a result of lfetch, or a code fetch for a mispredicted branch that reference poisoned data may not generate an MCA (since the processor can know that the poisoned data will not be consumed. Speculative reads from "ld.s" have less scope to avoid the MCA

There's a new bit coming for PAL_PROC_{GET,SET}_FEATURES which will at least tell you (and may allow you to request, if the implementation supports it) whether the processor will respond to poison with CMCI, or upgrade to MCA ... watch the web for a spec update to the SDV

-Tony

#17 lore raw
In reply to Hidetoshi Seto

Hi Tony,

Thank you for the clarification.

Quoted 9 lines
Quoted 3 lines

I still believe that a failed speculative read (for example of poisoned data) will generate an MCA. Perhaps someone from Intel can confirm or deny?

It depends on exactly what you mean by "speculative read", and even there it is not architecturally defined, so different implementations may behave differently ("Ask not the elves for advice for they will say both yes and no" - Tolkien).

Quoted 5 lines

"Speculative" reads from memory as a result of lfetch, or a code fetch for a mispredicted branch that reference poisoned data may not generate an MCA (since the processor can know that the poisoned data will not be consumed. Speculative reads from "ld.s" have less scope to avoid the MCA

I actually meant a read generated as a result of a ld.s.

By the way, what do you mean by "less scope to avoid the MCA"? this statement seems to imply that some existing implementations do it (avoid generating and MCA when a ld.s accesses poisoned data) under some circumstances. Can you elaborate?

What implementation of the Itanium processor supports avoiding MCAs from lfetch or code fetch operations? I don't think Itanium 1 or 2 do this. How about Madison?

Quoted 5 lines

There's a new bit coming for PAL_PROC_{GET,SET}_FEATURES which will at least tell you (and may allow you to request, if the implementation supports it) whether the processor will respond to poison with CMCI, or upgrade to MCA ... watch the web for a spec update to the SDV

I assume this is only for the write path (not the read). As a CMCI on the read path cannot possibly guarantee containment (prevent unmarked corrupt data from making it to a register and eventually to memory, disk or the network). Unless the (architectural) position is that an OS sets this bit at its own risk...

Thanks again,

Bert Munoz

Quoted 3 lines

-Tony

#18 lore raw
In reply to Hidetoshi Seto
Quoted 3 lines · Alberto Munoz

What implementation of the Itanium processor supports avoiding MCAs from lfetch or code fetch operations? I don't think Itanium 1 or 2 do this. How about Madison?

Oops, misunderstanding here. Madison and Deerfield are also Itanium 2. It's like Coppermine and Tualatin are both Pentium 3. When the difference is only cache size, die size, clock frequency and so on, they're not going to change the number. For bigger changes, they might ;-)

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
#19 lore raw
In reply to Hidetoshi Seto
Quoted 16 lines

-----Original Message-----
From: Matthew Wilcox [mailto:[email protected]]
Sent: Wednesday, November 05, 2003 9:31 AM
To: Alberto Munoz
Cc: Luck, Tony; Jack Steiner; Matthias Fouquet-Lapar; Russ Anderson; [email protected]
Subject: Re: [RFC] Better MCA recovery on IPF

Quoted 1 line · Alberto Munoz

What implementation of the Itanium processor supports

avoiding MCAs from

Quoted 1 line

lfetch or code fetch operations? I don't think Itanium 1 or

2 do this. How

Quoted 8 lines
Quoted 1 line

about Madison?

Oops, misunderstanding here. Madison and Deerfield are also Itanium 2.
It's like Coppermine and Tualatin are both Pentium 3. When the difference
is only cache size, die size, clock frequency and so on, they're not going to change the number. For bigger changes, they might ;-)

Not really a misunderstanding. I just got a little lose with the names. I should have said Merced, McKinley instead of Itanium 1 and 2.

In any case, it has been my experience that some times there are fairly major changes in RAS features (typically addressing shortcomings of a predecessor) betweens processors of the same vintage (McKinley, Madison and Deerfield).

Bert Munoz

Quoted 9 lines
Signature · 6 lines
"It's not Hollywood.  War is real, war is primarily not about 
defeat or
victory, it is about death.  I've seen thousands and 
thousands of dead bodies.
Do you think I want to have an academic debate on this 
subject?" -- Robert Fisk
#20 lore raw
In reply to Hidetoshi Seto

Thanks, all.

I was pleased to hear many thought in active developer's mind.

I want to check some in MCA recovery.
Here is some steps, roughly say:

- Stop whole system on MCA even if kernel not suffered.
- Keep working except the suffered application.
- Keep working, and give a chance to self-rehabilitation for the suffered application.

Just now, Linux is making an effort to step up the second from the first.

It also seems there are some key procedures, such as:

- Specify process/thread(s) must be killed
- Specify damaged resources (for example, poisoned pages)

To specify affected ones, OS requires:

- Physical address (and if possible, Virtual address) of the offending operation.
- Interruption must be synchronized.
- States in registers when interrupted.

Is it wrong?

------

H.Seto <[email protected]>

#21 lore raw
In reply to Hidetoshi Seto
Quoted 2 lines

I want to check some in MCA recovery.
Here is some steps, roughly say:

One of the complexities is recovery on a large-scale system, if for example, multiple CPUs access a poisoned memory location at the same time.

Other "interesting" errors scenarious are if data is DEX with bad ECC in CPU A's cache and CPU B requests the line from CPU A.

Quoted 1 line

- Interruption must be synchronized.

I'm not sure what you mean by this.

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127
#22 lore raw
In reply to Hidetoshi Seto
Quoted 4 lines

By the way, what do you mean by "less scope to avoid the
MCA"? this statement seems to imply that some existing implementations do it (avoid generating and MCA when a ld.s accesses poisoned data) under some circumstances. Can you elaborate?

I was just being vague. In the case of "ld.s" the data is targetted at a register ... and we don't have a way to indicate that the contents of a register are poisoned (NaT would at least stop someone consuming the poisoned data, but they wouldn't know why) ... so an MCA is inevitable. In the "lfetch" case we only requested that the data be pulled into cache, which can keep track of whether it is poisoned or not ... so it is architecturally possible to avoid the MCA.

I don't have the details to hand on how existing implementations actually handle this.

-Tony

#23 lore raw
In reply to Hidetoshi Seto

Hi.

Quoted 5 lines

One of the complexities is recovery on a large-scale system, if for example, multiple CPUs access a poisoned memory location at the same time.

Other "interesting" errors scenarious are if data is DEX with bad ECC in CPU A's cache and CPU B requests the line from CPU A.

My concern for poisoning is that I'm not sure the way to clear the poisoned data. Maybe, not so many people know the timing and the guaranteed procedure. I can estimate what the procedure includes, such as changing poisoned memory to uncacheable, clearing suspect data in cache, and storing zeros to the poisoned area.
Even for a single poisoned line in memory, it is need to pause all CPUs on a large-scale system, like Global MCA?

Quoted 3 lines
Quoted 1 line

- Interruption must be synchronized.

I'm not sure what you mean by this.

What I mean by poor English is synchronous MCA.
Executing process can change in the case of asynchronous MCA from platform.

Thanks.

------

H.Seto <[email protected]>

#24 lore raw
In reply to Hidetoshi Seto

Hi,

Quoted 7 lines

My concern for poisoning is that I'm not sure the way to clear the poisoned data. Maybe, not so many people know the timing and the guaranteed procedure. I can estimate what the procedure includes, such as changing poisoned memory to uncacheable, clearing suspect data in cache, and storing zeros to the poisoned area.
Even for a single poisoned line in memory, it is need to pause all CPUs on a large-scale system, like Global MCA?

I think before the poisoned location can be cleared, all objects having potential references must have been terminated (or suspended ?? but there are a lot of problems with this).

Once the reference count of the corresponding page is 0, you should be able to lock the page and clear out the memory. However, you might have a hard error in which case it probably would not be good to put the page back into production. So either adding a flag indicating that the page is not longer usable or attaching the page to some reaper thread might work.

( On our IRIX implementation I also had added a flag which would note that the
page had an increased number of SBEs, so it also would not get re-allocated. It's an interesting disussion if a failure can de-generate and a SBE can turn into a UCE, but we might get everyone bored with that :-))

Quoted 2 lines

What I mean by poor English is synchronous MCA.
Executing process can change in the case of asynchronous MCA from platform.

It's my french :)

Are you meaning

synchronous MCA is caused within an execution context, for example
a process is doing a load and hits an exception

whereas a asynchronous MCA could happen when a line is written back to main memory and this could happen outside of the process's context ?

Thanks

Matthias Fouquet-Lapar  Core Platform Software    [email protected]  VNET 521-8213
Principal Engineer      Silicon Graphics          Home Office (+33) 1 3047 4127
#25 lore raw
In reply to Hidetoshi Seto
Quoted 3 lines

I can estimate what the procedure includes, such as changing poisoned memory to uncacheable, clearing suspect data in cache, and storing zeros to the poisoned area.

There is no way to tell if the error is soft/transient and can be cleared by that sequence, or hard/permanent.

The safest option is to simply take the page with
the error out of service and not re-use it.

-Tony

#26 lore raw
In reply to Hidetoshi Seto
Quoted 6 lines
Quoted 3 lines

I can estimate what the procedure includes, such as changing poisoned memory to uncacheable, clearing suspect data in cache, and storing zeros to the poisoned area.

There is no way to tell if the error is soft/transient and can be cleared by that sequence, or hard/permanent.

I think there is. Depending on your chipset you can re-read the memory uncached after all outstanding references have terminated. If you don't get the same error, it is transient.

Since I would expect that the majority of errors to be transient, I think this really is the right approach. Again, depending on the chipset architecture you might want to do some uncached write/reads ("micro-diagnostics") to see if the problem can be identified to confirm the nature of the problem.

I used similar approaches on other architectures when figuring out if a Single Bit was transient or hard. The goal was to stop triggering for SBEs once you know that you have a hard SBE due to the large overhead

Quoted 2 lines

The safest option is to simply take the page with
the error out of service and not re-use it.

One problem might be that you now miss a page of main memory and it might require an additional TLB entry if you use large memory segments

- Matthias

Quoted 7 lines
(quoted diff) +0 -2
-Tony
-

To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#27 lore raw
In reply to Hidetoshi Seto

Hi.

Quoted 7 lines

Are you meaning

synchronous MCA is caused within an execution context, for example
a process is doing a load and hits an exception

whereas a asynchronous MCA could happen when a line is written back to main memory and this could happen outside of the process's context ?

That's right.

Thus, the platform should use data poisoning to get capacity of sending synchronous MCA instead of asynchronous MCA, even though the poisoning is not the best way.

Thanks.

------

H.Seto <[email protected]>