Question on Error Injection
A message that is not in the archive (sent off-list or not yet received): [email protected]
I am playing around with ndctl inject-error and have a few questions around the behavior of the application when an error occurs. After successfully injecting error with --no-notify, I am able to read and write to the namespace device with no problems. For e.g.:
# ndctl inject-error --block=35000 --count=10 --no-notify namespace2.0{
"dev":"namespace2.0",
"mode":"raw",
"size":17179869184,
"blockdev":"pmem2"
}# dd if=/dev/pmem2 of=/tmp/pmem-dump bs=512 count=10 seek=35000 oflag=direct 10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.0128088 s, 400 kB/s
# pwd /sys/block/pmem2 # cat badblocks # ----------> empty badblock list
[Question] Shouldn't my "dd" get a SIGBUS (default machine-check handling) when it encounters badblocks that its not aware of (no-notify) ?
I tried to do both reading and writing to badblocks and things just work. If I scrub my nvdimm's (ndctl start-scrub) and the badblocks show up in device badblock list (/sys/block/pmem/badblocks) but dd can still work and writing the blocks clears out the badblock list:
# cat /sys/block/pmem2/badblocks35000 10
# dd if=/dev/zero of=/dev/pmem2 bs=512 count=10 seek=35000 oflag=direct 10+0 records in 10+0 records out
- KK
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Verma, Vishal L
maintainerQuoted 14 lines · Kamal Kakri
I am playing around with ndctl inject-error and have a few questions around the behavior of the application when an error occurs. After successfully injecting error with --no-notify, I am able to read and write to the namespace device with no problems. For e.g.:
# ndctl inject-error --block=35000 --count=10 --no-notify namespace2.0{ "dev":"namespace2.0", "mode":"raw", "size":17179869184, "blockdev":"pmem2" }
Quoted 1 line
# dd if=/dev/pmem2 of=/tmp/pmem-dump bs=512 count=10 seek=35000 oflag=direct
I think you want 'skip=35000' here instead of seek= to read from that offset in the input.
Quoted 8 lines
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.0128088 s, 400 kB/s# pwd /sys/block/pmem2 # cat badblocks # ----------> empty badblock list
With --no-notify badblocks is expected to be empty, as ACPI will not notify the OS of new errors.
Quoted 5 lines
[Question] Shouldn't my "dd" get a SIGBUS (default machine-check handling) when it encounters badblocks that its not aware of (no- notify) ?
Yes it should - I'd be curious to see if you still don't get a machine check with the seek/skip fix above.
Quoted 6 lines
I tried to do both reading and writing to badblocks and things just work. If I scrub my nvdimm's (ndctl start-scrub) and the badblocks show up in device badblock list (
/sys/block/pmem/badblocks) but dd can still work and writing the blocks clears out the badblock list:
Quoted 6 lines
# cat /sys/block/pmem2/badblocks35000 10# dd if=/dev/zero of=/dev/pmem2 bs=512 count=10 seek=35000 oflag=direct 10+0 records in 10+0 records out
Writing with O_DIRECT is the canonical way to clear errors - what you might see here is a corrected machine check notification in your kernel logs (CMCI), but that is just a notification that the platform has handled the error and no action is required.
-Vishal
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
A message that is not in the archive (sent off-list or not yet received): 873fd52419a7561f983927597dfb6fb613fc937e.camel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
My device has errors injected:
# ndctl inject-error --status namespace2.0{
"badblocks":[
{
"block":35000,
"count":10
}
]
}No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/s
Kernel doesn't know of the badblocks yet so this should have resulted in sigbus for dd:
# cat /sys/block/pmem2/badblocks# I dont have mcelog daemon running but there is no error in /var/log/messages for pmem device. Is there some setting/config that I am missing ?
-KK
On Thursday, January 3, 2019, 1:39:44 PM EST, Verma, Vishal L <[email protected]> wrote:
Quoted 14 lines · Kamal Kakri
I am playing around with ndctl inject-error and have a few questions around the behavior of the application when an error occurs. After successfully injecting error with --no-notify, I am able to read and write to the namespace device with no problems. For e.g.:
# ndctl inject-error --block=35000 --count=10 --no-notify namespace2.0{ "dev":"namespace2.0", "mode":"raw", "size":17179869184, "blockdev":"pmem2" }
Quoted 1 line
# dd if=/dev/pmem2 of=/tmp/pmem-dump bs=512 count=10 seek=35000 oflag=direct
I think you want 'skip=35000' here instead of seek= to read from that offset in the input.
Quoted 8 lines
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.0128088 s, 400 kB/s# pwd /sys/block/pmem2 # cat badblocks # ----------> empty badblock list
With --no-notify badblocks is expected to be empty, as ACPI will not notify the OS of new errors.
Quoted 5 lines
[Question] Shouldn't my "dd" get a SIGBUS (default machine-check handling) when it encounters badblocks that its not aware of (no- notify) ?
Yes it should - I'd be curious to see if you still don't get a machine check with the seek/skip fix above.
Quoted 6 lines
I tried to do both reading and writing to badblocks and things just work. If I scrub my nvdimm's (ndctl start-scrub) and the badblocks show up in device badblock list (
/sys/block/pmem/badblocks) but dd can still work and writing the blocks clears out the badblock list:
Quoted 6 lines
# cat /sys/block/pmem2/badblocks35000 10# dd if=/dev/zero of=/dev/pmem2 bs=512 count=10 seek=35000 oflag=direct 10+0 records in 10+0 records out
Writing with O_DIRECT is the canonical way to clear errors - what you might see here is a corrected machine check notification in your kernel logs (CMCI), but that is just a notification that the platform has handled the error and no action is required.
-Vishal
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Verma, Vishal L
maintainerQuoted 11 lines · Kamal Kakri
My device has errors injected:
# ndctl inject-error --status namespace2.0 { "badblocks":[ { "block":35000, "count":10 } ] }
Quoted 5 lines
No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/s
Did you ever read from /dev/pmem2 before injecting the error? There is a possibility that the page is already present in the page cache and the read gets serviced from there. You can set iflag=direct to ensure you're reading from the device.
Other than that, there /should/ have been an MCE/sigbus in this case. I'd check with your hardware/platform vendor to ensure machine checks are available, and to ensure that injecting error does result in a memory error/poison consumption by the CPU.
Quoted 77 lines
Kernel doesn't know of the badblocks yet so this should have resulted in sigbus for dd:
# cat /sys/block/pmem2/badblocks #I dont have mcelog daemon running but there is no error in
/var/log/messagesfor pmem device. Is there some setting/config that I am missing ?-KK
On Thursday, January 3, 2019, 1:39:44 PM EST, Verma, Vishal L < [email protected]> wrote:
Quoted 1 line · Kamal Kakri
I am playing around with ndctl inject-error and have a few
questions
Quoted 13 lines
around the behavior of the application when an error occurs. After successfully injecting error with --no-notify, I am able to read and write to the namespace device with no problems. For e.g.:
# ndctl inject-error --block=35000 --count=10 --no-notify namespace2.0{ "dev":"namespace2.0", "mode":"raw", "size":17179869184, "blockdev":"pmem2" }Quoted 1 line
# dd if=/dev/pmem2 of=/tmp/pmem-dump bs=512 count=10 seek=35000oflag=direct
I think you want 'skip=35000' here instead of seek= to read from that offset in the input.
Quoted 8 lines
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.0128088 s, 400 kB/s# pwd /sys/block/pmem2 # cat badblocks # ----------> empty badblock listWith --no-notify badblocks is expected to be empty, as ACPI will not notify the OS of new errors.
Quoted 5 lines
[Question] Shouldn't my "dd" get a SIGBUS (default machine-check handling) when it encounters badblocks that its not aware of (no- notify) ?
Yes it should - I'd be curious to see if you still don't get a machine
check with the seek/skip fix above.Quoted 6 lines
I tried to do both reading and writing to badblocks and things just work. If I scrub my nvdimm's (ndctl start-scrub) and the badblocks show up in device badblock list (
/sys/block/pmem/badblocks) but dd can still work and writing the blocks clears out the badblock list:Quoted 6 lines
# cat /sys/block/pmem2/badblocks35000 10# dd if=/dev/zero of=/dev/pmem2 bs=512 count=10 seek=35000 oflag=direct 10+0 records in 10+0 records out
Quoted 9 lines
Writing with
O_DIRECTis the canonical way to clear errors - what you might see here is a corrected machine check notification in your kernel
logs (CMCI), but that is just a notification that the platform has handled the error and no action is required.-Vishal
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 33 lines
-----Original Message-----
From: Linux-nvdimm <[email protected]> On Behalf Of Verma, Vishal L
Sent: Thursday, January 3, 2019 3:27 PM
To: [email protected]; [email protected]
Subject: Re: Question on Error InjectionQuoted 11 lines · Kamal Kakri
My device has errors injected:
# ndctl inject-error --status namespace2.0 { "badblocks":[ { "block":35000, "count":10 } ] }Quoted 5 lines
No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/sDid you ever read from
/dev/pmem2before injecting the error? There is a possibility that the page is already present in the page cache and the read gets serviced from there. You can set iflag=direct to ensure you're reading from the device.
Quoted 4 lines
Other than that, there /should/ have been an MCE/sigbus in this case. I'd check with your hardware/platform vendor to ensure machine checks are available, and to ensure that injecting error does result in a memory error/poison consumption by the CPU.
An application like dd making traditional read() calls should see them fail and report it this:
dd: error reading '/dev/pmem2': Input/output error
The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved.
---
Robert Elliott, HPE Persistent Memory
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
A message that is not in the archive (sent off-list or not yet received): AT5PR8401MB1169B2838F6DB72BC62B308DAB8D0-TASIVIpwxD5P8eCFU2xCp5pmD3emsgjRq6DjORPBYUW4xSMpw2roznzNABE0Ld/[email protected]
Robert,
What do you mean by:
"The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved." Can you elaborate with an example ?
-KK
Quoted 31 lines · Robert (Persistent Memory)
-----Original Message-----
From: Linux-nvdimm <[email protected]> On Behalf Of Verma, Vishal L
Sent: Thursday, January 3, 2019 3:27 PM
To: [email protected]; [email protected]
Subject: Re: Question on Error InjectionQuoted 11 lines · Kamal Kakri
My device has errors injected:
# ndctl inject-error --status namespace2.0 { "badblocks":[ { "block":35000, "count":10 } ] }Quoted 5 lines
No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/sDid you ever read from
/dev/pmem2before injecting the error? There is a possibility that the page is already present in the page cache and the read gets serviced from there. You can set iflag=direct to ensure you're reading from the device.
Quoted 4 lines
Other than that, there /should/ have been an MCE/sigbus in this case. I'd check with your hardware/platform vendor to ensure machine checks are available, and to ensure that injecting error does result in a memory error/poison consumption by the CPU.
An application like dd making traditional read() calls should see them fail and report it this:
dd: error reading '/dev/pmem2': Input/output error
The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved.
---
Robert Elliott, HPE Persistent Memory
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 6 lines · Kamal Kakri
What do you mean by:
"The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved."
Can you elaborate with an example ?
DAX (direct access). If the application uses mmap() of a file or device so it can do loads directly from persistent memory addresses and it tries to load from an address with an uncorrectable error, the CPU cannot complete that instruction without causing data corruption. There's no data value that means "this is bad data." So, the only remedy to unhang that thread is for the kernel to send a signal to the application; it might decide it doesn't really need that data after all and do something else, or might terminate if it doesn't know how to recover.
---
Robert Elliott, HPE Persistent Memory
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
A message that is not in the archive (sent off-list or not yet received): AT5PR8401MB11699659E6B9E6EF513248C1AB8E0-TASIVIpwxD5P8eCFU2xCp5pmD3emsgjRq6DjORPBYUW4xSMpw2roznzNABE0Ld/[email protected]
Will it be OK to say that the behavior shall be same irrespective of if its load or store to a memory location with uncorrectable error (UCE) ?
I see errors being reported at the granularity of blocks (badblock list). Is SIGBUS generated when the bad location is accessed or if the page (or block) having bad location is accessed?
- KK
On Thursday, January 3, 2019, 9:04:09 PM EST, Elliott, Robert (Persistent Memory) <[email protected]> wrote:
Quoted 6 lines · Kamal Kakri
What do you mean by:
"The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved."
Can you elaborate with an example ?
DAX (direct access). If the application uses mmap() of a file or device so it can do loads directly from persistent memory addresses and it tries to load from an address with an uncorrectable error, the CPU cannot complete that instruction without causing data corruption. There's no data value that means "this is bad data." So, the only remedy to unhang that thread is for the kernel to send a signal to the application; it might decide it doesn't really need that data after all and do something else, or might terminate if it doesn't know how to recover.
---
Robert Elliott, HPE Persistent Memory
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 2 lines · Kamal Kakri
Will it be OK to say that the behavior shall be same irrespective of if its load or store to a memory location with uncorrectable error (UCE) ?
No. Machine check exceptions occur on the consumption of poison (UCE), stores don't consume poison.
Quoted 1 line
I see errors being reported at the granularity of blocks (badblock list). Is SIGBUS generated when the bad location is accessed or if the page (or block) having bad location is accessed?
SIGBUS is triggered preemptively if the kernel can determine that a badblock exists in a page it is about to be faulted-in to the process. It also triggers in the recovery path after a process has consumed poison and the kernel was able to contain it by killing that affected process.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
A message that is not in the archive (sent off-list or not yet received): CAPcyv4giNsLO_c2fPmnhsQAetYSinuTMFKHFMoEo8cGYqCfU5A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Pardon my ignorance but Im still a little bit confused.
Are these statements correct?
1) Reading from a memory location (mmaped) with uncorrectable AND unknown error (also called as latent error) results in a machine-check (which usually results in a SIGBUS , default action in /sys/devices/system/machinecheck/machinecheckN).
2) Reading from a memory location (mmaped) with known error (badblock list) also results in a SIGBUS.
3) Reading via file interface (read()) will result in IO error returned in "both" the above cases.
4) Writing, however, in all the above cases does not result in any error. -KK
On Monday, January 7, 2019, 5:13:25 PM EST, Dan Williams <[email protected]> wrote:
Quoted 2 lines · Kamal Kakri
Will it be OK to say that the behavior shall be same irrespective of if its load or store to a memory location with uncorrectable error (UCE) ?
No. Machine check exceptions occur on the consumption of poison (UCE), stores don't consume poison.
Quoted 1 line
I see errors being reported at the granularity of blocks (badblock list). Is SIGBUS generated when the bad location is accessed or if the page (or block) having bad location is accessed?
SIGBUS is triggered preemptively if the kernel can determine that a badblock exists in a page it is about to be faulted-in to the process. It also triggers in the recovery path after a process has consumed poison and the kernel was able to contain it by killing that affected process.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
For mailing lists, please use plaintext rather than HTML emails, and don’t top-post.
Quoted 8 lines
Are these statements correct?
1) Reading from a memory location (mmaped) with uncorrectable AND unknown error (also called as latent error) results in a machine-check (which usually results in a SIGBUS , default action
in/sys/devices/system/machinecheck/machinecheckN).4) Writing, however, in all the above cases does not result in any error.
When a CPU load instruction gets an uncorrectable error (i.e., the hardware cannot provide any data to complete that instruction), the CPU creates a machine check exception. The machine check handler decides what to do next. That thread is stuck and cannot proceed, so the handler must do something like:
* if the instruction was part of the kernel's memcpy_mcsafe() function (which is called by the read() system call), force the function to return an error which results in read() returning an I/O error to its caller
* if the instruction was issued by application (e.g. that had used mmap() to get direct access to persistent memory), send SIGBUS to the application. Most applications will just exit (and create a core dump). Some applications can process SIGBUS and proceed (e.g., like the kernel does with memcpy_mcsafe()).
* if the instruction was issued by the kernel outside memcpy_mcsafe(), do a panic crash of the kernel
Historically, "EP class" CPUs always crashed the kernel while "EX class" CPUs could proceed. For persistent memory, tolerance is a key CPU feature. This indicates the CPU does support proceeding:
/sys/devices/system/machinecheck/machinecheck*/tolerant:1
Quoted 2 lines
2) Reading from a memory location (mmaped) with known error (badblock list) also results in a SIGBUS.
The pmem driver consults the badblock list to avoid asking the CPU to load from a bad address and triggering a machine check; it can directly start the appropriate error handling.
Quoted 2 lines
3) Reading via file interface (
read()) will result in IO error returned in "both" the above cases.
A non-tolerant CPU will crash the kernel before read() completes. A tolerant CPU will complete read() with an error.
Quoted 1 line
4) Writing, however, in all the above cases does not result in any error.
A write of less than the ECC granularity triggers a read-modify-write, and the CPU might detect that the read failed first. This isn't an immediate problem for the store instruction, but is an early hint that there will be problems reading those other bytes in the future.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Verma, Vishal L
maintainerQuoted 36 lines · Robert (Persistent Memory)
Quoted 31 lines
-----Original Message-----
From: Linux-nvdimm <[email protected]> On Behalf Of Verma, Vishal L
Sent: Thursday, January 3, 2019 3:27 PM
To: [email protected]; [email protected]
Subject: Re: Question on Error InjectionQuoted 11 lines · Kamal Kakri
My device has errors injected:
# ndctl inject-error --status namespace2.0 { "badblocks":[ { "block":35000, "count":10 } ] }Quoted 5 lines
No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/sDid you ever read from
/dev/pmem2before injecting the error? There is a possibility that the page is already present in the page cache and the read gets serviced from there. You can set iflag=direct to ensure you're reading from the device.Quoted 4 lines
Other than that, there /should/ have been an MCE/sigbus in this case. I'd check with your hardware/platform vendor to ensure machine checks are available, and to ensure that injecting error does result in a memory error/poison consumption by the CPU.
Quoted 6 lines
An application like dd making traditional
read()calls should see them fail and report it this:
dd: error reading '/dev/pmem2': Input/output errorThe application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved.
Correct, an IO error will occur when the error is 'known', i.e. in badblocks. But in the case of injection with --no-notify, the error is a latent one, and the kernel can't prevent access to the poison location, and in this case the application will receive a SIGBUS when the page is faulted in.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 53 lines
-----Original Message-----
From: Verma, Vishal L <[email protected]>
Sent: Thursday, January 3, 2019 6:03 PM
To: [email protected]; [email protected]; Elliott, Robert (Persistent Memory) <[email protected]>
Subject: Re: Question on Error InjectionQuoted 2 lines · Robert (Persistent Memory)
Quoted 2 lines
-----Original Message-----
From: Linux-nvdimm <[email protected]> On Behalf OfVerma,
Quoted 38 lines
Quoted 29 lines
Vishal L
Sent: Thursday, January 3, 2019 3:27 PM
To: [email protected]; [email protected]
Subject: Re: Question on Error InjectionQuoted 11 lines · Kamal Kakri
My device has errors injected:
# ndctl inject-error --status namespace2.0 { "badblocks":[ { "block":35000, "count":10 } ] }Quoted 5 lines
No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/sDid you ever read from
/dev/pmem2before injecting the error? There is a possibility that the page is already present in the page cache and the read gets serviced from there. You can set iflag=direct to ensure you're reading from the device.Quoted 4 lines
Other than that, there /should/ have been an MCE/sigbus in this case. I'd check with your hardware/platform vendor to ensure machine checks are available, and to ensure that injecting error does result in a memory error/poison consumption by the CPU.
An application like dd making traditional
read()calls should see them fail and report it this:
dd: error reading '/dev/pmem2': Input/output error
Quoted 8 lines
Quoted 2 lines
The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved.
Correct, an IO error will occur when the error is 'known', i.e. in badblocks. But in the case of injection with --no-notify, the error is a latent one, and the kernel can't prevent access to the poison location, and in this case the application will receive a SIGBUS when the page is faulted in.
memcpy_mcsafe() used by the kernel pmem driver is designed to tolerate those problems. There will be a machine check so the kernel can record the bad block, but it won't be seen by the application except as read() failing.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Verma, Vishal L
maintainerQuoted 61 lines · Robert (Persistent Memory)
Quoted 52 lines
-----Original Message-----
From: Verma, Vishal L <[email protected]>
Sent: Thursday, January 3, 2019 6:03 PM
To: [email protected]; [email protected]; Elliott, Robert (Persistent Memory) <[email protected]>
Subject: Re: Question on Error InjectionQuoted 2 lines · Robert (Persistent Memory)
Quoted 2 lines
-----Original Message-----
From: Linux-nvdimm <[email protected]> On Behalf OfVerma,
Quoted 38 lines
Quoted 29 lines
Vishal L
Sent: Thursday, January 3, 2019 3:27 PM
To: [email protected]; [email protected]
Subject: Re: Question on Error InjectionQuoted 11 lines · Kamal Kakri
My device has errors injected:
# ndctl inject-error --status namespace2.0 { "badblocks":[ { "block":35000, "count":10 } ] }Quoted 5 lines
No problem reading from the bad offsets:
# dd if=/dev/pmem2 of=/tmp/pmem_out bs=512 count=10 skip=35000 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.000108226 s, 47.3 MB/sDid you ever read from
/dev/pmem2before injecting the error? There is a possibility that the page is already present in the page cache and the read gets serviced from there. You can set iflag=direct to ensure you're reading from the device.Quoted 4 lines
Other than that, there /should/ have been an MCE/sigbus in this case. I'd check with your hardware/platform vendor to ensure machine checks are available, and to ensure that injecting error does result in a memory error/poison consumption by the CPU.
An application like dd making traditional
read()calls should see them fail and report it this:
dd: error reading '/dev/pmem2': Input/output errorQuoted 8 lines
Quoted 2 lines
The application itself shouldn't be terminated with SIGBUS - that's for an application doing memory accesses that cannot be resolved.
Correct, an IO error will occur when the error is 'known', i.e. in badblocks. But in the case of injection with --no-notify, the error is a latent one, and the kernel can't prevent access to the poison location, and in this case the application will receive a SIGBUS when the page is faulted in.
Quoted 5 lines
memcpy_mcsafe()used by the kernel pmem driver is designed to tolerate those problems. There will be a machine check so the kernel can record the bad block, but it won't be seen by the application except asread()failing.
Ah yes I keep forgetting about that :)
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm