rslib: RS decoder is severely broken
v2 cover rslib: RS decoder is severely broken
This is the second revision of this series that fixes the bugs/flaws in the RS decoder. This addresses all of Thomas' comments/suggestions.
Changes in v2:
- Replace code that requires the FPU (one small thing in test_rslib.c)
- More verbose changelog (patch 2/7)
- Clarifying comment (patch 7/7)
- Fixed some small whitespace issues (patches 1 and 5)
------
The Reed_Solomon library used in the kernel is based on Phil Karn's fec library. When playing with this library I found a couple of bugs. It turn out that all of these bugs, and some additional flaws, are present in rslib.
The Reed-Solomon decoder has several bugs/flaws:
- Decoding of shortened codes is broken. The decoder only works as expected if there are no erasures.
- The decoder sometimes fails silently, i.e. it announces success but returns a word that is not a codeword.
- The return value of the decoder is incoherent with respect to how fixed erasures are counted. If the word to be decoded is a codeword, then the decoder always returns zero even if some erasures are given. On the other hand, if the word to be decoded contains errors, then the number of erasures is always included in the count of corrected symbols. So the decoder handles erasures without symbol corruption inconsistently. This inconsistency probably doesn't affect anyone using the decoder, but it is inconsistent with the documentation.
- The error positions returned in eras_pos include all erasures, but the corrections are only set in the correction buffer if there actually is a symbol error. So if there are erasures without symbol corruption, then the correction buffer will contain errors (unless initialized to zero before calling the decoder) or some values will be unset (if the correction buffer is uninitialized).
- Assumes that the syndromes provided by callers are in index form. This is simply undocumented.
- When correcting data in-place the decoder does not correct errors in the parity. On the other hand, when returning the errors in correction buffers, errors in the parity are included.
This series provides a module with tests for rslib and fixes all the bugs/flaws. I am not sure that the provided self tests are written in the 'right way'. I just looked at other self tests in lib and implemented something similar.
The fixes are tested with the self tests. They should probably also be tested with drivers etc. that use rslib, but it is unclear to me how to do this.
I looked a bit on two of the drivers that use rslib:
drivers/mtd/nand/raw/cafe_nand.cdrivers/mtd/nand/raw/diskonchip.c
Both of them seem to do some additional error checking after calling decode_rs16. Maybe this is needed because of the bugs in the decoder?
Ferdinand Blomqvist (7):
rslib: Add tests for the encoder and decoder
rslib: Fix decoding of shortened codes
rslib: decode_rs: Fix length parameter check
rslib: decode_rs: code cleanup
rslib: Fix handling of of caller provided syndrome
rslib: Update documentation
rslib: Fix remaining decoder flaws
lib/Kconfig.debug | 12 + lib/reed_solomon/Makefile | 2 +- lib/reed_solomon/decode_rs.c | 115 +++++-- lib/reed_solomon/reed_solomon.c | 12 +- lib/reed_solomon/test_rslib.c | 516 ++++++++++++++++++++++++++++++++ 5 files changed, 622 insertions(+), 35 deletions(-) create mode 100644 lib/reed_solomon/test_rslib.c
Signature · 1 line
2.17.2
PATCH v2 1/7 rslib: Add tests for the encoder and decoder
A Reed-Solomon code with minimum distance d can correct any error and erasure pattern that satisfies 2 * #error + #erasures < d. If the error correction capacity is exceeded, then correct decoding cannot be guaranteed. The decoder must, however, return a valid codeword or report failure.
There are two main tests:
- Check for correct behaviour up to the error correction capacity
- Check for correct behaviour beyond error corrupted capacity
Both tests are simple:
1. Generate random data
2. Encode data with the chosen code
3. Add errors and erasures to data
4. Decode the corrupted word
5. Check for correct behaviour
When testing up to capacity we test for:
- Correct decoding
- Correct return value (i.e. the number of corrected symbols)
- That the returned error positions are correct
There are two kinds of erasures; the erased symbol can be corrupted or not. When counting the number of corrected symbols, erasures without symbol corruption should not be counted. Similarly, the returned error positions should only include positions where a correction is necessary.
We run the up to capacity tests for three different interfaces of decode_rs:
- Use the correction buffers
- Use the correction buffers with syndromes provided by the caller
- Error correction in place (does not check the error positions)
When testing beyond capacity we test for silent failures. A silent failure is when the decoder returns success but the returned word is not a valid codeword.
There are a couple of options for the tests:
- Verbosity.
- Whether to test for correct behaviour beyond capacity. Default is to test beyond capacity.
- Whether to allow erasures without symbol corruption. Defaults to yes.
Note that the tests take a couple of minutes to complete.
--- lib/Kconfig.debug | 12 + lib/reed_solomon/Makefile | 2 +- lib/reed_solomon/test_rslib.c | 516 ++++++++++++++++++++++++++++++++++ 3 files changed, 529 insertions(+), 1 deletion(-) create mode 100644 lib/reed_solomon/test_rslib.c
lib/Kconfig.debug +12 -0
@@ -1754,6 +1754,18 @@ config RBTREE_TEST A benchmark measuring the performance of the rbtree library. Also includes rbtree invariant checks. +config REED_SOLOMON_TEST + tristate "Reed-Solomon library test" + depends on DEBUG_KERNEL || m + select REED_SOLOMON + select REED_SOLOMON_ENC16 + select REED_SOLOMON_DEC16 + help + This option enables the self-test function of rslib at boot, + or at module load time. + + If unsure, say N. + config INTERVAL_TREE_TEST tristate "Interval tree test" depends on DEBUG_KERNEL
lib/reed_solomon/Makefile +1 -1
@@ -4,4 +4,4 @@ # obj-$(CONFIG_REED_SOLOMON) += reed_solomon.o - +obj-$(CONFIG_REED_SOLOMON_TEST) += test_rslib.o
lib/reed_solomon/test_rslib.c new file +516 -0
516 lines not on this page · raw message
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: Add tests for the encoder and decoder
rslib: Add tests for the encoder and decoder
Commit-ID: 4b4f3accd80304781c648b26ce4d53df082a4087 Gitweb: https://git.kernel.org/tip/4b4f3accd80304781c648b26ce4d53df082a4087 Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:33 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:45 +0200
rslib: Add tests for the encoder and decoder
A Reed-Solomon code with minimum distance d can correct any error and erasure pattern that satisfies 2 * #error + #erasures < d. If the error correction capacity is exceeded, then correct decoding cannot be guaranteed. The decoder must, however, return a valid codeword or report failure.
There are two main tests:
- Check for correct behaviour up to the error correction capacity
- Check for correct behaviour beyond error corrupted capacity
Both tests are simple:
1. Generate random data
2. Encode data with the chosen code
3. Add errors and erasures to data
4. Decode the corrupted word
5. Check for correct behaviour
When testing up to capacity we test for:
- Correct decoding
- Correct return value (i.e. the number of corrected symbols)
- That the returned error positions are correct
There are two kinds of erasures; the erased symbol can be corrupted or not. When counting the number of corrected symbols, erasures without symbol corruption should not be counted. Similarly, the returned error positions should only include positions where a correction is necessary.
We run the up to capacity tests for three different interfaces of decode_rs:
- Use the correction buffers
- Use the correction buffers with syndromes provided by the caller
- Error correction in place (does not check the error positions)
When testing beyond capacity test for silent failures. A silent failure is when the decoder returns success but the returned word is not a valid codeword.
There are a couple of options for the tests:
- Verbosity.
- Whether to test for correct behaviour beyond capacity. Default is to test beyond capacity.
- Whether to allow erasures without symbol corruption. Defaults to yes.
Note that the tests take a couple of minutes to complete.
--- lib/Kconfig.debug | 12 + lib/reed_solomon/Makefile | 2 +- lib/reed_solomon/test_rslib.c | 518 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 531 insertions(+), 1 deletion(-)
lib/Kconfig.debug +12 -0
@@ -1754,6 +1754,18 @@ config RBTREE_TEST A benchmark measuring the performance of the rbtree library. Also includes rbtree invariant checks. +config REED_SOLOMON_TEST + tristate "Reed-Solomon library test" + depends on DEBUG_KERNEL || m + select REED_SOLOMON + select REED_SOLOMON_ENC16 + select REED_SOLOMON_DEC16 + help + This option enables the self-test function of rslib at boot, + or at module load time. + + If unsure, say N. + config INTERVAL_TREE_TEST tristate "Interval tree test" depends on DEBUG_KERNEL
lib/reed_solomon/Makefile +1 -1
@@ -4,4 +4,4 @@ # obj-$(CONFIG_REED_SOLOMON) += reed_solomon.o - +obj-$(CONFIG_REED_SOLOMON_TEST) += test_rslib.o
lib/reed_solomon/test_rslib.c new file +518 -0
518 lines not on this page · raw message
Re: [PATCH v2 1/7] rslib: Add tests for the encoder and decoder
In reply to Ferdinand BlomqvistHi Ferdinand,
Quoted 7 lines · Ferdinand Blomqvist
A Reed-Solomon code with minimum distance d can correct any error and erasure pattern that satisfies 2 * #error + #erasures < d. If the error correction capacity is exceeded, then correct decoding cannot be guaranteed. The decoder must, however, return a valid codeword or report failure.
Note that the tests take a couple of minutes to complete.
On which hardware? ;-)
JFTR, the test succeeded on m68k, after 6 hours and 12 minutes of runtime on an emulated Atari Falcon (ARAnyM hosted by an i7-4770).
So far I have no plans to run it on bare metal.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 1/7] rslib: Add tests for the encoder and decoder
Quoted 10 lines · Geert Uytterhoeven
Hi Ferdinand,
Quoted 5 lines · Ferdinand Blomqvist
A Reed-Solomon code with minimum distance d can correct any error and erasure pattern that satisfies 2 * #error + #erasures < d. If the error correction capacity is exceeded, then correct decoding cannot be guaranteed. The decoder must, however, return a valid codeword or report failure.
Quoted 7 lines
Quoted 1 line
Note that the tests take a couple of minutes to complete.
On which hardware? ;-)
JFTR, the test succeeded on m68k, after 6 hours and 12 minutes of runtime on an emulated Atari Falcon (ARAnyM hosted by an i7-4770).
So far I have no plans to run it on bare metal.
It took about 4 minutes on a 10 year old lame desktop and was way faster on one of the big irons.
Thanks,
tglx
PATCH v2 2/7 rslib: Fix decoding of shortened codes
In reply to Ferdinand BlomqvistThe decoding of shortenend codes is broken. It only works as expected if there are no erasures.
When decoding with erasures, Lambda (the error and erasure locator polynomial) is initialized from the given erasure positions. The pad parameter is not accounted for by the initialisation code, and hence Lambda is initialized from incorrect erasure positions. The fix is to adjust the erasure positions by the supplied pad.
--- lib/reed_solomon/decode_rs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/reed_solomon/decode_rs.c +2 -2
@@ -99,9 +99,9 @@if (no_eras >0 ) {/* Init lambda to be the erasure locator polynomial */ lambda[1 ] = alpha_to[rs_modnn (rs, - prim * (nn -1 - eras_pos[0 ]))]; + prim * (nn -1 - (eras_pos[0 ] + pad)))];for (i =1 ; i < no_eras; i++) { - u =rs_modnn (rs, prim * (nn -1 - eras_pos[i])); + u =rs_modnn (rs, prim * (nn -1 - (eras_pos[i] + pad)));for (j = i +1 ; j >0 ; j--) { tmp = index_of[lambda[j -1 ]];if (tmp != nn) {
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: Fix decoding of shortened codes
rslib: Fix decoding of shortened codes
Commit-ID: 2034a42d1747fc1e1eeef2c6f1789c4d0762cb9c Gitweb: https://git.kernel.org/tip/2034a42d1747fc1e1eeef2c6f1789c4d0762cb9c Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:34 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:45 +0200
rslib: Fix decoding of shortened codes
The decoding of shortenend codes is broken. It only works as expected if there are no erasures.
When decoding with erasures, Lambda (the error and erasure locator polynomial) is initialized from the given erasure positions. The pad parameter is not accounted for by the initialisation code, and hence Lambda is initialized from incorrect erasure positions.
The fix is to adjust the erasure positions by the supplied pad.
--- lib/reed_solomon/decode_rs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/reed_solomon/decode_rs.c +2 -2
@@ -99,9 +99,9 @@if (no_eras >0 ) {/* Init lambda to be the erasure locator polynomial */ lambda[1 ] = alpha_to[rs_modnn (rs, - prim * (nn -1 - eras_pos[0 ]))]; + prim * (nn -1 - (eras_pos[0 ] + pad)))];for (i =1 ; i < no_eras; i++) { - u =rs_modnn (rs, prim * (nn -1 - eras_pos[i])); + u =rs_modnn (rs, prim * (nn -1 - (eras_pos[i] + pad)));for (j = i +1 ; j >0 ; j--) { tmp = index_of[lambda[j -1 ]];if (tmp != nn) {
PATCH v2 3/7 rslib: decode_rs: Fix length parameter check
In reply to Ferdinand BlomqvistThe length of the data load must be at least one. Or in other words, there must be room for at least 1 data and nroots parity symbols after shortening the RS code.
--- lib/reed_solomon/decode_rs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/reed_solomon/decode_rs.c +1 -1
@@ -39,7 +39,7 @@/* Check length parameter for validity */ pad = nn - nroots - len; -BUG_ON (pad <0 || pad >= nn); +BUG_ON (pad <0 || pad >= nn - nroots);/* Does the caller provide the syndrome ? */ if (s !=NULL )
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: decode_rs: Fix length parameter check
rslib: decode_rs: Fix length parameter check
Commit-ID: a343536f8f482be6932803a023f46d0fa723ae56 Gitweb: https://git.kernel.org/tip/a343536f8f482be6932803a023f46d0fa723ae56 Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:35 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:46 +0200
rslib: decode_rs: Fix length parameter check
The length of the data load must be at least one. Or in other words, there must be room for at least 1 data and nroots parity symbols after shortening the RS code.
--- lib/reed_solomon/decode_rs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/reed_solomon/decode_rs.c +1 -1
@@ -39,7 +39,7 @@/* Check length parameter for validity */ pad = nn - nroots - len; -BUG_ON (pad <0 || pad >= nn); +BUG_ON (pad <0 || pad >= nn - nroots);/* Does the caller provide the syndrome ? */ if (s !=NULL )
PATCH v2 4/7 rslib: decode_rs: code cleanup
In reply to Ferdinand BlomqvistNothing useful was done after the finish label when count is negative so return directly instead of jumping to finish.
--- lib/reed_solomon/decode_rs.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
lib/reed_solomon/decode_rs.c +2 -5
@@ -88,8 +88,7 @@/* if syndrome is zero, data[] is a codeword and there are no * errors to correct. So return data[] unmodified */ - count =0 ; - goto finish; +return 0 ; } decode:
@@ -202,8 +201,7 @@* deg(lambda) unequal to number of roots => uncorrectable * error detected */ - count = -EBADMSG ; - goto finish; +return -EBADMSG ; }/* * Compute err+eras evaluator poly omega(x) = s(x)*lambda(x) (modulo
@@ -261,7 +259,6 @@ } } -finish:if (eras_pos !=NULL ) {for (i =0 ; i < count; i++) eras_pos[i] = loc[i] - pad;
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: decode_rs: Code cleanup
rslib: decode_rs: Code cleanup
Commit-ID: 647cc9ece63fdba573a31bdafa54fb2d388c3c83 Gitweb: https://git.kernel.org/tip/647cc9ece63fdba573a31bdafa54fb2d388c3c83 Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:36 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:46 +0200
rslib: decode_rs: Code cleanup
Nothing useful was done after the finish label when count is negative so return directly instead of jumping to finish.
--- lib/reed_solomon/decode_rs.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
lib/reed_solomon/decode_rs.c +2 -5
@@ -88,8 +88,7 @@/* if syndrome is zero, data[] is a codeword and there are no * errors to correct. So return data[] unmodified */ - count =0 ; - goto finish; +return 0 ; } decode:
@@ -202,8 +201,7 @@* deg(lambda) unequal to number of roots => uncorrectable * error detected */ - count = -EBADMSG ; - goto finish; +return -EBADMSG ; }/* * Compute err+eras evaluator poly omega(x) = s(x)*lambda(x) (modulo
@@ -261,7 +259,6 @@ } } -finish:if (eras_pos !=NULL ) {for (i =0 ; i < count; i++) eras_pos[i] = loc[i] - pad;
PATCH v2 5/7 rslib: Fix handling of of caller provided syndrome
In reply to Ferdinand BlomqvistCheck if the syndrome provided by the caller is zero, and act accordingly.
--- lib/reed_solomon/decode_rs.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
lib/reed_solomon/decode_rs.c +12 -2
@@ -42,8 +42,18 @@BUG_ON (pad <0 || pad >= nn - nroots);/* Does the caller provide the syndrome ? */ -if (s !=NULL ) - goto decode; +if (s !=NULL ) { +for (i =0 ; i < nroots; i++) { +/* The syndrome is in index form, +* so nn represents zero +*/ +if (s[i] != nn) + goto decode; + } + +/* syndrome is zero, no errors to correct */ +return 0 ; + }/* form the syndromes; i.e., evaluate data(x) at roots of * g(x) */
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: Fix handling of of caller provided syndrome
rslib: Fix handling of of caller provided syndrome
Commit-ID: ef4d6a8556b637ad27c8c2a2cff1dda3da38e9a9 Gitweb: https://git.kernel.org/tip/ef4d6a8556b637ad27c8c2a2cff1dda3da38e9a9 Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:37 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:47 +0200
rslib: Fix handling of of caller provided syndrome
Check if the syndrome provided by the caller is zero, and act accordingly.
--- lib/reed_solomon/decode_rs.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
lib/reed_solomon/decode_rs.c +12 -2
@@ -42,8 +42,18 @@BUG_ON (pad <0 || pad >= nn - nroots);/* Does the caller provide the syndrome ? */ -if (s !=NULL ) - goto decode; +if (s !=NULL ) { +for (i =0 ; i < nroots; i++) { +/* The syndrome is in index form, +* so nn represents zero +*/ +if (s[i] != nn) + goto decode; + } + +/* syndrome is zero, no errors to correct */ +return 0 ; + }/* form the syndromes; i.e., evaluate data(x) at roots of * g(x) */
PATCH v2 6/7 rslib: Update documentation
In reply to Ferdinand BlomqvistThe decoder returns the number of corrected symbols, not bits. The caller provided syndrome must be in index form.
--- lib/reed_solomon/reed_solomon.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
lib/reed_solomon/reed_solomon.c +8 -4
@@ -340,7 +340,8 @@ EXPORT_SYMBOL_GPL(encode_rs8);* @data: data field of a given type * @par: received parity data field * @len: data length -* @s: syndrome data field (if NULL, syndrome is calculated) +* @s: syndrome data field, must be in index form +* (if NULL, syndrome is calculated) * @no_eras: number of erasures * @eras_pos: position of erasures, can be NULL * @invmsk: invert data mask (will be xored on data, not on parity!)
@@ -354,7 +355,8 @@ EXPORT_SYMBOL_GPL(encode_rs8);* decoding, so the caller has to ensure that decoder invocations are * serialized. * -* Returns the number of corrected bits or -EBADMSG for uncorrectable errors. +* Returns the number of corrected symbols or -EBADMSG for uncorrectable +* errors. The count includes errors in the parity. */ int decode_rs8(struct rs_control *rsc, uint8_t *data, uint16_t *par, int len, uint16_t *s, int no_eras, int *eras_pos, uint16_t invmsk,
@@ -391,7 +393,8 @@ EXPORT_SYMBOL_GPL(encode_rs16);* @data: data field of a given type * @par: received parity data field * @len: data length -* @s: syndrome data field (if NULL, syndrome is calculated) +* @s: syndrome data field, must be in index form +* (if NULL, syndrome is calculated) * @no_eras: number of erasures * @eras_pos: position of erasures, can be NULL * @invmsk: invert data mask (will be xored on data, not on parity!)
@@ -403,7 +406,8 @@ EXPORT_SYMBOL_GPL(encode_rs16);* decoding, so the caller has to ensure that decoder invocations are * serialized. * -* Returns the number of corrected bits or -EBADMSG for uncorrectable errors. +* Returns the number of corrected symbols or -EBADMSG for uncorrectable +* errors. The count includes errors in the parity. */ int decode_rs16(struct rs_control *rsc, uint16_t *data, uint16_t *par, int len, uint16_t *s, int no_eras, int *eras_pos, uint16_t invmsk,
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: Update documentation
rslib: Update documentation
Commit-ID: 38cbae1434f8f7bbd8eaf24b29a385a4b88938fb Gitweb: https://git.kernel.org/tip/38cbae1434f8f7bbd8eaf24b29a385a4b88938fb Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:38 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:47 +0200
rslib: Update documentation
The decoder returns the number of corrected symbols, not bits. The caller provided syndrome must be in index form.
--- lib/reed_solomon/reed_solomon.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
lib/reed_solomon/reed_solomon.c +8 -4
@@ -340,7 +340,8 @@ EXPORT_SYMBOL_GPL(encode_rs8);* @data: data field of a given type * @par: received parity data field * @len: data length -* @s: syndrome data field (if NULL, syndrome is calculated) +* @s: syndrome data field, must be in index form +* (if NULL, syndrome is calculated) * @no_eras: number of erasures * @eras_pos: position of erasures, can be NULL * @invmsk: invert data mask (will be xored on data, not on parity!)
@@ -354,7 +355,8 @@ EXPORT_SYMBOL_GPL(encode_rs8);* decoding, so the caller has to ensure that decoder invocations are * serialized. * -* Returns the number of corrected bits or -EBADMSG for uncorrectable errors. +* Returns the number of corrected symbols or -EBADMSG for uncorrectable +* errors. The count includes errors in the parity. */ int decode_rs8(struct rs_control *rsc, uint8_t *data, uint16_t *par, int len, uint16_t *s, int no_eras, int *eras_pos, uint16_t invmsk,
@@ -391,7 +393,8 @@ EXPORT_SYMBOL_GPL(encode_rs16);* @data: data field of a given type * @par: received parity data field * @len: data length -* @s: syndrome data field (if NULL, syndrome is calculated) +* @s: syndrome data field, must be in index form +* (if NULL, syndrome is calculated) * @no_eras: number of erasures * @eras_pos: position of erasures, can be NULL * @invmsk: invert data mask (will be xored on data, not on parity!)
@@ -403,7 +406,8 @@ EXPORT_SYMBOL_GPL(encode_rs16);* decoding, so the caller has to ensure that decoder invocations are * serialized. * -* Returns the number of corrected bits or -EBADMSG for uncorrectable errors. +* Returns the number of corrected symbols or -EBADMSG for uncorrectable +* errors. The count includes errors in the parity. */ int decode_rs16(struct rs_control *rsc, uint16_t *data, uint16_t *par, int len, uint16_t *s, int no_eras, int *eras_pos, uint16_t invmsk,
PATCH v2 7/7 rslib: Fix remaining decoder flaws
In reply to Ferdinand BlomqvistThe decoder is flawed in the following ways:
- The decoder sometimes fails silently, i.e. it announces success but returns a word that is not a codeword.
- The return value of the decoder is incoherent with respect to how fixed erasures are counted. If the word to be decoded is a codeword, then the decoder always returns zero even if some erasures are given. On the other hand, if the word to be decoded contains errors, then the number of erasures is always included in the count of corrected symbols. So the decoder handles erasures without symbol corruption inconsistently. This inconsistency probably doesn't affect anyone using the decoder, but it is inconsistent with the documentation.
- The error positions returned in eras_pos include all erasures, but the corrections are only set in the correction buffer if there actually is a symbol error. So if there are erasures without symbol corruption, then the correction buffer will contain errors (unless initialized to zero before calling the decoder) or some values will be unset (if the correction buffer is uninitialized).
- When correcting data in-place the decoder does not correct errors in the parity. On the other hand, when returning the errors in correction buffers, errors in the parity are included.
The respective fixed are:
- The syndrome of a codeword is always zero, and the syndrome is linear, .i.e, S(x+e) = S(x) + S(e). So compute the syndrome for the error and check whether it equals the syndrome of the received word. If it does, then we have decoded to a valid codeword, otherwise we know that we have an uncorrectable error. Fortunately, some unrecoverable error conditions can be detected earlier in the decoding, which saves some processing power.
- Simply count and return the number of symbols actually corrected.
- Make sure to only return positions where symbols were corrected.
- Also fix errors in parity when correcting in-place. Another option would be to completely disregard errors in the parity, but then the interface makes it impossible to write tests that test for silent failures.
Other changes:
- Only fill the correction buffer and error position buffer if both of them are provided. Otherwise correct in place. Previously the error position buffer was always populated with the positions of the corrected errors, irrespective of whether a correction buffer was supplied or not. The rationale for this change is that there seems to be two use cases for the decoder; correct in-place or use the correction buffers. The caller does not need the positions of the corrected errors when in-place correction is used. If in-place correction is not used, then both the correction buffer and error position buffer need to be populated.
--- lib/reed_solomon/decode_rs.c | 88 ++++++++++++++++++++++++++++-------- 1 file changed, 68 insertions(+), 20 deletions(-)
lib/reed_solomon/decode_rs.c +68 -20
@@ -22,6 +22,7 @@uint16_t *index_of = rs->index_of;uint16_t u, q, tmp, num1, num2, den, discr_r, syn_error;int count =0 ; +int num_corrected;uint16_t msk = (uint16_t ) rs->nn;/*
@@ -184,6 +185,15 @@if (lambda[i] != nn) deg_lambda = i; } + +if (deg_lambda ==0 ) { +/* +* deg(lambda) is zero even though the syndrome is non-zero +* => uncorrectable error detected +*/ +return -EBADMSG ; + } +/* Find roots of error+erasure locator polynomial by Chien search */ memcpy (®[1 ], &lambda[1 ], nroots *sizeof (reg[0 ])); count =0 ;/* Number of roots of lambda(x) */
@@ -197,6 +207,12 @@ }if (q !=0 )continue ;/* Not a root */ + +if (k < pad) { +/* Impossible error location. Uncorrectable error. */ +return -EBADMSG ; + } +/* store root (index-form) and error location number */ root[count] = i; loc[count] = k;
@@ -231,7 +247,9 @@/* * Compute error values in poly-form. num1 = omega(inv(X(l))), num2 = * inv(X(l))**(fcr-1) and den = lambda_pr(inv(X(l))) all in poly-form +* Note: we reuse the buffer for b to store the correction pattern */ + num_corrected =0 ;for (j = count -1 ; j >=0 ; j--) { num1 =0 ;for (i = deg_omega; i >=0 ; i--) {
@@ -239,6 +257,13 @@ num1 ^= alpha_to[rs_modnn (rs, omega[i] + i * root[j])]; } + +if (num1 ==0 ) { +/* Nothing to correct at this position */ + b[j] =0 ; +continue ; + } + num2 = alpha_to[rs_modnn (rs, root[j] * (fcr -1 ) + nn)]; den =0 ;
@@ -250,29 +275,52 @@ i * root[j])]; } } -/* Apply error to data */ -if (num1 !=0 && loc[j] >= pad) { -uint16_t cor = alpha_to[rs_modnn (rs,index_of[num1] + - index_of[num2] + - nn - index_of[den])]; -/* Store the error correction pattern, if a -* correction buffer is available */ -if (corr) { - corr[j] = cor; - }else { -/* If a data buffer is given and the -* error is inside the message, -* correct it */ -if (data && (loc[j] < (nn - nroots))) - data[loc[j] - pad] ^= cor; - } + + b[j] = alpha_to[rs_modnn (rs, index_of[num1] + + index_of[num2] + + nn - index_of[den])]; + num_corrected++; + } + +/* +* We compute the syndrome of the 'error' and check that it matches +* the syndrome of the received word +*/ +for (i =0 ; i < nroots; i++) { + tmp =0 ; +for (j =0 ; j < count; j++) { +if (b[j] ==0 ) +continue ; + + k = (fcr + i) * prim * (nn-loc[j]-1 ); + tmp ^= alpha_to[rs_modnn (rs, index_of[b[j]] + k)]; } + +if (tmp != alpha_to[s[i]]) +return -EBADMSG ; } -if (eras_pos !=NULL ) { -for (i =0 ; i < count; i++) - eras_pos[i] = loc[i] - pad; +/* +* Store the error correction pattern, if a +* correction buffer is available +*/ +if (corr && eras_pos) { + j =0 ; +for (i =0 ; i < count; i++) { +if (b[i]) { + corr[j] = b[i]; + eras_pos[j++] = loc[i] - pad; + } + } + }else if (data && par) { +/* Apply error to data and parity */ +for (i =0 ; i < count; i++) { +if (loc[i] < (nn - nroots)) + data[loc[i] - pad] ^= b[i]; +else + par[loc[i] - pad - len] ^= b[i]; + } } -return count; +return num_corrected; }
Signature · 1 line
2.17.2
tip-bot for Ferdinand Blomqvist
merge bot[tip:core/rslib] rslib: Fix remaining decoder flaws
rslib: Fix remaining decoder flaws
Commit-ID: 991305dee585dd9e3107b2e253778ed02ee3fbd1 Gitweb: https://git.kernel.org/tip/991305dee585dd9e3107b2e253778ed02ee3fbd1 Author: Ferdinand Blomqvist <[email protected]> AuthorDate: Thu, 20 Jun 2019 17:10:39 +0300 Committer: Thomas Gleixner <[email protected]> CommitDate: Wed, 26 Jun 2019 14:55:47 +0200
rslib: Fix remaining decoder flaws
The decoder is flawed in the following ways:
- The decoder sometimes fails silently, i.e. it announces success but returns a word that is not a codeword.
- The return value of the decoder is incoherent with respect to how fixed erasures are counted. If the word to be decoded is a codeword, then the decoder always returns zero even if some erasures are given. On the other hand, if the word to be decoded contains errors, then the number of erasures is always included in the count of corrected symbols. So the decoder handles erasures without symbol corruption inconsistently. This inconsistency probably doesn't affect anyone using the decoder, but it is inconsistent with the documentation.
- The error positions returned in eras_pos include all erasures, but the corrections are only set in the correction buffer if there actually is a symbol error. So if there are erasures without symbol corruption, then the correction buffer will contain errors (unless initialized to zero before calling the decoder) or some values will be unset (if the correction buffer is uninitialized).
- When correcting data in-place the decoder does not correct errors in the parity. On the other hand, when returning the errors in correction buffers, errors in the parity are included.
The respective fixed are:
- The syndrome of a codeword is always zero, and the syndrome is linear, .i.e, S(x+e) = S(x) + S(e). So compute the syndrome for the error and check whether it equals the syndrome of the received word. If it does, then we have decoded to a valid codeword, otherwise we know that we have an uncorrectable error. Fortunately, some unrecoverable error conditions can be detected earlier in the decoding, which saves some processing power.
- Simply count and return the number of symbols actually corrected.
- Make sure to only return positions where symbols were corrected.
- Also fix errors in parity when correcting in-place. Another option would be to completely disregard errors in the parity, but then the interface makes it impossible to write tests that test for silent failures.
Other changes:
- Only fill the correction buffer and error position buffer if both of them are provided. Otherwise correct in place. Previously the error position buffer was always populated with the positions of the corrected errors, irrespective of whether a correction buffer was supplied or not. The rationale for this change is that there seems to be two use cases for the decoder; correct in-place or use the correction buffers. The caller does not need the positions of the corrected errors when in-place correction is used. If in-place correction is not used, then both the correction buffer and error position buffer need to be populated.
--- lib/reed_solomon/decode_rs.c | 88 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 68 insertions(+), 20 deletions(-)
lib/reed_solomon/decode_rs.c +68 -20
@@ -22,6 +22,7 @@uint16_t *index_of = rs->index_of;uint16_t u, q, tmp, num1, num2, den, discr_r, syn_error;int count =0 ; +int num_corrected;uint16_t msk = (uint16_t ) rs->nn;/*
@@ -184,6 +185,15 @@if (lambda[i] != nn) deg_lambda = i; } + +if (deg_lambda ==0 ) { +/* +* deg(lambda) is zero even though the syndrome is non-zero +* => uncorrectable error detected +*/ +return -EBADMSG ; + } +/* Find roots of error+erasure locator polynomial by Chien search */ memcpy (®[1 ], &lambda[1 ], nroots *sizeof (reg[0 ])); count =0 ;/* Number of roots of lambda(x) */
@@ -197,6 +207,12 @@ }if (q !=0 )continue ;/* Not a root */ + +if (k < pad) { +/* Impossible error location. Uncorrectable error. */ +return -EBADMSG ; + } +/* store root (index-form) and error location number */ root[count] = i; loc[count] = k;
@@ -231,7 +247,9 @@/* * Compute error values in poly-form. num1 = omega(inv(X(l))), num2 = * inv(X(l))**(fcr-1) and den = lambda_pr(inv(X(l))) all in poly-form +* Note: we reuse the buffer for b to store the correction pattern */ + num_corrected =0 ;for (j = count -1 ; j >=0 ; j--) { num1 =0 ;for (i = deg_omega; i >=0 ; i--) {
@@ -239,6 +257,13 @@ num1 ^= alpha_to[rs_modnn (rs, omega[i] + i * root[j])]; } + +if (num1 ==0 ) { +/* Nothing to correct at this position */ + b[j] =0 ; +continue ; + } + num2 = alpha_to[rs_modnn (rs, root[j] * (fcr -1 ) + nn)]; den =0 ;
@@ -250,29 +275,52 @@ i * root[j])]; } } -/* Apply error to data */ -if (num1 !=0 && loc[j] >= pad) { -uint16_t cor = alpha_to[rs_modnn (rs,index_of[num1] + - index_of[num2] + - nn - index_of[den])]; -/* Store the error correction pattern, if a -* correction buffer is available */ -if (corr) { - corr[j] = cor; - }else { -/* If a data buffer is given and the -* error is inside the message, -* correct it */ -if (data && (loc[j] < (nn - nroots))) - data[loc[j] - pad] ^= cor; - } + + b[j] = alpha_to[rs_modnn (rs, index_of[num1] + + index_of[num2] + + nn - index_of[den])]; + num_corrected++; + } + +/* +* We compute the syndrome of the 'error' and check that it matches +* the syndrome of the received word +*/ +for (i =0 ; i < nroots; i++) { + tmp =0 ; +for (j =0 ; j < count; j++) { +if (b[j] ==0 ) +continue ; + + k = (fcr + i) * prim * (nn-loc[j]-1 ); + tmp ^= alpha_to[rs_modnn (rs, index_of[b[j]] + k)]; } + +if (tmp != alpha_to[s[i]]) +return -EBADMSG ; } -if (eras_pos !=NULL ) { -for (i =0 ; i < count; i++) - eras_pos[i] = loc[i] - pad; +/* +* Store the error correction pattern, if a +* correction buffer is available +*/ +if (corr && eras_pos) { + j =0 ; +for (i =0 ; i < count; i++) { +if (b[i]) { + corr[j] = b[i]; + eras_pos[j++] = loc[i] - pad; + } + } + }else if (data && par) { +/* Apply error to data and parity */ +for (i =0 ; i < count; i++) { +if (loc[i] < (nn - nroots)) + data[loc[i] - pad] ^= b[i]; +else + par[loc[i] - pad - len] ^= b[i]; + } } -return count; +return num_corrected; }
Thomas Gleixner
AppliedFerdinand,
Quoted 2 lines · Ferdinand Blomqvist
This is the second revision of this series that fixes the bugs/flaws in the RS decoder. This addresses all of Thomas' comments/suggestions.
I've picked up the lot and it's en route to 5.3.
Thanks a lot for your excellent work and your patience!
tglx