UBIFS CRC error, calculated from ubifs_check_node

DISCUSSION Tanya Brokhman 3 participants 5 messages Apr 20, 2014 – Apr 27, 2014 Memory technology devices (MTD)

Share & tools

Tanya Brokhman

author

2014-04-20 18:39 UTC #0 lore raw

Hi All

I'm hitting "UBIFS error (pid 1026): ubifs_check_node: bad CRC: calculated 0x79dbca01, read 0x31afb5d7" and as a result the partition is remounted as RO.

I'm using 3.10 based kernel.
The test scenario to reproduce this is:
1. fill the partition tested using dd
2. try to write some more data

There are no errors in the log from the ubi layer or from the underlying nand driver, both also testing crc.

I've tried disabling compression, mounting the partition as sync and issuing the write command with sync param. None of the former has any effect on this crash.
I tried disabling crc on data nodes. This lowered the possibility of hitting the above error but eventually, with stress test of concurrent read and write we hit the same error with a different node type.

Has any one encountered similar behavior? I'm not that much familiar with ubifs so any help will be highly apritiated!

Bellow is the snippet of the call stack in question and the command I used to hit it:
dd if=/dev/zero of=/firmware/file2 bs=1M count=5000

[ 187.472289] UBIFS error (pid 1026): ubifs_check_node: bad CRC: calculated 0x79dbca01, read 0x31afb5d7
[ 187.480925] UBIFS error (pid 1026): ubifs_check_node: bad node at LEB 263:94456
[ 187.488385] magic 0x6101831
[ 187.491483] crc 0x31afb5d7
[ 187.495198] node_type 1 (data node)
[ 187.499147] group_type 0 (no node group)
[ 187.503522] sqnum 10721
[ 187.506791] len 4013
[ 187.510002] key (102, data, 2102)
[ 187.514369] size 4096
[ 187.517552] compr_typ 1
[ 187.520508] data size 3965
[ 187.523740] data:
...
188.849392] CPU: 0 PID: 1026 Comm: dd Tainted: G W 3.10.28+ #1
[ 188.855598] [sched_delayed] sched: RT throttling activated
[ 188.861091] [<c0014674>] (unwind_backtrace+0x0/0x11c) from [<c0011a34>] (show_stack+0x10/0x14)
[ 188.869687] [<c0011a34>] (show_stack+0x10/0x14) from [<c0163e40>] (ubifs_check_node+0x280/0x2b8)
[ 188.878447] [<c0163e40>] (ubifs_check_node+0x280/0x2b8) from [<c016ba6c>] (ubifs_scan_a_node+0x190/0x304)
[ 188.888002] [<c016ba6c>] (ubifs_scan_a_node+0x190/0x304) from [<c016c038>] (ubifs_scan+0xa0/0x348)
[ 188.896939] [<c016c038>] (ubifs_scan+0xa0/0x348) from [<c0170bb0>] (ubifs_garbage_collect_leb+0x1e0/0x76c)
[ 188.906569] [<c0170bb0>] (ubifs_garbage_collect_leb+0x1e0/0x76c) from [<c01713c8>] (ubifs_garbage_collect+0x28c/0x5fc)
[ 188.917248] [<c01713c8>] (ubifs_garbage_collect+0x28c/0x5fc) from [<c0173b0c>] (ubifs_budget_space+0x64c/0x898)
[ 188.927320] [<c0173b0c>] (ubifs_budget_space+0x64c/0x898) from [<c015a320>] (ubifs_write_begin+0x3b0/0x524)
[ 188.937047] [<c015a320>] (ubifs_write_begin+0x3b0/0x524) from [<c009a2d8>] (generic_file_buffered_write+0xd8/0x250)
[ 188.947461] [<c009a2d8>] (generic_file_buffered_write+0xd8/0x250) from [<c009b614>] (__generic_file_aio_write+0x364/0x3b0)
[ 188.958486] [<c009b614>] (__generic_file_aio_write+0x364/0x3b0) from [<c009b6b4>] (generic_file_aio_write+0x54/0xb0)
[ 188.968982] [<c009b6b4>] (generic_file_aio_write+0x54/0xb0) from [<c01597f8>] (ubifs_aio_write+0x16c/0x180)
[ 188.978708] [<c01597f8>] (ubifs_aio_write+0x16c/0x180) from [<c00d47d4>] (do_sync_write+0x74/0x98)
[ 188.987648] [<c00d47d4>] (do_sync_write+0x74/0x98) from [<c00d5138>] (vfs_write+0xd4/0x16c)
[ 188.995979] [<c00d5138>] (vfs_write+0xd4/0x16c) from [<c00d5478>] (SyS_write+0x3c/0x60)
[ 189.005059] [<c00d5478>] (SyS_write+0x3c/0x60) from [<c000e440>] (ret_fast_syscall+0x0/0x30)
[ 189.012651] UBIFS error (pid 1026): ubifs_scan: bad node
[ 189.018546] UBIFS error (pid 1026): ubifs_scanned_corruption: corruption at LEB 263:94456
[ 189.026124] UBIFS error (pid 1026): ubifs_scanned_corruption: first 8192 bytes from LEB 263:94456
[ 189.039633] UBIFS error (pid 1026): ubifs_scan: LEB 263 scanning failed
[ 189.045313] UBIFS warning (pid 1026): ubifs_ro_mode: switched to read-only mode, error -117
[ 189.053634] CPU: 0 PID: 1026 Comm: dd Tainted: G W 3.10.28+ #1

Thanks!
Tanya Brokhman

Signature · 2 lines

QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Richard Weinberger

2014-04-20 20:55 UTC #1 lore raw

Quoted 21 lines · Tanya Brokhman

Hi All
I'm hitting "UBIFS error (pid 1026): ubifs_check_node: bad CRC: calculated 0x79dbca01, read 0x31afb5d7" and as a result the partition is remounted as RO.
I'm using 3.10 based kernel.
The test scenario to reproduce this is:
1. fill the partition tested using dd
2. try to write some more data
There are no errors in the log from the ubi layer or from the underlying nand driver, both also testing crc.
I've tried disabling compression, mounting the partition as sync and issuing the write command with sync param. None of the former has any effect on this crash.
I tried disabling crc on data nodes. This lowered the possibility of hitting the above error but eventually, with stress test of concurrent read and write we hit the same error with a different node type.

Quoted 2 lines

Has any one encountered similar behavior? I'm not that much familiar with ubifs so any help will be highly apritiated!

Before we start digging into this, do mtd-tests and ubi-tests pass?

Signature · 2 lines

Thanks,
//richard

Tanya Brokhman

author

2014-04-20 21:24 UTC #2 lore raw

Quoted 22 lines · Richard Weinberger

Quoted 21 lines · Tanya Brokhman
Hi All
I'm hitting "UBIFS error (pid 1026): ubifs_check_node: bad CRC: calculated 0x79dbca01, read 0x31afb5d7" and as a result the partition is remounted as RO.
I'm using 3.10 based kernel.
The test scenario to reproduce this is:
1. fill the partition tested using dd
2. try to write some more data
There are no errors in the log from the ubi layer or from the underlying nand driver, both also testing crc.
I've tried disabling compression, mounting the partition as sync and issuing the write command with sync param. None of the former has any effect on this crash.
I tried disabling crc on data nodes. This lowered the possibility of hitting the above error but eventually, with stress test of concurrent read and write we hit the same error with a different node type.

Quoted 5 lines

Quoted 2 lines
Has any one encountered similar behavior? I'm not that much familiar with ubifs so any help will be highly apritiated!
Before we start digging into this, do mtd-tests and ubi-tests pass?

mtd-tests: yes.
ubi-tests: never run those. Where can I find them? I see only debug.c at mtd/ubi with helping functions but no tests.

We're just switching to ubifs from yaffs2. I looked at the diff between linux-next and our branch (3.10 based) and tried to bring in all of the fixes in mtd and ubi layers. I'll try running ubi-tests and update on the result.

Do you think we should run ubifs tests as well? Are there such tests? Thank you for the prompt reply!

Signature · 2 lines

QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Richard Weinberger

maintainer

2014-04-20 21:31 UTC #3 lore raw

Am 20.04.2014 23:24, schrieb Tanya Brokhman:

Quoted 23 lines · Richard Weinberger

Quoted 22 lines · Tanya Brokhman
Quoted 21 lines
Hi All
I'm hitting "UBIFS error (pid 1026): ubifs_check_node: bad CRC: calculated 0x79dbca01, read 0x31afb5d7" and as a result the partition is remounted as RO.
I'm using 3.10 based kernel.
The test scenario to reproduce this is:
1. fill the partition tested using dd
2. try to write some more data
There are no errors in the log from the ubi layer or from the underlying nand driver, both also testing crc.
I've tried disabling compression, mounting the partition as sync and issuing the write command with sync param. None of the former has any effect on this crash.
I tried disabling crc on data nodes. This lowered the possibility of hitting the above error but eventually, with stress test of concurrent read and write we hit the same error with a different node type.

Quoted 8 lines

Quoted 5 lines
Quoted 2 lines
Has any one encountered similar behavior? I'm not that much familiar with ubifs so any help will be highly apritiated!
Before we start digging into this, do mtd-tests and ubi-tests pass?
mtd-tests: yes.

Good. :)

Quoted 6 lines

ubi-tests: never run those. Where can I find them? I see only debug.c at mtd/ubi with helping functions but no tests.
We're just switching to ubifs from yaffs2. I looked at the diff between linux-next and our branch (3.10 based) and tried to bring in all of the fixes in mtd and ubi layers. I'll try running ubi-tests and update on the result.
Do you think we should run ubifs tests as well? Are there such tests?

Yes. Fetch mtd-utils source.
See tests/ubi-tests directory.
It contains UBI and UBIFS tests.

Please run first ubi tests (runtests.sh).

Thanks,
//richard

Tanya Brokhman

author

2014-04-27 11:20 UTC #4 lore raw

Quoted 42 lines · Richard Weinberger

Am 20.04.2014 23:24, schrieb Tanya Brokhman:
Quoted 23 lines · Richard Weinberger
Quoted 22 lines · Tanya Brokhman
Quoted 21 lines
Hi All
I'm hitting "UBIFS error (pid 1026): ubifs_check_node: bad CRC: calculated 0x79dbca01, read 0x31afb5d7" and as a result the partition is remounted as RO.
I'm using 3.10 based kernel.
The test scenario to reproduce this is:
1. fill the partition tested using dd
2. try to write some more data
There are no errors in the log from the ubi layer or from the underlying nand driver, both also testing crc.
I've tried disabling compression, mounting the partition as sync and issuing the write command with sync param. None of the former has any effect on this crash.
I tried disabling crc on data nodes. This lowered the possibility of hitting the above error but eventually, with stress test of concurrent read and write we hit the same error with a different node type.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Has any one encountered similar behavior? I'm not that much familiar with ubifs so any help will be highly apritiated!
Before we start digging into this, do mtd-tests and ubi-tests pass?
mtd-tests: yes.
Good. :)
Quoted 6 lines
ubi-tests: never run those. Where can I find them? I see only debug.c at mtd/ubi with helping functions but no tests.
We're just switching to ubifs from yaffs2. I looked at the diff between linux-next and our branch (3.10 based) and tried to bring in all of the fixes in mtd and ubi layers. I'll try running ubi-tests and update on the result.
Do you think we should run ubifs tests as well? Are there such tests?

Quoted 9 lines

Yes. Fetch mtd-utils source.
See tests/ubi-tests directory.
It contains UBI and UBIFS tests.
Please run first ubi tests (runtests.sh).
Thanks,
//richard

Hi Richard

Eventually it turned out to be driver issue. Thank you for your help!

- Tanya

Signature · 2 lines

QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation