Skip to contentSkip to conversation

Possible race condition when accessing SPI NOR Flash ?

DISCUSSION Valentin Longchamp 3 participants 5 messages May 1, 2014 – May 2, 2014

Subsystems (2)
Memory technology devices (MTD) SPI
lore
Share & tools
lore

Valentin Longchamp

author
#0 lore raw

Hello,

I am currently seeing a very strange behavior on one our PowerPC based systems (P2041 from Freescale). We have u-boot as a bootloader, currently using the 2013.10 release. The u-boot environment variables are stored in a SPI NOR Flash (from which the CPU also boots u-boot), in two dedicated partitions in a Spansion s25fl256s1 NOR Flash:

Quoted 17 lines
	spi@110000 {
			flash@0 {
				#address-cells = <1>;
				#size-cells = <1>;
				compatible = "spansion,s25fl256s1";
				reg = <0>;
				spi-max-frequency = <20000000>; /* input clock */
				partition@u-boot {
					label = "u-boot";
					reg = <0x00000000 0x00100000>;
					read-only;
				};
				partition@env {
					label = "env";
					reg = <0x00100000 0x00010000>;
				};
				partition@envred {
Quoted 9 lines
					label = "envred";
					reg = <0x00110000 0x00010000>;
				};
				partition@fman {
					label = "fman-ucode";
					reg = <0x00120000 0x00010000>;
				};
			};

I have never noticed a problem when accessing the environment variables from u-boot itself, nor the CPU not booting from the NOR Flash, so I assume the hardware is OK (and I am able to reproduce it on Freescale's dev board).

Now when fw_setenv ([1], [2]) updates the environment variables, it sometimes (approximately 1 out of 100 runs) triggers such kernel errors:

Quoted 19 lines
Unable to handle kernel paging request for data at address 0x54cc2178
Faulting instruction address: 0xc01187e8
Oops: Kernel access of bad area, sig: 11 [#2]
SMP NR_CPUS=8 Keymile kmp204x
Modules linked in:
CPU: 2 PID: 1628 Comm: loop_setenv Tainted: G    B D      3.10.36-7.0.9-00435-g228e7e3 #21
task: db720cb0 ti: db92e000 task.ti: db92e000
NIP: c01187e8 LR: c011866c CTR: 00000000
REGS: db92fd80 TRAP: 0300   Tainted: G    B D       (3.10.36-7.0.9-00435-g228e7e3)
MSR: 00021002 <CE,ME>  CR: 24002424  XER: 20000000
DEAR: 54cc2178, ESR: 00800000

GPR00: c011978c db92fe30 db720cb0 db8400c0 000000d0 00000000 db66f5bc 7f7f7f7f
GPR08: db0b4e90 db008a44 54cc2174 00000fec 24002422 1001c6ec 00000000 c0890000
GPR16: db008a54 db008a4c c0890000 00100100 00200200 00000001 db008a44 db008a40
GPR24: c0893444 00000000 000000d0 db92e000 db003dc0 db0b4e00 0000001b db840000 NIP [c01187e8] cache_alloc_refill+0x208/0x710
LR [c011866c] cache_alloc_refill+0x8c/0x710

Quoted 15 lines
Call Trace:
[db92fe30] [c000ede8] handle_page_fault+0xc/0x80 (unreliable)
[db92fe80] [c011978c] kmem_cache_alloc+0xec/0x180
[db92fea0] [c00580a4] prepare_creds+0x24/0x1c0
[db92feb0] [c0058964] prepare_exec_creds+0x14/0x50
[db92fec0] [c012635c] prepare_bprm_creds+0x2c/0x70
[db92fed0] [c01264bc] do_execve+0xbc/0x520
[db92ff20] [c0126b6c] SyS_execve+0x3c/0x70
[db92ff40] [c000e944] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xfed2d24
    LR = 0xfe6aef8
Instruction dump:
39090004 39290001 5508103a 7d1d4214 90680004 913d0000 813f0010 815c0018 
7f095040 41f8ff94 813f0004 815f0000 <912a0004> 91490000 927f0000 929f0004 
---[ end trace 85ac86b7cd18b604 ]---

OR:

Quoted 19 lines
Unable to handle kernel paging request for data at address 0x54cc2178
Faulting instruction address: 0xc0117c5c
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=8 Keymile kmp204x
Modules linked in:
CPU: 2 PID: 29 Comm: kworker/2:1 Not tainted 3.10.36-7.0.9-00435-g228e7e3 #21
Workqueue: events cache_reap
task: db2c12f0 ti: db2c6000 task.ti: db2c6000
NIP: c0117c5c LR: c011816c CTR: 00000000
REGS: db2c7d50 TRAP: 0300   Not tainted  (3.10.36-7.0.9-00435-g228e7e3)
MSR: 00021002 <CE,ME>  CR: 24002424  XER: 20000000
DEAR: 54cc2178, ESR: 00800000

GPR00: c011816c db2c7e00 db2c12f0 db003cc0 db800000 00000000 00000080 54cc2174
GPR08: 00005000 db008980 db800040 00021002 24000422 00000000 c0050300 db0d3de8
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c1167740
GPR24: 00000000 c0893260 00100100 00200200 00000002 db003cc0 db0afdd4 00000000 NIP [c0117c5c] free_block+0xbc/0x180
LR [c011816c] drain_array+0xfc/0x150

Quoted 11 lines
Call Trace:
[db2c7e30] [c011816c] drain_array+0xfc/0x150
[db2c7e60] [c01183cc] cache_reap+0xac/0x170
[db2c7e90] [c0046ce0] process_one_work+0x150/0x3d0
[db2c7ec0] [c00473ac] worker_thread+0x12c/0x390
[db2c7ef0] [c00503a8] kthread+0xa8/0xb0
[db2c7f40] [c000ea7c] ret_from_kernel_thread+0x5c/0x64
Instruction dump:
3d0a4000 5508c9f4 81390000 7cc9402e 7ce94214 70c58000 40c200c4 80870014 
813d0050 80e40000 81040004 7d29c02e <91070004> 90e80000 38c90004 93440000 
---[ end trace 721e12de4b707566 ]---

Notice that this does not necessarily happen in fw_setenv itself, but rather in the next "task" that tries to allocate/free some virtual memory.

I see the same behavior with both the 2013.10 and the 2014.04 releases of u-boot/fw_env. The kernel we are using is 3.10.36.

I suspect that the problem is related to SPI NOR/m25p80 driver: on the system we have a NAND Flash device with UBI volumes. If I create 2 ubi volumes on the NAND Flash and configure fw_setenv (/etc/fw_env.config) to use them instead of the the mtd devices targetting the s25fl256s1, I am not able to reproduce the problem, even over more than 10'000 runs.

I suspect that it is a race condition because it happens ~1 out of 100 times and if I disable SMP in the kernel I am never able to reproduce the problem.

Finally, after having analyzed the fw_env behavior, I have tried to write a very silly program that mimics the fw_env mtd accesses in my use case (attached with this email) and I am able to reproduce the problem with it.

One other possible culprit that I see is the fsl_espi.c driver for the underlying hardware connection from the CPU to the NOR Flash, but I wanted to ask here if someone had an idea about what's going wrong.

Valentin

[1] http://elinux.org/U-boot_environment_variables_in_linux
[2]
http://git.denx.de/?p=u-boot.git;a=tree;f=tools/env;h=99706ce668db90bd4a476a12e6c07fe2e4b85fbb;hb=master

#include <sys/types.h>
#include <sys/ioctl.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>

#include <mtd/mtd-user.h>

int mtd_open(const char *file, mode_t mode) {
	int fd = open(file, mode);
	if (fd == -1) {
		printf("Error opening file %s: %s\n", file, strerror(errno));
		return -1;
	}

	return fd;
}
int mtd_read(int fd, void *buf, size_t count) {
	int sz = read(fd, buf, count);
	if (sz == -1) {
		printf("Error reading fd %d: %s\n", fd, strerror(errno));
	}

	return sz;
}
int mtd_write(int fd, void *buf, size_t count) {
	int sz = write(fd, buf, count);
	if (sz == -1) {
		printf("Error reading fd %d: %s\n", fd, strerror(errno));
	}

	return sz;
}
int mtd_seek(int fd, off_t offset) {
	int ret = lseek(fd, offset, SEEK_SET);
	if (ret == -1) {
		printf("Error closing fd %d: %s\n", fd, strerror(errno));
	}

	return ret;
}
int mtd_erase(int fd, off_t offset) {
	int ret = ioctl(fd, MEMERASE, &offset);
	if (ret == -1) {
		printf("Error erasing fd %d: %s\n", fd, strerror(errno));
	}

	return ret;
}
int mtd_close(int fd) {
	int ret = close(fd);
	if (ret) {
		printf("Error closing fd %d: %s\n", fd, strerror(errno));
	}

	return ret;
}

int main(int argc, char *argv[])

{
	int	retValue;
	int	i;
	int	readSz1 = 0x4000;
	int	readSz2 = 0x10000;
	char	buffer1[readSz1];
	char	buffer2[readSz2];
	char *	file1 = "/dev/mtd6";
	char *	file2 = "/dev/mtd7";

	for (i=0; i < 1000; i++) {
		int ret, sz1, sz2, fd1, fd2;

		fd1 = mtd_open(file1, O_RDONLY);
		ret = mtd_seek(fd1, 0);
		sz1 = mtd_read(fd1, buffer1, readSz1);
		ret = mtd_close(fd1);
		fd2 = mtd_open(file2, O_RDONLY);
		ret = mtd_seek(fd2, 0);
		sz2 = mtd_read(fd2, buffer2, readSz1);
		ret = mtd_close(fd2);

		fd1 = mtd_open(file1, O_RDWR);
		fd2 = mtd_open(file2, O_RDWR);
		sz1 = mtd_read(fd2, buffer2, readSz2);
		ret = mtd_seek(fd2, 0);
		ret = mtd_erase(fd2, 0);
		sz1 = mtd_write(fd2, buffer2, readSz2);
		ret = mtd_seek(fd1, 4);
		sz2 = mtd_write(fd1, "\0", 1);
		ret = mtd_close(fd1);
		ret = mtd_close(fd2);

		if (!(i%50))
			printf("mtd, loop i=%d \n", i);
	}

	return 0;
}
#1 lore raw

Dear Valentin,

Quoted 3 lines · In message <[email protected]> you wrote:

Now when fw_setenv ([1], [2]) updates the environment variables, it sometimes (approximately 1 out of 100 runs) triggers such kernel errors:

...

Quoted 8 lines

I see the same behavior with both the 2013.10 and the 2014.04 releases of u-boot/fw_env. The kernel we are using is 3.10.36.

I suspect that the problem is related to SPI NOR/m25p80 driver: on the system we have a NAND Flash device with UBI volumes. If I create 2 ubi volumes on the NAND Flash and configure fw_setenv (/etc/fw_env.config) to use them instead of the the mtd devices targetting the s25fl256s1, I am not able to reproduce the problem, even over more than 10'000 runs.

Um... this is _Linux_ crashing - at this point the U-Boot driver has nothing to do with it.. If there is a problem, then with the Linux kernel / driver code. I guess you should rather post this on the MTD mailing ist.

Quoted 3 lines

One other possible culprit that I see is the fsl_espi.c driver for the underlying hardware connection from the CPU to the NOR Flash, but I wanted to ask here if someone had an idea about what's going wrong.

It's Linux code....

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Don't hit a man when he's down - kick him; it's easier.
author
#2 lore raw

Hello Wolfgang,

Quoted 16 lines · Wolfgang Denk

Dear Valentin,

Quoted 3 lines · In message <[email protected]> you wrote:

Now when fw_setenv ([1], [2]) updates the environment variables, it sometimes (approximately 1 out of 100 runs) triggers such kernel errors:

...

Quoted 8 lines

I see the same behavior with both the 2013.10 and the 2014.04 releases of u-boot/fw_env. The kernel we are using is 3.10.36.

I suspect that the problem is related to SPI NOR/m25p80 driver: on the system we have a NAND Flash device with UBI volumes. If I create 2 ubi volumes on the NAND Flash and configure fw_setenv (/etc/fw_env.config) to use them instead of the the mtd devices targetting the s25fl256s1, I am not able to reproduce the problem, even over more than 10'000 runs.

Quoted 5 lines

Um... this is _Linux_ crashing - at this point the U-Boot driver has nothing to do with it.. If there is a problem, then with the Linux kernel / driver code. I guess you should rather post this on the MTD mailing ist.

Yes, it was clear for me that this is Linux crashing, as I write "it [..] triggers [..] kernel errors." I have posted it to the linux-mtd ML, I have just added the u-boot ML as CC since fw_env is u-boot code and it triggers it, so I though it could be of interest.

Best regards,

Valentin

#3 lore raw
In reply to Valentin Longchamp

Hi.

I now fell a bit guilty. I had this same problem a while ago, and I haven't ever really pushed the fix. In fact, I ended up rewriting most of drivers/spi/spi-fsl-espi.c, and that's really the problem - I can only test about two boards from over 20, so I know that nobody will take my complete re-write seriously, and I sort of lost interest.

But, the bug you are hitting is actually much simpler to fix than that. If you look in drivers/spi/spi-fsl-espi.c, there is a function called fsl_espi_cmd_trans. There are two important things to note about this function; 1) It does a 64K kmalloc for every transfer (thanks). 2) It overwrites the buffer if the transfer size is large.

It packs the TX command into the start of the 64K block and sets up a TX pointer to the start of that block. It then points the RX pointer to be after the TX command. This is completely wrong. The RX pointer should also be set to point to the start of the block.

In the failing case, you are probably doing a 64K read from the Flash.
We will have a 5 or 6 byte command followed by 65531 bytes of TX data
(the value of the TX data does not matter). The driver will then TX the command followed by the data and simultaneously it will RX the "command" and the data that we want (the value of stuff RX'd when we are TX'ing the command is ignored / irrelevant). The point is that the last few bytes of data the we receive go after the end of the 64K block that we have allocated.

I found that I got similar kernel panics to you. Basically we are overwriting 5 random bytes of kernel memory at the start of a page.

        espi_trans->tx_buf = local_buf;
-       espi_trans->rx_buf = local_buf + espi_trans->n_tx;
+       espi_trans->rx_buf = local_buf;
        fsl_espi_do_trans(m, espi_trans);

So, something like this would be a fix. This is untested.

I hope this helps, and I'm sorry that I've not pushed this before. If people want I'll try to produce a proper fix, but I currently using an oldish kernel (3.2.10), and don't really want to try to build a newer one. I've only got limited freescale test hardware (P1010RDB & P2020RDB).

Regards,

Mark.

PS.
My re-write does nice things like use the half-duplex and simplex modes of the hardware (when suitable), and runs much much faster, and uses much less CPU, and has many fewer interrupts (and therefore works much better with the RT patches). But it probably breaks badly on any hardware that isn't like the P1010 or P2020.

Quoted 144 lines · Valentin Longchamp

Hello,

I am currently seeing a very strange behavior on one our PowerPC based systems (P2041 from Freescale). We have u-boot as a bootloader, currently using the 2013.10 release. The u-boot environment variables are stored in a SPI NOR Flash (from which the CPU also boots u-boot), in two dedicated partitions in a Spansion s25fl256s1 NOR Flash:

Quoted 17 lines
      spi at 110000 {
                      flash at 0 {
                              #address-cells = <1>;
                              #size-cells = <1>;
                              compatible = "spansion,s25fl256s1";
                              reg = <0>;
                              spi-max-frequency = <20000000>; /* input clock */
                              partition at u-boot {
                                      label = "u-boot";
                                      reg = <0x00000000 0x00100000>;
                                      read-only;
                              };
                              partition at env {
                                      label = "env";
                                      reg = <0x00100000 0x00010000>;
                              };
                              partition at envred {
Quoted 9 lines
                                      label = "envred";
                                      reg = <0x00110000 0x00010000>;
                              };
                              partition at fman {
                                      label = "fman-ucode";
                                      reg = <0x00120000 0x00010000>;
                              };
                      };

I have never noticed a problem when accessing the environment variables from u-boot itself, nor the CPU not booting from the NOR Flash, so I assume the hardware is OK (and I am able to reproduce it on Freescale's dev board).

Now when fw_setenv ([1], [2]) updates the environment variables, it sometimes (approximately 1 out of 100 runs) triggers such kernel errors:

Quoted 19 lines
Unable to handle kernel paging request for data at address 0x54cc2178
Faulting instruction address: 0xc01187e8
Oops: Kernel access of bad area, sig: 11 [#2]
SMP NR_CPUS=8 Keymile kmp204x
Modules linked in:
CPU: 2 PID: 1628 Comm: loop_setenv Tainted: G    B D      3.10.36-7.0.9-00435-g228e7e3 #21
task: db720cb0 ti: db92e000 task.ti: db92e000
NIP: c01187e8 LR: c011866c CTR: 00000000
REGS: db92fd80 TRAP: 0300   Tainted: G    B D       (3.10.36-7.0.9-00435-g228e7e3)
MSR: 00021002 <CE,ME>  CR: 24002424  XER: 20000000
DEAR: 54cc2178, ESR: 00800000

GPR00: c011978c db92fe30 db720cb0 db8400c0 000000d0 00000000 db66f5bc 7f7f7f7f
GPR08: db0b4e90 db008a44 54cc2174 00000fec 24002422 1001c6ec 00000000 c0890000
GPR16: db008a54 db008a4c c0890000 00100100 00200200 00000001 db008a44 db008a40
GPR24: c0893444 00000000 000000d0 db92e000 db003dc0 db0b4e00 0000001b db840000 NIP [c01187e8] cache_alloc_refill+0x208/0x710
LR [c011866c] cache_alloc_refill+0x8c/0x710

Quoted 15 lines
Call Trace:
[db92fe30] [c000ede8] handle_page_fault+0xc/0x80 (unreliable)
[db92fe80] [c011978c] kmem_cache_alloc+0xec/0x180
[db92fea0] [c00580a4] prepare_creds+0x24/0x1c0
[db92feb0] [c0058964] prepare_exec_creds+0x14/0x50
[db92fec0] [c012635c] prepare_bprm_creds+0x2c/0x70
[db92fed0] [c01264bc] do_execve+0xbc/0x520
[db92ff20] [c0126b6c] SyS_execve+0x3c/0x70
[db92ff40] [c000e944] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xfed2d24
    LR = 0xfe6aef8
Instruction dump:
39090004 39290001 5508103a 7d1d4214 90680004 913d0000 813f0010 815c0018
7f095040 41f8ff94 813f0004 815f0000 <912a0004> 91490000 927f0000 929f0004
---[ end trace 85ac86b7cd18b604 ]---

OR:

Quoted 19 lines
Unable to handle kernel paging request for data at address 0x54cc2178
Faulting instruction address: 0xc0117c5c
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=8 Keymile kmp204x
Modules linked in:
CPU: 2 PID: 29 Comm: kworker/2:1 Not tainted 3.10.36-7.0.9-00435-g228e7e3 #21
Workqueue: events cache_reap
task: db2c12f0 ti: db2c6000 task.ti: db2c6000
NIP: c0117c5c LR: c011816c CTR: 00000000
REGS: db2c7d50 TRAP: 0300   Not tainted  (3.10.36-7.0.9-00435-g228e7e3)
MSR: 00021002 <CE,ME>  CR: 24002424  XER: 20000000
DEAR: 54cc2178, ESR: 00800000

GPR00: c011816c db2c7e00 db2c12f0 db003cc0 db800000 00000000 00000080 54cc2174
GPR08: 00005000 db008980 db800040 00021002 24000422 00000000 c0050300 db0d3de8
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c1167740
GPR24: 00000000 c0893260 00100100 00200200 00000002 db003cc0 db0afdd4 00000000 NIP [c0117c5c] free_block+0xbc/0x180
LR [c011816c] drain_array+0xfc/0x150

Quoted 11 lines
Call Trace:
[db2c7e30] [c011816c] drain_array+0xfc/0x150
[db2c7e60] [c01183cc] cache_reap+0xac/0x170
[db2c7e90] [c0046ce0] process_one_work+0x150/0x3d0
[db2c7ec0] [c00473ac] worker_thread+0x12c/0x390
[db2c7ef0] [c00503a8] kthread+0xa8/0xb0
[db2c7f40] [c000ea7c] ret_from_kernel_thread+0x5c/0x64
Instruction dump:
3d0a4000 5508c9f4 81390000 7cc9402e 7ce94214 70c58000 40c200c4 80870014
813d0050 80e40000 81040004 7d29c02e <91070004> 90e80000 38c90004 93440000
---[ end trace 721e12de4b707566 ]---

Notice that this does not necessarily happen in fw_setenv itself, but rather in the next "task" that tries to allocate/free some virtual memory.

I see the same behavior with both the 2013.10 and the 2014.04 releases of u-boot/fw_env. The kernel we are using is 3.10.36.

I suspect that the problem is related to SPI NOR/m25p80 driver: on the system we have a NAND Flash device with UBI volumes. If I create 2 ubi volumes on the NAND Flash and configure fw_setenv (/etc/fw_env.config) to use them instead of the the mtd devices targetting the s25fl256s1, I am not able to reproduce the problem, even over more than 10'000 runs.

I suspect that it is a race condition because it happens ~1 out of 100 times and if I disable SMP in the kernel I am never able to reproduce the problem.

Finally, after having analyzed the fw_env behavior, I have tried to write a very silly program that mimics the fw_env mtd accesses in my use case (attached with this email) and I am able to reproduce the problem with it.

One other possible culprit that I see is the fsl_espi.c driver for the underlying hardware connection from the CPU to the NOR Flash, but I wanted to ask here if someone had an idea about what's going wrong.

Valentin

[1] http://elinux.org/U-boot_environment_variables_in_linux
[2]
http://git.denx.de/?p=u-boot.git;a=tree;f=tools/env;h=99706ce668db90bd4a476a12e6c07fe2e4b85fbb;hb=master

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

author
#4 lore raw

Hi Mark,

Quoted 7 lines · Mark Marshall

Hi.

I now fell a bit guilty. I had this same problem a while ago, and I haven't ever really pushed the fix. In fact, I ended up rewriting most of drivers/spi/spi-fsl-espi.c, and that's really the problem - I can only test about two boards from over 20, so I know that nobody will take my complete re-write seriously, and I sort of lost interest.

Don't feel guilty at all. Even though I have suffered a bit to isolate the problem, your answer has already helped a lot: with your proposed fix, I have not been able to reproduce the error anymore.

Quoted 29 lines

But, the bug you are hitting is actually much simpler to fix than that. If you look in drivers/spi/spi-fsl-espi.c, there is a function called fsl_espi_cmd_trans. There are two important things to note about this function; 1) It does a 64K kmalloc for every transfer (thanks). 2) It overwrites the buffer if the transfer size is large.

It packs the TX command into the start of the 64K block and sets up a TX pointer to the start of that block. It then points the RX pointer to be after the TX command. This is completely wrong. The RX pointer should also be set to point to the start of the block.

In the failing case, you are probably doing a 64K read from the Flash.
We will have a 5 or 6 byte command followed by 65531 bytes of TX data
(the value of the TX data does not matter). The driver will then TX the command followed by the data and simultaneously it will RX the "command" and the data that we want (the value of stuff RX'd when we are TX'ing the command is ignored / irrelevant). The point is that the last few bytes of data the we receive go after the end of the 64K block that we have allocated.

I found that I got similar kernel panics to you. Basically we are overwriting 5 random bytes of kernel memory at the start of a page.

        espi_trans->tx_buf = local_buf;
(quoted diff) +1 -1
-       espi_trans->rx_buf = local_buf + espi_trans->n_tx;
+       espi_trans->rx_buf = local_buf;
        fsl_espi_do_trans(m, espi_trans);
Quoted 1 line

So, something like this would be a fix. This is untested.

As I have said above, this fixes the problem for me (as far as I have had time to test it until now, I will go on in the next days).

Quoted 6 lines

I hope this helps, and I'm sorry that I've not pushed this before. If people want I'll try to produce a proper fix, but I currently using an oldish kernel (3.2.10), and don't really want to try to build a newer one. I've only got limited freescale test hardware (P1010RDB & P2020RDB).

If you don't feel like sending a patch because of the above reasons, I will happily submit one (just to avoid a third person to be struck by it ;-) ). If you want to submit one, just put me on CC and I will happily test it on P2041RDB and and our P2041 based system on a newer kernel.

Thank you very much & best Regards

Valentin

Quoted 5 lines

Regards,

Mark.

[snip]

Quoted 16 lines
Quoted 16 lines

Notice that this does not necessarily happen in fw_setenv itself, but rather in the next "task" that tries to allocate/free some virtual memory.

I see the same behavior with both the 2013.10 and the 2014.04 releases of u-boot/fw_env. The kernel we are using is 3.10.36.

I suspect that the problem is related to SPI NOR/m25p80 driver: on the system we have a NAND Flash device with UBI volumes. If I create 2 ubi volumes on the NAND Flash and configure fw_setenv (/etc/fw_env.config) to use them instead of the the mtd devices targetting the s25fl256s1, I am not able to reproduce the problem, even over more than 10'000 runs.

I suspect that it is a race condition because it happens ~1 out of 100 times and if I disable SMP in the kernel I am never able to reproduce the problem.

Quoted 8 lines
Quoted 8 lines

Finally, after having analyzed the fw_env behavior, I have tried to write a very silly program that mimics the fw_env mtd accesses in my use case (attached with this email) and I am able to reproduce the problem with it.

One other possible culprit that I see is the fsl_espi.c driver for the underlying hardware connection from the CPU to the NOR Flash, but I wanted to ask here if someone had an idea about what's going wrong.

--
To unsubscribe from this list: send the line "unsubscribe linux-spi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html