Skip to contentSkip to conversation

mtd: rawnand: marvell: check for RDY bits after enabling the IRQ

PATCH Applied Daniel Mack 4 participants 28 messages Sep 27, 2018 – Oct 3, 2018

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mtd: rawnand: marvell: check for RDY bits after enabling the IRQ

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---
v1 → v2:

* Use complete(&nfc->complete) when the condition is met, and do wait_for_completion_timeout() in all cases. Suggested by Boris Brezillon.

 drivers/mtd/nand/raw/marvell_nand.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/mtd/nand/raw/marvell_nand.c +10 -0
index 666f34b58dec..4870b5bae296 100644
@@ -614,6 +614,7 @@ static int marvell_nfc_wait_op(struct nand_chip *chip, unsigned int timeout_ms)
 {
 	struct marvell_nfc *nfc = to_marvell_nfc(chip->controller);
 	int ret;
+	u32 st;
 
 	/* Timeout is expressed in ms */
 	if (!timeout_ms)
@@ -622,6 +623,15 @@ static int marvell_nfc_wait_op(struct nand_chip *chip, unsigned int timeout_ms)
 	init_completion(&nfc->complete);
 
 	marvell_nfc_enable_int(nfc, NDCR_RDYM);
+
+	/*
+	 * Check if the NDSR_RDY bits have already been set before the
+	 * interrupt was enabled.
+	 */
+	st = readl_relaxed(nfc->regs + NDSR);
+	if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+		complete(&nfc->complete);
+
 	ret = wait_for_completion_timeout(&nfc->complete,
 					  msecs_to_jiffies(timeout_ms));
 	marvell_nfc_disable_int(nfc, NDCR_RDYM);
Signature · 1 line
2.17.1
maintainer Acked-by on v2 1/1
#1 lore raw

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Thanks,
Miquèl

#2 lore raw
Quoted 26 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Quoted 7 lines

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Queued to mtd/master.

Thanks,

Boris

#3 lore raw

Hi All,

Quoted 32 lines · Boris Brezillon
Quoted 29 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Quoted 6 lines
Quoted 4 lines

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Queued to mtd/master.

After fixing my R/B configuration I get a new error with this patch when running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

My board is a custom design using an Armada-385 SoC with Macronix NAND.

# stress_1
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ/WRDREQ while 
draining raw data (NDSR: 0x00000000)
ubi0 error: ubi_io_write: error -5 while writing 4096 bytes to PEB 
1858:110592, written 0 bytes
CPU: 1 PID: 1170 Comm: stress_1 Not tainted 4.19.0-rc5-at1+ #8
Hardware name: Marvell Armada 380/385 (Device Tree)
[<8011143c>] (unwind_backtrace) from [<8010c17c>] (show_stack+0x10/0x14)
[<8010c17c>] (show_stack) from [<805ec28c>] (dump_stack+0x88/0x9c)
[<805ec28c>] (dump_stack) from [<80418a28>] (ubi_io_write+0x55c/0x6c0)
[<80418a28>] (ubi_io_write) from [<80415b4c>] (ubi_eba_write_leb+0x80/0x780)
[<80415b4c>] (ubi_eba_write_leb) from [<80414580>] (ubi_leb_write+0xbc/0xe0)
[<80414580>] (ubi_leb_write) from [<802d46b4>] (ubifs_leb_write+0xa0/0x118)
[<802d46b4>] (ubifs_leb_write) from [<802d5620>] 
(ubifs_wbuf_write_nolock+0x184/0x6ac)
[<802d5620>] (ubifs_wbuf_write_nolock) from [<802c8a18>] 
(ubifs_jnl_write_data+0x1c0/0x2bc)
[<802c8a18>] (ubifs_jnl_write_data) from [<802caed8>] 
(do_writepage+0xa4/0x1b0)
[<802caed8>] (do_writepage) from [<801aa160>] (__writepage+0x14/0x48)
[<801aa160>] (__writepage) from [<801aa900>] (write_cache_pages+0x1d0/0x3e4)
[<801aa900>] (write_cache_pages) from [<801aab68>] 
(generic_writepages+0x54/0x80)
[<801aab68>] (generic_writepages) from [<801ac9a0>] 
(do_writepages+0x68/0x8c)
[<801ac9a0>] (do_writepages) from [<801a0ac8>] 
(__filemap_fdatawrite_range+0x88/0xc0)
[<801a0ac8>] (__filemap_fdatawrite_range) from [<801a0cc4>] 
(file_write_and_wait_range+0x3c/0x98)
[<801a0cc4>] (file_write_and_wait_range) from [<802cb600>] 
(ubifs_fsync+0x3c/0xb0)
[<802cb600>] (ubifs_fsync) from [<801a2828>] 
(generic_file_write_iter+0x198/0x24c)
[<801a2828>] (generic_file_write_iter) from [<802ccb84>] 
(ubifs_write_iter+0xf0/0x158)
[<802ccb84>] (ubifs_write_iter) from [<801ef854>] (__vfs_write+0xfc/0x160)
[<801ef854>] (__vfs_write) from [<801efa60>] (vfs_write+0xa4/0x1ac)
[<801efa60>] (vfs_write) from [<801efcac>] (ksys_write+0x54/0xb8)
[<801efcac>] (ksys_write) from [<80101000>] (ret_fast_syscall+0x0/0x54)
Exception stack(0xbd789fa8 to 0xbd789ff0)
9fa0:                   0ca5d000 00000000 00000003 7e9f2900 00008000 
ffffffff
9fc0: 0ca5d000 00000000 00008000 00000004 00000003 00000000 76f24fb4 
00000000
9fe0: 00000000 7e9f27fc 00010fd8 76e775ec
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
ttyS ttyS1: tty_port_close_start: tty->count = 1 port count = 2
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)

... (RDDREQ messages repeat).

#4 lore raw
Quoted 108 lines · On Thu, 27 Sep 2018 21:55:57 +0000 Chris Packham…

Hi All,

Quoted 32 lines · Boris Brezillon
Quoted 29 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Quoted 6 lines
Quoted 4 lines

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Queued to mtd/master.

After fixing my R/B configuration I get a new error with this patch when running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

My board is a custom design using an Armada-385 SoC with Macronix NAND.

# stress_1
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ/WRDREQ while 
draining raw data (NDSR: 0x00000000)
ubi0 error: ubi_io_write: error -5 while writing 4096 bytes to PEB 
1858:110592, written 0 bytes
CPU: 1 PID: 1170 Comm: stress_1 Not tainted 4.19.0-rc5-at1+ #8
Hardware name: Marvell Armada 380/385 (Device Tree)
[<8011143c>] (unwind_backtrace) from [<8010c17c>] (show_stack+0x10/0x14)
[<8010c17c>] (show_stack) from [<805ec28c>] (dump_stack+0x88/0x9c)
[<805ec28c>] (dump_stack) from [<80418a28>] (ubi_io_write+0x55c/0x6c0)
[<80418a28>] (ubi_io_write) from [<80415b4c>] (ubi_eba_write_leb+0x80/0x780)
[<80415b4c>] (ubi_eba_write_leb) from [<80414580>] (ubi_leb_write+0xbc/0xe0)
[<80414580>] (ubi_leb_write) from [<802d46b4>] (ubifs_leb_write+0xa0/0x118)
[<802d46b4>] (ubifs_leb_write) from [<802d5620>] 
(ubifs_wbuf_write_nolock+0x184/0x6ac)
[<802d5620>] (ubifs_wbuf_write_nolock) from [<802c8a18>] 
(ubifs_jnl_write_data+0x1c0/0x2bc)
[<802c8a18>] (ubifs_jnl_write_data) from [<802caed8>] 
(do_writepage+0xa4/0x1b0)
[<802caed8>] (do_writepage) from [<801aa160>] (__writepage+0x14/0x48)
[<801aa160>] (__writepage) from [<801aa900>] (write_cache_pages+0x1d0/0x3e4)
[<801aa900>] (write_cache_pages) from [<801aab68>] 
(generic_writepages+0x54/0x80)
[<801aab68>] (generic_writepages) from [<801ac9a0>] 
(do_writepages+0x68/0x8c)
[<801ac9a0>] (do_writepages) from [<801a0ac8>] 
(__filemap_fdatawrite_range+0x88/0xc0)
[<801a0ac8>] (__filemap_fdatawrite_range) from [<801a0cc4>] 
(file_write_and_wait_range+0x3c/0x98)
[<801a0cc4>] (file_write_and_wait_range) from [<802cb600>] 
(ubifs_fsync+0x3c/0xb0)
[<802cb600>] (ubifs_fsync) from [<801a2828>] 
(generic_file_write_iter+0x198/0x24c)
[<801a2828>] (generic_file_write_iter) from [<802ccb84>] 
(ubifs_write_iter+0xf0/0x158)
[<802ccb84>] (ubifs_write_iter) from [<801ef854>] (__vfs_write+0xfc/0x160)
[<801ef854>] (__vfs_write) from [<801efa60>] (vfs_write+0xa4/0x1ac)
[<801efa60>] (vfs_write) from [<801efcac>] (ksys_write+0x54/0xb8)
[<801efcac>] (ksys_write) from [<80101000>] (ret_fast_syscall+0x0/0x54)
Exception stack(0xbd789fa8 to 0xbd789ff0)
9fa0:                   0ca5d000 00000000 00000003 7e9f2900 00008000 
ffffffff
9fc0: 0ca5d000 00000000 00008000 00000004 00000003 00000000 76f24fb4 
00000000
9fe0: 00000000 7e9f27fc 00010fd8 76e775ec
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
ttyS ttyS1: tty_port_close_start: tty->count = 1 port count = 2
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
Quoted 1 line

... (RDDREQ messages repeat).

Hm, that's weird, unless RDDREQ is a 'clear-on-read' bit, that shouldn't happen.

#5 lore raw
Quoted 16 lines · On Fri, 28 Sep 2018 08:40:46 +0200 Boris Brezillon…
Quoted 16 lines

marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)
ttyS ttyS1: tty_port_close_start: tty->count = 1 port count = 2 marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining FIFO (data) (NDSR: 0x00000810)

Quoted 4 lines
Quoted 1 line

... (RDDREQ messages repeat).

Hm, that's weird, unless RDDREQ is a 'clear-on-read' bit, that shouldn't happen.

BTW, I dropped the patch.

maintainer
#6 lore raw
In reply to Boris Brezillon

Hi Boris,

Boris Brezillon <[email protected]> wrote on Fri, 28 Sep 2018 08:40:46 +0200:

Quoted 111 lines
Quoted 108 lines · On Thu, 27 Sep 2018 21:55:57 +0000 Chris Packham…

Hi All,

Quoted 32 lines · Boris Brezillon
Quoted 29 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Quoted 6 lines
Quoted 4 lines

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Queued to mtd/master.

After fixing my R/B configuration I get a new error with this patch when running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

My board is a custom design using an Armada-385 SoC with Macronix NAND.

# stress_1
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ/WRDREQ while 
draining raw data (NDSR: 0x00000000)
ubi0 error: ubi_io_write: error -5 while writing 4096 bytes to PEB 
1858:110592, written 0 bytes
CPU: 1 PID: 1170 Comm: stress_1 Not tainted 4.19.0-rc5-at1+ #8
Hardware name: Marvell Armada 380/385 (Device Tree)
[<8011143c>] (unwind_backtrace) from [<8010c17c>] (show_stack+0x10/0x14)
[<8010c17c>] (show_stack) from [<805ec28c>] (dump_stack+0x88/0x9c)
[<805ec28c>] (dump_stack) from [<80418a28>] (ubi_io_write+0x55c/0x6c0)
[<80418a28>] (ubi_io_write) from [<80415b4c>] (ubi_eba_write_leb+0x80/0x780)
[<80415b4c>] (ubi_eba_write_leb) from [<80414580>] (ubi_leb_write+0xbc/0xe0)
[<80414580>] (ubi_leb_write) from [<802d46b4>] (ubifs_leb_write+0xa0/0x118)
[<802d46b4>] (ubifs_leb_write) from [<802d5620>] 
(ubifs_wbuf_write_nolock+0x184/0x6ac)
[<802d5620>] (ubifs_wbuf_write_nolock) from [<802c8a18>] 
(ubifs_jnl_write_data+0x1c0/0x2bc)
[<802c8a18>] (ubifs_jnl_write_data) from [<802caed8>] 
(do_writepage+0xa4/0x1b0)
[<802caed8>] (do_writepage) from [<801aa160>] (__writepage+0x14/0x48)
[<801aa160>] (__writepage) from [<801aa900>] (write_cache_pages+0x1d0/0x3e4)
[<801aa900>] (write_cache_pages) from [<801aab68>] 
(generic_writepages+0x54/0x80)
[<801aab68>] (generic_writepages) from [<801ac9a0>] 
(do_writepages+0x68/0x8c)
[<801ac9a0>] (do_writepages) from [<801a0ac8>] 
(__filemap_fdatawrite_range+0x88/0xc0)
[<801a0ac8>] (__filemap_fdatawrite_range) from [<801a0cc4>] 
(file_write_and_wait_range+0x3c/0x98)
[<801a0cc4>] (file_write_and_wait_range) from [<802cb600>] 
(ubifs_fsync+0x3c/0xb0)
[<802cb600>] (ubifs_fsync) from [<801a2828>] 
(generic_file_write_iter+0x198/0x24c)
[<801a2828>] (generic_file_write_iter) from [<802ccb84>] 
(ubifs_write_iter+0xf0/0x158)
[<802ccb84>] (ubifs_write_iter) from [<801ef854>] (__vfs_write+0xfc/0x160)
[<801ef854>] (__vfs_write) from [<801efa60>] (vfs_write+0xa4/0x1ac)
[<801efa60>] (vfs_write) from [<801efcac>] (ksys_write+0x54/0xb8)
[<801efcac>] (ksys_write) from [<80101000>] (ret_fast_syscall+0x0/0x54)
Exception stack(0xbd789fa8 to 0xbd789ff0)
9fa0:                   0ca5d000 00000000 00000003 7e9f2900 00008000 
ffffffff
9fc0: 0ca5d000 00000000 00008000 00000004 00000003 00000000 76f24fb4 
00000000
9fe0: 00000000 7e9f27fc 00010fd8 76e775ec
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
ttyS ttyS1: tty_port_close_start: tty->count = 1 port count = 2
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
marvell-nfc f10d0000.nand-controller: Timeout on RDDREQ while draining 
FIFO (data) (NDSR: 0x00000810)
Quoted 5 lines
Quoted 1 line

... (RDDREQ messages repeat).

Hm, that's weird, unless RDDREQ is a 'clear-on-read' bit, that shouldn't happen.

The spec clearly states it's not. In the status register (NDSR), you must write a bit to clear it.

The RDDREQ and RDY0/1 bit checks do not share the same path. Would it be possible that in some situations all the bits are not actually cleared, and so reading the NDSR register tells us the event happened while it has not in reality?

IIRC this patch fixes pxa implementation (NFCv1) while the errors happen on an Armada 385 (NFCv2). It would be great to understand (and document) this behavior, otherwise it is still possible to do the manual check only for NFCv1 if this is the only IP suffering about the race.

Thanks,
Miquèl

#7 lore raw
In reply to Chris Packham

Hi Chris,

Quoted 35 lines · Chris Packham
Quoted 34 lines · Boris Brezillon
Quoted 31 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Queued to mtd/master.

After fixing my R/B configuration I get a new error with this patch when running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

That's strange. So your controller sets the RDY bits before it is ready? Could you check whether only checking for NDSR_RDY(0) changes anything? Not sure about the handling of NDSR_RDY(1) in this driver anyway ...

Also, does my .EALREADY approach (v1) make any difference?

Thanks,
Daniel

maintainer
#8 lore raw

Hi Daniel,

Daniel Mack <[email protected]> wrote on Fri, 28 Sep 2018 09:43:18 +0200:

Quoted 41 lines

Hi Chris,

Quoted 38 lines · Chris Packham
Quoted 37 lines · Boris Brezillon
Quoted 33 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Queued to mtd/master.
After fixing my R/B configuration I get a new error with this patch when

running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

That's strange. So your controller sets the RDY bits before it is ready? Could you check whether only checking for NDSR_RDY(0) changes anything? Not sure about the handling of NDSR_RDY(1) in this driver anyway ...

I suppose you mean this portion of code is not clear enough?

        u32 st = readl_relaxed(nfc->regs + NDSR);
        u32 ien = (~readl_relaxed(nfc->regs + NDCR)) & NDCR_ALL_INT;

        /*
         * RDY interrupt mask is one bit in NDCR while there are two status
         * bit in NDSR (RDY[cs0/cs2] and RDY[cs1/cs3]).
         */
        if (st & NDSR_RDY(1))
                st |= NDSR_RDY(0);

        if (!(st & ien))
                return IRQ_NONE;

-> st is the status in the NDSR register which has two RDY bits, one
for each RDY line.
-> ien is a view of the NDCR register which commands the interrupts
and has one bit to enable both interrupt lines (let's call it NDCR_RDYM).

The trick is that NDSR_RDY(0) is the same bit as NDCR_RDYM. So whenever NDSR_RDY(1) is set, we fake NDSR_RDY(0) to be set (by setting manually the bit in 'st') so that the (st & ien) comparison can be true if NDSR_RDY(1) is valid and RDY interrupts are enabled.

With this in mind, I don't see why this

+ st = readl_relaxed(nfc->regs + NDSR);
+ if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+ complete(&nfc->complete);

would break the driver.

Thanks,
Miquèl

#9 lore raw
Quoted 85 lines · Miquel Raynal

Hi Daniel,

Daniel Mack <[email protected]> wrote on Fri, 28 Sep 2018 09:43:18 +0200:

Quoted 41 lines

Hi Chris,

Quoted 38 lines · Chris Packham
Quoted 37 lines · Boris Brezillon
Quoted 33 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Queued to mtd/master.
After fixing my R/B configuration I get a new error with this patch when

running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

That's strange. So your controller sets the RDY bits before it is ready? Could you check whether only checking for NDSR_RDY(0) changes anything? Not sure about the handling of NDSR_RDY(1) in this driver anyway ...

I suppose you mean this portion of code is not clear enough?

         u32 st = readl_relaxed(nfc->regs + NDSR);
         u32 ien = (~readl_relaxed(nfc->regs + NDCR)) & NDCR_ALL_INT;

         /*
          * RDY interrupt mask is one bit in NDCR while there are two status
          * bit in NDSR (RDY[cs0/cs2] and RDY[cs1/cs3]).
          */
         if (st & NDSR_RDY(1))
                 st |= NDSR_RDY(0);

         if (!(st & ien))
                 return IRQ_NONE;

-> st is the status in the NDSR register which has two RDY bits, one
for each RDY line.
-> ien is a view of the NDCR register which commands the interrupts
and has one bit to enable both interrupt lines (let's call it NDCR_RDYM).

The trick is that NDSR_RDY(0) is the same bit as NDCR_RDYM. So whenever NDSR_RDY(1) is set, we fake NDSR_RDY(0) to be set (by setting manually the bit in 'st') so that the (st & ien) comparison can be true if NDSR_RDY(1) is valid and RDY interrupts are enabled.

With this in mind, I don't see why this

drivers/mtd/nand/raw/marvell_nand.c static int marvell_nfc_wait_op(struct nand_chip *chip, unsigned int timeout_ms) · line 633

Quoted 3 lines
drivers/mtd/nand/raw/marvell_nand.c +3 -0
+	st = readl_relaxed(nfc->regs + NDSR);
+	if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+		complete(&nfc->complete);

Yeah, me neither. Chris, are you absolutely sure this is the reason? I'm asking because it took me several tries sometimes to trigger the bug, so is there a chance that you see an error at all times, regardless of whether my patch is applied?

Thanks,
Daniel

#10 lore raw
Quoted 86 lines · Daniel Mack
Quoted 85 lines · Miquel Raynal

Hi Daniel,

Daniel Mack <[email protected]> wrote on Fri, 28 Sep 2018 09:43:18 +0200:

Quoted 41 lines

Hi Chris,

Quoted 38 lines · Chris Packham
Quoted 37 lines · Boris Brezillon
Quoted 33 lines · On Thu, 27 Sep 2018 10:11:45 +0200 Miquel Raynal <[email protected]>…

Hi Daniel,

Daniel Mack <[email protected]> wrote on Thu, 27 Sep 2018 09:17:51 +0200:

Quoted 16 lines

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 4 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

---

Sorry I haven't had the time to check on my Armada, but you figured it out, and the fix looks good to me!

Boris, do you plan to send another fixes PR of can I take it into the nand/next branch?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Queued to mtd/master.
After fixing my R/B configuration I get a new error with this patch when

running stress_1 from mtd-utils-2.0.0. I don't see this without the patch.

That's strange. So your controller sets the RDY bits before it is ready? Could you check whether only checking for NDSR_RDY(0) changes anything? Not sure about the handling of NDSR_RDY(1) in this driver anyway ...

I suppose you mean this portion of code is not clear enough?

          u32 st = readl_relaxed(nfc->regs + NDSR);
          u32 ien = (~readl_relaxed(nfc->regs + NDCR)) & NDCR_ALL_INT;

          /*
           * RDY interrupt mask is one bit in NDCR while there are two status
           * bit in NDSR (RDY[cs0/cs2] and RDY[cs1/cs3]).
           */
          if (st & NDSR_RDY(1))
                  st |= NDSR_RDY(0);

          if (!(st & ien))
                  return IRQ_NONE;

-> st is the status in the NDSR register which has two RDY bits, one
for each RDY line.
-> ien is a view of the NDCR register which commands the interrupts
and has one bit to enable both interrupt lines (let's call it NDCR_RDYM).

The trick is that NDSR_RDY(0) is the same bit as NDCR_RDYM. So whenever NDSR_RDY(1) is set, we fake NDSR_RDY(0) to be set (by setting manually the bit in 'st') so that the (st & ien) comparison can be true if NDSR_RDY(1) is valid and RDY interrupts are enabled.

With this in mind, I don't see why this

Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+	st = readl_relaxed(nfc->regs + NDSR);
+	if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+		complete(&nfc->complete);

Yeah, me neither. Chris, are you absolutely sure this is the reason? I'm asking because it took me several tries sometimes to trigger the bug, so is there a chance that you see an error at all times, regardless of whether my patch is applied?

It seems pretty consistent. Without this patch there seems to be no problem. With this patch it triggers pretty much straight away. I can't discount that there might be something wrong with my dts (the R/B configuration was missing initially).

I've also been able to run this on the DB-88F6820-AMC board with the same result (the dts for this is in the for-next branch of
git://git.infradead.org/linux-mvebu.git).

The really odd thing is the following seems to avoid the problem

+        st = readl_relaxed(nfc->regs + NDSR);
+        udelay(1000);
+        if (st & (NDSR_RDY(0) | NDSR_RDY(1)))

+ complete(&nfc->complete);

Which is weird because the st value has already been read so the udelay should have no effect.

Quoted 3 lines · Daniel Mack

Also, does my .EALREADY approach (v1) make any difference?

The v1 of this patch doesn't show the problem.

#11 lore raw
Quoted 25 lines · Chris Packham
Quoted 9 lines

Review

Quoted 5 lines

With this in mind, I don't see why this

(quoted diff) +3 -0
+	st = readl_relaxed(nfc->regs + NDSR);
+	if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+		complete(&nfc->complete);

Yeah, me neither. Chris, are you absolutely sure this is the reason? I'm asking because it took me several tries sometimes to trigger the bug, so is there a chance that you see an error at all times, regardless of whether my patch is applied?

It seems pretty consistent. Without this patch there seems to be no problem. With this patch it triggers pretty much straight away. I can't discount that there might be something wrong with my dts (the R/B configuration was missing initially).

I've also been able to run this on the DB-88F6820-AMC board with the same result (the dts for this is in the for-next branch of
git://git.infradead.org/linux-mvebu.git).

The really odd thing is the following seems to avoid the problem

(quoted diff) +4 -0
+        st = readl_relaxed(nfc->regs + NDSR);
+        udelay(1000);
+        if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+                complete(&nfc->complete);
Quoted 2 lines

Which is weird because the st value has already been read so the udelay should have no effect.

Erm, yes. That's totally weird. Which gcc are you using for this?

Could you please try and use readl() here instead of readl_relaxed()? That will place a memory barrier after the read to enforce ordering.

But if this is a problem, many other parts of that driver should be equally affected.

Quoted 6 lines · Daniel Mack
Quoted 3 lines

Also, does my .EALREADY approach (v1) make any difference?

The v1 of this patch doesn't show the problem.

That's also very strange because the condition it triggers on is exactly the same.

Thanks,
Daniel

#12 lore raw
Quoted 26 lines · Daniel Mack
Quoted 25 lines · Chris Packham
Quoted 9 lines

Review

Quoted 5 lines

With this in mind, I don't see why this

(quoted diff) +3 -0
+	st = readl_relaxed(nfc->regs + NDSR);
+	if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+		complete(&nfc->complete);

Yeah, me neither. Chris, are you absolutely sure this is the reason? I'm asking because it took me several tries sometimes to trigger the bug, so is there a chance that you see an error at all times, regardless of whether my patch is applied?

It seems pretty consistent. Without this patch there seems to be no problem. With this patch it triggers pretty much straight away. I can't discount that there might be something wrong with my dts (the R/B configuration was missing initially).

I've also been able to run this on the DB-88F6820-AMC board with the same result (the dts for this is in the for-next branch of
git://git.infradead.org/linux-mvebu.git).

The really odd thing is the following seems to avoid the problem

(quoted diff) +4 -0
+        st = readl_relaxed(nfc->regs + NDSR);
+        udelay(1000);
+        if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+                complete(&nfc->complete);
Quoted 4 lines
Quoted 2 lines

Which is weird because the st value has already been read so the udelay should have no effect.

Erm, yes. That's totally weird. Which gcc are you using for this?

arm-softfloat-linux-gnueabi-gcc (crosstool-NG crosstool-ng-1.22.0) 4.9.3

Quoted 2 lines

Could you please try and use readl() here instead of readl_relaxed()? That will place a memory barrier after the read to enforce ordering.

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

Quoted 5 lines

But if this is a problem, many other parts of that driver should be equally affected.

Quoted 2 lines · Daniel Mack
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Also, does my .EALREADY approach (v1) make any difference?

The v1 of this patch doesn't show the problem.

That's also very strange because the condition it triggers on is exactly the same.

One difference is that by calling complete() interrupts will be disabled in the spinlock.

Quoted 5 lines

Thanks,
Daniel

#13 lore raw
Quoted 37 lines · On Mon, 1 Oct 2018 19:59:11 +0000 Chris Packham…
Quoted 26 lines · Daniel Mack
Quoted 25 lines · Chris Packham
Quoted 9 lines

Review

Quoted 5 lines

With this in mind, I don't see why this

(quoted diff) +3 -0
+	st = readl_relaxed(nfc->regs + NDSR);
+	if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+		complete(&nfc->complete);  

Yeah, me neither. Chris, are you absolutely sure this is the reason? I'm asking because it took me several tries sometimes to trigger the bug, so is there a chance that you see an error at all times, regardless of whether my patch is applied?

It seems pretty consistent. Without this patch there seems to be no problem. With this patch it triggers pretty much straight away. I can't discount that there might be something wrong with my dts (the R/B configuration was missing initially).

I've also been able to run this on the DB-88F6820-AMC board with the same result (the dts for this is in the for-next branch of
git://git.infradead.org/linux-mvebu.git).

The really odd thing is the following seems to avoid the problem

(quoted diff) +4 -0
+        st = readl_relaxed(nfc->regs + NDSR);
+        udelay(1000);
+        if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
+                complete(&nfc->complete);
Quoted 4 lines
Quoted 2 lines

Which is weird because the st value has already been read so the udelay should have no effect.

Erm, yes. That's totally weird. Which gcc are you using for this?

arm-softfloat-linux-gnueabi-gcc (crosstool-NG crosstool-ng-1.22.0) 4.9.3

Quoted 2 lines

Could you please try and use readl() here instead of readl_relaxed()? That will place a memory barrier after the read to enforce ordering.

Quoted 7 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

#14 lore raw
Quoted 11 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

[1]http://code.bulix.org/lgs30c-468205

#15 lore raw
Quoted 12 lines · Boris Brezillon
Quoted 9 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

Quoted 8 lines
Quoted 2 lines

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

[1]http://code.bulix.org/lgs30c-468205

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

[1] - http://code.bulix.org/gc7a6d-468226

#16 lore raw
Quoted 19 lines · On Mon, 1 Oct 2018 22:01:27 +0000 Chris Packham…
Quoted 18 lines · Boris Brezillon
Quoted 11 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

Quoted 8 lines
Quoted 2 lines

[1]http://code.bulix.org/lgs30c-468205

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

Except we still don't know why this is happening, and I'm not sure I want to take a fix without understanding why it does fix the problem.

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/mtd/nand/raw/marvell_nand.c?h=v4.19-rc6#n689

#17 lore raw
Quoted 25 lines · Boris Brezillon
Quoted 22 lines · On Mon, 1 Oct 2018 22:01:27 +0000 Chris Packham…
Quoted 20 lines · Boris Brezillon
Quoted 11 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

[1]http://code.bulix.org/lgs30c-468205

Quoted 8 lines
Quoted 5 lines

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

Except we still don't know why this is happening, and I'm not sure I want to take a fix without understanding why it does fix the problem.

Agreed. My only guess is that there is some interrupt that is missed in the short period they are disabled when calling complete().

Quoted 6 lines

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/mtd/nand/raw/marvell_nand.c?h=v4.19-rc6#n689

#18 lore raw
Quoted 32 lines · On Mon, 1 Oct 2018 22:15:28 +0000 Chris Packham…
Quoted 31 lines · Boris Brezillon
Quoted 27 lines · On Mon, 1 Oct 2018 22:01:27 +0000 Chris Packham…
Quoted 12 lines · Boris Brezillon
Quoted 9 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

Quoted 8 lines
Quoted 2 lines

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

[1]http://code.bulix.org/lgs30c-468205

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

Quoted 5 lines
Quoted 2 lines

Except we still don't know why this is happening, and I'm not sure I want to take a fix without understanding why it does fix the problem.

Agreed. My only guess is that there is some interrupt that is missed in the short period they are disabled when calling complete().

Disabling interrupts when taking a spinlock means masking the IRQ line, but the interrupt still exists and should be there when linux unmasks the IRQ line. I don't think this is the problem we're chasing.

Looks more like something

Quoted 8 lines
Quoted 7 lines

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/mtd/nand/raw/marvell_nand.c?h=v4.19-rc6#n689

#19 lore raw
Quoted 38 lines · On Tue, 2 Oct 2018 11:36:10 +0200 Boris Brezillon…
Quoted 35 lines · On Mon, 1 Oct 2018 22:15:28 +0000 Chris Packham…
Quoted 33 lines · Boris Brezillon
Quoted 19 lines · On Mon, 1 Oct 2018 22:01:27 +0000 Chris Packham…
Quoted 18 lines · Boris Brezillon
Quoted 11 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

Quoted 8 lines
Quoted 2 lines

[1]http://code.bulix.org/lgs30c-468205

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

Except we still don't know why this is happening, and I'm not sure I want to take a fix without understanding why it does fix the problem.

Quoted 8 lines
Quoted 2 lines

Agreed. My only guess is that there is some interrupt that is missed in the short period they are disabled when calling complete().

Disabling interrupts when taking a spinlock means masking the IRQ line, but the interrupt still exists and should be there when linux unmasks the IRQ line. I don't think this is the problem we're chasing.

Looks more like something

Please ignore this email, I inadvertently hit the send button on a draft I started yesterday :-)

maintainer
#20 lore raw
In reply to Boris Brezillon

Hi Boris,

Boris Brezillon <[email protected]> wrote on Tue, 2 Oct 2018 00:13:28 +0200:

Quoted 31 lines
Quoted 19 lines · On Mon, 1 Oct 2018 22:01:27 +0000 Chris Packham…
Quoted 18 lines · Boris Brezillon
Quoted 11 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

Quoted 8 lines
Quoted 2 lines

[1]http://code.bulix.org/lgs30c-468205

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

Quoted 6 lines

Except we still don't know why this is happening, and I'm not sure I want to take a fix without understanding why it does fix the problem.

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

The RDDREQ, WRDREQ and WRCMDREQ events might potentially happen while the interrupts are enabled while we only wait for R/B signalling. This check is to avoid calling complete() on these situations.

Quoted 2 lines

[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/mtd/nand/raw/marvell_nand.c?h=v4.19-rc6#n689

Thanks,
Miquèl

maintainer
#21 lore raw

Hi Daniel, Chris,

Miquel Raynal <[email protected]> wrote on Tue, 2 Oct 2018 08:46:01 +0200:

Quoted 39 lines

Hi Boris,

Boris Brezillon <[email protected]> wrote on Tue, 2 Oct 2018 00:13:28 +0200:

Quoted 34 lines
Quoted 19 lines · On Mon, 1 Oct 2018 22:01:27 +0000 Chris Packham…
Quoted 18 lines · Boris Brezillon
Quoted 11 lines · On Mon, 1 Oct 2018 22:34:38 +0200 Boris Brezillon…
Quoted 8 lines

I'd previously tried readl() based on the same hunch. No change.

I think my snippet above might be misleading. While a delay between readl_relaxed() and the if should not change the outcome, this is also a delay between marvell_nfc_enable_int() and marvell_nfc_disable_int() which is probably more significant. Sure enough if I move the delay to just before the marvell_nfc_disable_int() the error is not seen.

AFAICT, your timeout always happens when waiting for RDREQ, not RDYM. So maybe disabling MRDY too early has a side-effect on the RDREQ event.

Can you try with this patch [1]? It should ensure that NDSR_RDY bits are cleared before starting an operation.

Quoted 8 lines
Quoted 2 lines

[1]http://code.bulix.org/lgs30c-468205

No luck. I applied that on top of Daniel's and got the same result.

One thing that does look promising is the following modification of Daniel's patch[1]. Which moves the RDY check to before where the interrupts are enabled.

Except we still don't know why this is happening, and I'm not sure I want to take a fix without understanding why it does fix the problem.

Quoted 7 lines
Quoted 3 lines

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

The RDDREQ, WRDREQ and WRCMDREQ events might potentially happen while the interrupts are enabled while we only wait for R/B signalling. This check is to avoid calling complete() on these situations.

Actually Boris is right on the fact that while the intention is good, the writing of [1] is not accurate. Daniel, could you please test if the following diff changes something with your setup, without your patch?

drivers/mtd/nand/raw/marvell_nand.c +1 -1
index bc2ef5209783..c7573ccdbacd 100644
@@ -686,7 +686,7 @@ static irqreturn_t marvell_nfc_isr(int irq, void *dev_id)
 
        marvell_nfc_disable_int(nfc, st & NDCR_ALL_INT);
 
-       if (!(st & (NDSR_RDDREQ | NDSR_WRDREQ | NDSR_WRCMDREQ)))
+       if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
                complete(&nfc->complete);
 
        return IRQ_HANDLED;
Quoted 4 lines
Quoted 3 lines

[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/mtd/nand/raw/marvell_nand.c?h=v4.19-rc6#n689

Thanks,
Miquèl

#22 lore raw

Hi Miquel,

Quoted 24 lines · Miquel Raynal

Miquel Raynal <[email protected]> wrote on Tue, 2 Oct 2018 08:46:01 +0200:

Quoted 10 lines

Boris Brezillon <[email protected]> wrote on Tue, 2 Oct 2018 00:13:28 +0200:

Quoted 3 lines

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

The RDDREQ, WRDREQ and WRCMDREQ events might potentially happen while the interrupts are enabled while we only wait for R/B signalling. This check is to avoid calling complete() on these situations.

Actually Boris is right on the fact that while the intention is good, the writing of [1] is not accurate. Daniel, could you please test if the following diff changes something with your setup, without your patch?

drivers/mtd/nand/raw/marvell_nand.c +0 -0
index bc2ef5209783..c7573ccdbacd 100644

drivers/mtd/nand/raw/marvell_nand.c static irqreturn_t marvell_nfc_isr(int irq, void *dev_id) · line 692

Quoted 8 lines
drivers/mtd/nand/raw/marvell_nand.c +1 -1
@@ -686,7 +686,7 @@ static irqreturn_t marvell_nfc_isr(int irq, void *dev_id)
 
         marvell_nfc_disable_int(nfc, st & NDCR_ALL_INT);
 
-       if (!(st & (NDSR_RDDREQ | NDSR_WRDREQ | NDSR_WRCMDREQ)))
+       if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
                 complete(&nfc->complete);
 
         return IRQ_HANDLED;

Yes! That seems to work nicely as a replacement for my patch.

Chris, how is that going on your board?

Thanks,
Daniel

#23 lore raw
Quoted 34 lines · Daniel Mack

Hi Miquel,

Quoted 31 lines · Miquel Raynal

Miquel Raynal <[email protected]> wrote on Tue, 2 Oct 2018 08:46:01 +0200:

Quoted 10 lines

Boris Brezillon <[email protected]> wrote on Tue, 2 Oct 2018 00:13:28 +0200:

Quoted 3 lines

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

The RDDREQ, WRDREQ and WRCMDREQ events might potentially happen while the interrupts are enabled while we only wait for R/B signalling. This check is to avoid calling complete() on these situations.

Actually Boris is right on the fact that while the intention is good, the writing of [1] is not accurate. Daniel, could you please test if the following diff changes something with your setup, without your patch?

drivers/mtd/nand/raw/marvell_nand.c +1 -1
index bc2ef5209783..c7573ccdbacd 100644
@@ -686,7 +686,7 @@ static irqreturn_t marvell_nfc_isr(int irq, void *dev_id)
 
          marvell_nfc_disable_int(nfc, st & NDCR_ALL_INT);
 
-       if (!(st & (NDSR_RDDREQ | NDSR_WRDREQ | NDSR_WRCMDREQ)))
+       if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
                  complete(&nfc->complete);
 
Quoted 5 lines

drivers/mtd/nand/raw/marvell_nand.c static irqreturn_t marvell_nfc_isr(int irq, void *dev_id) · line 692

Quoted 1 line
drivers/mtd/nand/raw/marvell_nand.c +0 -0
          return IRQ_HANDLED;

Yes! That seems to work nicely as a replacement for my patch.

Chris, how is that going on your board?

Looks good to me. I've just re-run stress_{1,2,3} on the custom board and DB-88F6820-AMC.

maintainer
#24 lore raw

Hi Chris,

Chris Packham <[email protected]> wrote on Tue, 2 Oct 2018 20:53:14 +0000:

Quoted 35 lines · Daniel Mack
Quoted 34 lines

Hi Miquel,

Quoted 31 lines · Miquel Raynal

Miquel Raynal <[email protected]> wrote on Tue, 2 Oct 2018 08:46:01 +0200:

Quoted 10 lines

Boris Brezillon <[email protected]> wrote on Tue, 2 Oct 2018 00:13:28 +0200:

Quoted 3 lines

Also, it looks like complete() is not called until the RDDREQ, WRDREQ and WRCMDREQ are cleared in the interrupt handler [1], which is weird. Miquel, do you happen to remember why you had to do that?

The RDDREQ, WRDREQ and WRCMDREQ events might potentially happen while the interrupts are enabled while we only wait for R/B signalling. This check is to avoid calling complete() on these situations.

Actually Boris is right on the fact that while the intention is good, the writing of [1] is not accurate. Daniel, could you please test if the following diff changes something with your setup, without your patch?

drivers/mtd/nand/raw/marvell_nand.c +1 -1
index bc2ef5209783..c7573ccdbacd 100644
@@ -686,7 +686,7 @@ static irqreturn_t marvell_nfc_isr(int irq, void *dev_id)
 
          marvell_nfc_disable_int(nfc, st & NDCR_ALL_INT);
 
-       if (!(st & (NDSR_RDDREQ | NDSR_WRDREQ | NDSR_WRCMDREQ)))
+       if (st & (NDSR_RDY(0) | NDSR_RDY(1)))
                  complete(&nfc->complete);
 
Quoted 8 lines
Quoted 5 lines

drivers/mtd/nand/raw/marvell_nand.c static irqreturn_t marvell_nfc_isr(int irq, void *dev_id) · line 692

Quoted 1 line
drivers/mtd/nand/raw/marvell_nand.c +0 -0
          return IRQ_HANDLED;  

Yes! That seems to work nicely as a replacement for my patch.

Chris, how is that going on your board?

Looks good to me. I've just re-run stress_{1,2,3} on the custom board and DB-88F6820-AMC.

Great! There is still something weird about the complete() though.

Daniel, do you plan to send the above change or do you want me to do it? I would like to integrate the fix in nand/next before sending the PR.

Thanks,
Miquèl

#25 lore raw

Hi Miquel,

Quoted 10 lines · Miquel Raynal
Quoted 5 lines
Quoted 3 lines

Yes! That seems to work nicely as a replacement for my patch.

Chris, how is that going on your board?

Looks good to me. I've just re-run stress_{1,2,3} on the custom board and DB-88F6820-AMC.

Great! There is still something weird about the complete() though.

Daniel, do you plan to send the above change or do you want me to do it? I would like to integrate the fix in nand/next before sending the PR.

Nah, your version has barely anything to do with my patch anymore, except that it fixes the same effect, so go ahead please :)

You can add my Reported-and-tested-by though.

Thanks for chasing this, everyone!

Daniel

#26 lore raw
In reply to Daniel Mack

Hi Daniel,

Quoted 11 lines · On Thu, 27 Sep 2018 09:17:51 +0200 Daniel Mack <[email protected]> wrote:

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

Quoted 8 lines

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

Can you try to replace your patch by the following one and let me know if it solves your problem?

Thanks,

Boris

--->8---

drivers/mtd/nand/raw/marvell_nand.c +1 -1
index e63f714f7639..295a86a5545f 100644
@@ -1123,7 +1123,7 @@ static int marvell_nfc_hw_ecc_hmg_do_write_page(struct nand_chip *chip,
                memcpy(nfc->dma_buf, data_buf, lt->data_bytes);
                memcpy(nfc->dma_buf + lt->data_bytes, oob_buf, oob_bytes);
                marvell_nfc_xfer_data_dma(nfc, DMA_TO_DEVICE, lt->data_bytes +
-                                         lt->ecc_bytes + lt->spare_bytes);
+                                         oob_bytes);
        } else {
                marvell_nfc_xfer_data_out_pio(nfc, data_buf, lt->data_bytes);
                marvell_nfc_xfer_data_out_pio(nfc, oob_buf, oob_bytes);
#27 lore raw

Hi Boris,

Quoted 19 lines · Boris Brezillon
Quoted 16 lines · On Thu, 27 Sep 2018 09:17:51 +0200 Daniel Mack <[email protected]> wrote:

At least on PXA3xx platforms, enabling RDY interrupts in the NDCR register will only cause the IRQ to latch when the RDY lanes are changing, and not in case they are already asserted.

This means that if the controller finished the command in flight before marvell_nfc_wait_op() is called, that function will wait for a change in the bit that can't ever happen as it is already set.

To address this race, check for the RDY bits after the IRQ was enabled, and complete the completion immediately if the condition is already met.

This fixes a bug that was observed with a NAND chip that holds a UBIFS parition on which file system stress tests were executed. When marvell_nfc_wait_op() reports an error, UBI/UBIFS will eventually mount the filesystem read-only, reporting lots of warnings along the way.

Quoted 6 lines
Quoted 3 lines
Fixes 02f26ecf8c77 mtd: nand: add reworked Marvell NAND controller driver

Can you try to replace your patch by the following one and let me know if it solves your problem?

Nope, this doesn't fix it. Same problem as before.

Thanks,
Daniel

Quoted 15 lines

--->8---

drivers/mtd/nand/raw/marvell_nand.c +1 -1
index e63f714f7639..295a86a5545f 100644
@@ -1123,7 +1123,7 @@ static int marvell_nfc_hw_ecc_hmg_do_write_page(struct nand_chip *chip,
                 memcpy(nfc->dma_buf, data_buf, lt->data_bytes);
                 memcpy(nfc->dma_buf + lt->data_bytes, oob_buf, oob_bytes);
                 marvell_nfc_xfer_data_dma(nfc, DMA_TO_DEVICE, lt->data_bytes +
-                                         lt->ecc_bytes + lt->spare_bytes);
+                                         oob_bytes);
         } else {
                 marvell_nfc_xfer_data_out_pio(nfc, data_buf, lt->data_bytes);
                 marvell_nfc_xfer_data_out_pio(nfc, oob_buf, oob_bytes);