Skip to contentSkip to conversation

slight UBI scan time improvement

RFC In review Artem Bityutskiy 5 participants 26 messages Apr 22, 2008 – Apr 24, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 slight UBI scan time improvement

Hi,

I've prepared 2 UBI patches which may slightly improve the scan time. I am not sure though. Would you guys please try them and tell if UBI scan time changed? Thanks in advance.

Attached.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
Quoted 1 line

From b9207c3481acb395c36ca9e039dedaa440b6e881 Mon Sep 17 00:00:00 2001

From: Artem Bityutskiy <[email protected]>
Date: Tue, 22 Apr 2008 18:12:40 +0300
Subject: [PATCH] UBI: separate out header checking

This is a preparation patch which separates out the checking stuff from the EC and VID read&check functions. This will be needed for the follow-up scanning improvement patch.

---
 drivers/mtd/ubi/io.c  |  250 +++++++++++++++++++++++++++++++------------------
 drivers/mtd/ubi/ubi.h |    4 +
 2 files changed, 164 insertions(+), 90 deletions(-)
drivers/mtd/ubi/io.c +160 -90
index 4ac11df..f62728e 100644
@@ -604,70 +604,38 @@ bad:
 }
 
 /**
- * ubi_io_read_ec_hdr - read and check an erase counter header.
+ * ubi_io_check_ec_hdr - check an erase counter header.
  * @ubi: UBI device description object
- * @pnum: physical eraseblock to read from
- * @ec_hdr: a &struct ubi_ec_hdr object where to store the read erase counter
- * header
- * @verbose: be verbose if the header is corrupted or was not found
+ * @pnum: physical eraseblock the header was read from
+ * @ec_hdr: the header to check
+ * @verbose: be verbose if the header is corrupted
  *
- * This function reads erase counter header from physical eraseblock @pnum and
- * stores it in @ec_hdr. This function also checks CRC checksum of the read
- * erase counter header. The following codes may be returned:
+ * This function check erase counter header which was read from physical
+ * eraseblock @pnum. The following codes may be returned:
  *
  * o %0 if the CRC checksum is correct and the header was successfully read;
- * o %UBI_IO_BITFLIPS if the CRC is correct, but bit-flips were detected
- *   and corrected by the flash driver; this is harmless but may indicate that
- *   this eraseblock may become bad soon (but may be not);
  * o %UBI_IO_BAD_EC_HDR if the erase counter header is corrupted (a CRC error);
  * o %UBI_IO_PEB_EMPTY if the physical eraseblock is empty;
  * o a negative error code in case of failure.
  */
-int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum,
-		       struct ubi_ec_hdr *ec_hdr, int verbose)
+int ubi_io_check_ec_hdr(struct ubi_device *ubi, int pnum,
+			const struct ubi_ec_hdr *ec_hdr, int verbose)
 {
-	int err, read_err = 0;
 	uint32_t crc, magic, hdr_crc;
 
-	dbg_io("read EC header from PEB %d", pnum);
-	ubi_assert(pnum >= 0 && pnum < ubi->peb_count);
-	if (UBI_IO_DEBUG)
-		verbose = 1;
-
-	err = ubi_io_read(ubi, ec_hdr, pnum, 0, UBI_EC_HDR_SIZE);
-	if (err) {
-		if (err != UBI_IO_BITFLIPS && err != -EBADMSG)
-			return err;
-
-		/*
-		 * We read all the data, but either a correctable bit-flip
-		 * occurred, or MTD reported about some data integrity error,
-		 * like an ECC error in case of NAND. The former is harmless,
-		 * the later may mean that the read data is corrupted. But we
-		 * have a CRC check-sum and we will detect this. If the EC
-		 * header is still OK, we just report this as there was a
-		 * bit-flip.
-		 */
-		read_err = err;
-	}
-
 	magic = be32_to_cpu(ec_hdr->magic);
 	if (magic != UBI_EC_HDR_MAGIC) {
 		/*
 		 * The magic field is wrong. Let's check if we have read all
 		 * 0xFF. If yes, this physical eraseblock is assumed to be
 		 * empty.
-		 *
-		 * But if there was a read error, we do not test it for all
-		 * 0xFFs. Even if it does contain all 0xFFs, this error
-		 * indicates that something is still wrong with this physical
-		 * eraseblock and we anyway cannot treat it as empty.
 		 */
-		if (read_err != -EBADMSG &&
-		    check_pattern(ec_hdr, 0xFF, UBI_EC_HDR_SIZE)) {
-			/* The physical eraseblock is supposedly empty */
+		if (check_pattern(ec_hdr, 0xFF, UBI_EC_HDR_SIZE)) {
+			int err;
 
 			/*
+			 * The physical eraseblock is supposedly empty.
+			 *
 			 * The below is just a paranoid check, it has to be
 			 * compiled out if paranoid checks are disabled.
 			 */
@@ -675,7 +643,6 @@ int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum,
 						    ubi->peb_size);
 			if (err)
 				return err > 0 ? UBI_IO_BAD_EC_HDR : err;
-
 			if (verbose)
 				ubi_warn("no EC header found at PEB %d, "
 					 "only 0xFF bytes", pnum);
@@ -691,6 +658,7 @@ int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum,
 				 "%08x", pnum, magic, UBI_EC_HDR_MAGIC);
 			ubi_dbg_dump_ec_hdr(ec_hdr);
 		}
+
 		return UBI_IO_BAD_EC_HDR;
 	}
 
@@ -706,6 +674,72 @@ int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum,
 		return UBI_IO_BAD_EC_HDR;
 	}
 
+	return 0;
+}
+
+/**
+ * ubi_io_read_ec_hdr - read and check an erase counter header.
+ * @ubi: UBI device description object
+ * @pnum: physical eraseblock to read from
+ * @ec_hdr: a &struct ubi_ec_hdr object where to store the read erase counter
+ * header
+ * @verbose: be verbose if the header is corrupted or was not found
+ *
+ * This function reads erase counter header from physical eraseblock @pnum and
+ * stores it in @ec_hdr. This function also checks CRC checksum of the read
+ * erase counter header. The following codes may be returned:
+ *
+ * o %0 if the CRC checksum is correct and the header was successfully read;
+ * o %UBI_IO_BITFLIPS if the CRC is correct, but bit-flips were detected
+ *   and corrected by the flash driver; this is harmless but may indicate that
+ *   this eraseblock may become bad soon (but may be not);
+ * o %UBI_IO_BAD_EC_HDR if the erase counter header is corrupted (a CRC error);
+ * o %UBI_IO_PEB_EMPTY if the physical eraseblock is empty;
+ * o a negative error code in case of failure.
+ */
+int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum,
+		       struct ubi_ec_hdr *ec_hdr, int verbose)
+{
+	int err, read_err = 0;
+
+	dbg_io("read EC header from PEB %d", pnum);
+	ubi_assert(pnum >= 0 && pnum < ubi->peb_count);
+	if (UBI_IO_DEBUG)
+		verbose = 1;
+
+	err = ubi_io_read(ubi, ec_hdr, pnum, 0, UBI_EC_HDR_SIZE);
+	if (err) {
+		if (err != UBI_IO_BITFLIPS && err != -EBADMSG)
+			return err;
+
+		/*
+		 * We read all the data, but either a correctable bit-flip
+		 * occurred, or MTD reported about some data integrity error,
+		 * like an ECC error in case of NAND. The former is harmless,
+		 * the later may mean that the read data is corrupted. But we
+		 * have a CRC check-sum and we will detect this. If the EC
+		 * header is still OK, we just report this as there was a
+		 * bit-flip.
+		 */
+		read_err = err;
+	}
+
+	err = ubi_io_check_ec_hdr(ubi, pnum, ec_hdr, verbose);
+	if (err == UBI_IO_PEB_EMPTY && read_err) {
+		/*
+		 * There was a read error, and the read data contains only
+		 * 0xFFs. Do not treat this physical eraseblock as empty
+		 * because the read error indicates that something is still
+		 * wrong with it.
+		 */
+		if (verbose)
+			ubi_warn("PEB %d is supposedly empty, but there was "
+				 "read a error (%d)", pnum, read_err);
+		return UBI_IO_BAD_EC_HDR;
+	}
+	if (err)
+		return err;
+
 	/* And of course validate what has just been read from the media */
 	err = validate_ec_hdr(ubi, ec_hdr);
 	if (err) {
@@ -876,73 +910,39 @@ bad:
 }
 
 /**
- * ubi_io_read_vid_hdr - read and check a volume identifier header.
+ * ubi_io_check_vid_hdr - check a volume identifier header.
  * @ubi: UBI device description object
- * @pnum: physical eraseblock number to read from
- * @vid_hdr: &struct ubi_vid_hdr object where to store the read volume
- * identifier header
+ * @pnum: physical eraseblock number the header was read from
+ * @vid_hdr: the header to check
  * @verbose: be verbose if the header is corrupted or wasn't found
  *
- * This function reads the volume identifier header from physical eraseblock
- * @pnum and stores it in @vid_hdr. It also checks CRC checksum of the read
- * volume identifier header. The following codes may be returned:
+ * This function checks the volume identifier header @vid_hdr which was read
+ * from physical eraseblock @pnum. The following codes may be returned:
  *
  * o %0 if the CRC checksum is correct and the header was successfully read;
- * o %UBI_IO_BITFLIPS if the CRC is correct, but bit-flips were detected
- *   and corrected by the flash driver; this is harmless but may indicate that
- *   this eraseblock may become bad soon;
  * o %UBI_IO_BAD_VID_HRD if the volume identifier header is corrupted (a CRC
  *   error detected);
  * o %UBI_IO_PEB_FREE if the physical eraseblock is free (i.e., there is no VID
  *   header there);
  * o a negative error code in case of failure.
  */
-int ubi_io_read_vid_hdr(struct ubi_device *ubi, int pnum,
-			struct ubi_vid_hdr *vid_hdr, int verbose)
+int ubi_io_check_vid_hdr(struct ubi_device *ubi, int pnum,
+			 const struct ubi_vid_hdr *vid_hdr, int verbose)
 {
-	int err, read_err = 0;
 	uint32_t crc, magic, hdr_crc;
-	void *p;
-
-	dbg_io("read VID header from PEB %d", pnum);
-	ubi_assert(pnum >= 0 &&  pnum < ubi->peb_count);
-	if (UBI_IO_DEBUG)
-		verbose = 1;
-
-	p = (char *)vid_hdr - ubi->vid_hdr_shift;
-	err = ubi_io_read(ubi, p, pnum, ubi->vid_hdr_aloffset,
-			  ubi->vid_hdr_alsize);
-	if (err) {
-		if (err != UBI_IO_BITFLIPS && err != -EBADMSG)
-			return err;
-
-		/*
-		 * We read all the data, but either a correctable bit-flip
-		 * occurred, or MTD reported about some data integrity error,
-		 * like an ECC error in case of NAND. The former is harmless,
-		 * the later may mean the read data is corrupted. But we have a
-		 * CRC check-sum and we will identify this. If the VID header is
-		 * still OK, we just report this as there was a bit-flip.
-		 */
-		read_err = err;
-	}
 
 	magic = be32_to_cpu(vid_hdr->magic);
 	if (magic != UBI_VID_HDR_MAGIC) {
 		/*
 		 * If we have read all 0xFF bytes, the VID header probably does
 		 * not exist and the physical eraseblock is assumed to be free.
-		 *
-		 * But if there was a read error, we do not test the data for
-		 * 0xFFs. Even if it does contain all 0xFFs, this error
-		 * indicates that something is still wrong with this physical
-		 * eraseblock and it cannot be regarded as free.
 		 */
-		if (read_err != -EBADMSG &&
-		    check_pattern(vid_hdr, 0xFF, UBI_VID_HDR_SIZE)) {
-			/* The physical eraseblock is supposedly free */
+		if (check_pattern(vid_hdr, 0xFF, UBI_VID_HDR_SIZE)) {
+			int err;
 
 			/*
+			 * The physical eraseblock is supposedly free.
+			 *
 			 * The below is just a paranoid check, it has to be
 			 * compiled out if paranoid checks are disabled.
 			 */
@@ -981,6 +981,76 @@ int ubi_io_read_vid_hdr(struct ubi_device *ubi, int pnum,
 		return UBI_IO_BAD_VID_HDR;
 	}
 
+	return 0;
+}
+
+/**
+ * ubi_io_read_vid_hdr - read and check a volume identifier header.
+ * @ubi: UBI device description object
+ * @pnum: physical eraseblock number to read from
+ * @vid_hdr: &struct ubi_vid_hdr object where to store the read volume
+ * identifier header
+ * @verbose: be verbose if the header is corrupted or wasn't found
+ *
+ * This function reads the volume identifier header from physical eraseblock
+ * @pnum and stores it in @vid_hdr. It also checks CRC checksum of the read
+ * volume identifier header. The following codes may be returned:
+ *
+ * o %0 if the CRC checksum is correct and the header was successfully read;
+ * o %UBI_IO_BITFLIPS if the CRC is correct, but bit-flips were detected
+ *   and corrected by the flash driver; this is harmless but may indicate that
+ *   this eraseblock may become bad soon;
+ * o %UBI_IO_BAD_VID_HRD if the volume identifier header is corrupted (a CRC
+ *   error detected);
+ * o %UBI_IO_PEB_FREE if the physical eraseblock is free (i.e., there is no VID
+ *   header there);
+ * o a negative error code in case of failure.
+ */
+int ubi_io_read_vid_hdr(struct ubi_device *ubi, int pnum,
+			struct ubi_vid_hdr *vid_hdr, int verbose)
+{
+	int err, read_err = 0;
+	void *p;
+
+	dbg_io("read VID header from PEB %d", pnum);
+	ubi_assert(pnum >= 0 &&  pnum < ubi->peb_count);
+	if (UBI_IO_DEBUG)
+		verbose = 1;
+
+	p = (char *)vid_hdr - ubi->vid_hdr_shift;
+	err = ubi_io_read(ubi, p, pnum, ubi->vid_hdr_aloffset,
+			  ubi->vid_hdr_alsize);
+	if (err) {
+		if (err != UBI_IO_BITFLIPS && err != -EBADMSG)
+			return err;
+
+		/*
+		 * We read all the data, but either a correctable bit-flip
+		 * occurred, or MTD reported about some data integrity error,
+		 * like an ECC error in case of NAND. The former is harmless,
+		 * the later may mean the read data is corrupted. But we have a
+		 * CRC check-sum and we will identify this. If the VID header is
+		 * still OK, we just report this as there was a bit-flip.
+		 */
+		read_err = err;
+	}
+
+	err = ubi_io_check_vid_hdr(ubi, pnum, vid_hdr, verbose);
+	if (err == UBI_IO_PEB_EMPTY && read_err) {
+		/*
+		 * There was a read error, and the read data contains only
+		 * 0xFFs. Do not treat this physical eraseblock as empty
+		 * because the read error indicates that something is still
+		 * wrong with it.
+		 */
+		if (verbose)
+			ubi_warn("PEB %d is supposedly empty, but there was a "
+				 "read error (%d)", pnum, read_err);
+		return UBI_IO_BAD_VID_HDR;
+	}
+	if (err)
+		return err;
+
 	/* Validate the VID header that we have just read */
 	err = validate_vid_hdr(ubi, vid_hdr);
 	if (err) {
drivers/mtd/ubi/ubi.h +4 -0
index 28de80f..4e36028 100644
@@ -496,10 +496,14 @@ int ubi_io_write(struct ubi_device *ubi, const void *buf, int pnum, int offset,
 int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture);
 int ubi_io_is_bad(const struct ubi_device *ubi, int pnum);
 int ubi_io_mark_bad(const struct ubi_device *ubi, int pnum);
+int ubi_io_check_ec_hdr(struct ubi_device *ubi, int pnum,
+			const struct ubi_ec_hdr *ec_hdr, int verbose);
 int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum,
 		       struct ubi_ec_hdr *ec_hdr, int verbose);
 int ubi_io_write_ec_hdr(struct ubi_device *ubi, int pnum,
 			struct ubi_ec_hdr *ec_hdr);
+int ubi_io_check_vid_hdr(struct ubi_device *ubi, int pnum,
+			 const struct ubi_vid_hdr *vid_hdr, int verbose);
 int ubi_io_read_vid_hdr(struct ubi_device *ubi, int pnum,
 			struct ubi_vid_hdr *vid_hdr, int verbose);
 int ubi_io_write_vid_hdr(struct ubi_device *ubi, int pnum,
Signature · 1 line
1.5.4.1
Quoted 1 line

From 822dfc1c4799743b0edad0848f7ef6ffc30294f1 Mon Sep 17 00:00:00 2001

From: Artem Bityutskiy <[email protected]>
Date: Tue, 22 Apr 2008 18:41:05 +0300
Subject: [PATCH] UBI: optimize scanning a bit

When scanning the MTD device, we do 2 read requests - one for the EC header and one for the VID header. These are small 64-byte requests. In fact, MTD anyway reads whole NAND page and we might as well make only one read request.

Some flashes have a kind of "burst" read which makes it faster to read 2 consequtive NAND pages at once than reading them separately. This optimization is targeted on such flashes.

---
 drivers/mtd/ubi/scan.c |  104 ++++++++++++++++++++++++++---------------------
 drivers/mtd/ubi/scan.h |   11 +++++-
 2 files changed, 67 insertions(+), 48 deletions(-)
drivers/mtd/ubi/scan.c +57 -47

251 lines not on this page · raw message

drivers/mtd/ubi/scan.h +10 -1
index 966b9b6..57cf793 100644
@@ -93,7 +93,7 @@ struct ubi_scan_volume {
  * @erase: list of physical eraseblocks which have to be erased
  * @alien: list of physical eraseblocks which should not be used by UBI (e.g.,
  * @bad_peb_count: count of bad physical eraseblocks
- * those belonging to "preserve"-compatible internal volumes)
+ *                 those belonging to "preserve"-compatible internal volumes)
  * @vols_found: number of volumes found during scanning
  * @highest_vol_id: highest volume ID
  * @alien_peb_count: count of physical eraseblocks in the @alien list
@@ -104,6 +104,11 @@ struct ubi_scan_volume {
  * @mean_ec: mean erase counter value
  * @ec_sum: a temporary variable used when calculating @mean_ec
  * @ec_count: a temporary variable used when calculating @mean_ec
+ * @read_buf: temporary buffer where the EC and VID headers are read during
+ *            scanning
+ * @ech: points to the beginning of the EC header in @buf
+ * @vidh: points to the beginning of the VID header in @buf
+ * @read_len: how much is read from each eraseblock during scanning
  *
  * This data structure contains the result of scanning and may be used by other
  * UBI units to build final UBI data structures, further error-recovery and so
@@ -126,6 +131,10 @@ struct ubi_scan_info {
 	int mean_ec;
 	uint64_t ec_sum;
 	int ec_count;
+	void *read_buf;
+	struct ubi_ec_hdr *ech;
+	struct ubi_vid_hdr *vidh;
+	int read_len;
 };
 
 struct ubi_device;
Signature · 1 line
1.5.4.1
#1 lore raw
Quoted 5 lines

I've prepared 2 UBI patches which may slightly improve the scan time. I am not sure though. Would you guys please try them and tell if UBI scan time changed? Thanks in advance.

Hi Artem,

I'll be happy to do so, but it may be a couple of days before I can get results to you. I'm on the road for my company for a few days. If I can, I'll try it out before I leave tonight, but if I can't get to it, it'll be tommorrow night before I can get my equipment set up and running again.

See 'ya!

Bruce

#2 lore raw
Quoted 4 lines · On Tue, 2008-04-22 at 10:28 -0700, [email protected] wrote:

I'll be happy to do so, but it may be a couple of days before I can get results to you. I'm on the road for my company for a few days. If I can, I'll try it out before I leave tonight, but if I can't get to it, it'll be tommorrow night before I can get my equipment set up and running again.

Thanks Bruce,

no problem, this is not urgent and not that important.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#3 lore raw
In reply to Artem Bityutskiy
Quoted 3 lines

I've prepared 2 UBI patches which may slightly improve the scan time. I am not sure though. Would you guys please try them and tell if UBI scan time changed? Thanks in advance.

I have tried, as you said slightly improve the scan time, slightly, even don't feel it exist. en....I don't like the two stages displaying when ubi attach to mtd. First time I see it, I doubt the NFS server is too busy. It is just like a kid playing sliding board who do not like being block in the middle, or a group of men "cheers!", then drink off the beer. Oh, don't make it stop in the half way.

You still do not add patch to handle write fail issue, a little disappointment.

Signature · 2 lines
Best wishes,
Nancy
#4 lore raw
Quoted 6 lines · Nancy

I have tried, as you said slightly improve the scan time, slightly, even don't feel it exist. en....I don't like the two stages displaying when ubi attach to mtd. First time I see it, I doubt the NFS server is too busy. It is just like a kid playing sliding board who do not like being block in the middle, or a group of men "cheers!", then drink off the beer. Oh, don't make it stop in the half way.

Hmm, may be. But the reason for this is that we want to see this information even if UBI refuses your MTD device later.

Quoted 1 line

You still do not add patch to handle write fail issue, a little disappointment.

I do not see any UBI/UBIFS issue in your case. The issue was that you abused your flash using wrong tools, I think. Use the "ubiformat" utility which we have recently implemented - the issue should go.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#5 lore raw
Quoted 10 lines · Artem Bityutskiy
Quoted 6 lines · Nancy

I have tried, as you said slightly improve the scan time, slightly, even don't feel it exist. en....I don't like the two stages displaying when ubi attach to mtd. First time I see it, I doubt the NFS server is too busy. It is just like a kid playing sliding board who do not like being block in the middle, or a group of men "cheers!", then drink off the beer. Oh, don't make it stop in the half way.

Hmm, may be. But the reason for this is that we want to see this information even if UBI refuses your MTD device later.

Thank you for your explaination.

Quoted 5 lines
Quoted 1 line

You still do not add patch to handle write fail issue, a little disappointment.

I do not see any UBI/UBIFS issue in your case. The issue was that you abused your flash using wrong tools, I think. Use the "ubiformat" utility which we have recently implemented - the issue should go.

Thanks for the "ubiformat " tool.
Yes, the issue is go away by properly use MLC nand. But the write fail issue in UBI still exist! Suppose you are
properly used your Nand, when it return write fail, but can erase successfully, UBI still group this block in good block heap. That is wrong and dangerous.

Signature · 2 lines
Best wishes,
Nancy
#6 lore raw
Quoted 5 lines · Nancy

Yes, the issue is go away by properly use MLC nand. But the write fail issue in UBI still exist! Suppose you are
properly used your Nand, when it return write fail, but can erase successfully, UBI still group this block in good block heap. That is wrong and dangerous.

The driver may return I/O errors for many reasons, including bugs in the driver. Or this may be an dumb user who screwed up his flash. It might be something else. We do not wand to mark the eraseblock bad when it is _not_ bad.

And it's not just erase successfully. We torture the eraseblock. We erase it, read it back, check it has all 0xFFs, then write 0xA5 pattern, read it back, check, erase again, read back, check, write 0x5A pattern, read back, check, erase, read back, check, write 0x00 pattern, read back, check, erase, read back check. Enough? If it is not enough, feel free to send a patch for better torturing.

Please, refer the torture_peb() function for more details.

So, we do not mark eraseblock as bad if it passed the torture test. Note, a bit-flip during torture test is enough to mark the eraseblock bad as well.

In your particular case, you would have _all_ eraseblocks marked as bad if UBI did what you want.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#7 lore raw
Quoted 19 lines · Artem Bityutskiy
Quoted 5 lines · Nancy

Yes, the issue is go away by properly use MLC nand. But the write fail issue in UBI still exist! Suppose you are
properly used your Nand, when it return write fail, but can erase successfully, UBI still group this block in good block heap. That is wrong and dangerous.

The driver may return I/O errors for many reasons, including bugs in the driver. Or this may be an dumb user who screwed up his flash. It might be something else. We do not wand to mark the eraseblock bad when it is _not_ bad.

And it's not just erase successfully. We torture the eraseblock. We erase it, read it back, check it has all 0xFFs, then write 0xA5 pattern, read it back, check, erase again, read back, check, write 0x5A pattern, read back, check, erase, read back, check, write 0x00 pattern, read back, check, erase, read back check. Enough? If it is not enough, feel free to send a patch for better torturing.

Quoted 8 lines

Please, refer the torture_peb() function for more details.

So, we do not mark eraseblock as bad if it passed the torture test. Note, a bit-flip during torture test is enough to mark the eraseblock bad as well.

In your particular case, you would have _all_ eraseblocks marked as bad if UBI did what you want.

Torure is enough : )
Will you please explain why pattern 0x00 is not enough, it still need 0xA5 and 0x5A. What these information come from? Thanks in advance.

Signature · 2 lines
Best wishes,
Nancy
#8 lore raw
Quoted 3 lines · Nancy

Torure is enough : )
Will you please explain why pattern 0x00 is not enough, it still need 0xA5 and 0x5A. What these information come from? Thanks in advance.

Dunno, i just decided it might be good. And I just picked these values. If we fail on the first pattern, we do not continue.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#9 lore raw
Quoted 7 lines · Artem Bityutskiy
Quoted 3 lines · Nancy

Torure is enough : )
Will you please explain why pattern 0x00 is not enough, it still need 0xA5 and 0x5A. What these information come from? Thanks in advance.

Dunno, i just decided it might be good. And I just picked these values. If we fail on the first pattern, we do not continue.

Okay.
Another question:
Will you please explain why need leb_read_lock()?
If I ummap a leb(only operate the eba table, do not do
ubi_wl_put_peb() ), then, what will that PEB belong to? wl->free, wl->used, wl->prot.pnum, wl->prot.aec, wl->scrub? What will happen to that PEB?

--
Best wishes,
Nancy

#10 lore raw

Does volume size must be aligned to LEB size?
If I pass a unaligned LEB size to ubimkvol -s, then it will automatically make it aligned or what?

Signature · 2 lines
Best wishes,
Nancy
#11 lore raw
Quoted 1 line · Nancy

Does volume size must be aligned to LEB size?

Volume size is always multiple of LEB size.

Quoted 2 lines

If I pass a unaligned LEB size to ubimkvol -s, then it will automatically make it aligned or what?

Try and find out please.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#12 lore raw
In reply to Nancy
Quoted 1 line · Nancy

Will you please explain why need leb_read_lock()?

This locks the LEB and prevent concurrent I/O to this LEB. E.g., we do not want a situation when one process unmaps an LEB while another process is writing to it.

That was probably overkill, but I implemented per-LEB r/w locking. This is similar to rw semaphore in Linux. It means that several processes may simultaneously read for this LEB, but only one process at a time may write (or unmap).

Quoted 4 lines

If I ummap a leb(only operate the eba table, do not do
ubi_wl_put_peb() ), then, what will that PEB belong to? wl->free, wl->used, wl->prot.pnum, wl->prot.aec, wl->scrub? What will happen to that PEB?

If you unmap it you have to clean the corresponding EBA table record and return it to the WL unit.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#13 lore raw
In reply to Artem Bityutskiy
Quoted 5 lines · Artem Bityutskiy

Hi,

I've prepared 2 UBI patches which may slightly improve the scan time. I am not sure though. Would you guys please try them and tell if UBI scan time changed? Thanks in advance.

Hi Artem,

Thanks for your patch.. unfortunately I don't see a significant difference, although it is slightly faster.

Before:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.900000] UBI: attached mtd9 to ubi0

After:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.890000] UBI: attached mtd9 to ubi0

Hamish

Signature · 1 line
Hamish Moffatt VK3SB <[email protected]> <[email protected]>
#14 lore raw
Quoted 38 lines · Hamish Moffatt
Quoted 5 lines · Artem Bityutskiy

Hi,

I've prepared 2 UBI patches which may slightly improve the scan time. I am not sure though. Would you guys please try them and tell if UBI scan time changed? Thanks in advance.

Hi Artem,

Thanks for your patch.. unfortunately I don't see a significant difference, although it is slightly faster.

Before:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.900000] UBI: attached mtd9 to ubi0

After:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.890000] UBI: attached mtd9 to ubi0
Quoted 1 line

Hamish

Do you know when the bad block scanning finish and the ubi scan start ?

Matthieu

#15 lore raw

Hi,

Quoted 12 lines · Matthieu CASTET
Quoted 12 lines
[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.890000] UBI: attached mtd9 to ubi0
Quoted 3 lines
Quoted 1 line

Hamish

Do you know when the bad block scanning finish and the ubi scan start ?

Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver for scanning against bad eraseblocks to build in-memory bad block table (BBT). And it is probably more than 1.23 sec. If you start using on-flash bad block table, this should go away. I never used on-flash BBT, but I know MTD supports this and for example OLPC has on-flash BBT.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#16 lore raw
Quoted 11 lines · Artem Bityutskiy

Hi,

Quoted 3 lines
Quoted 2 lines

Hamish

Do you know when the bad block scanning finish and the ubi scan start ?

Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver for scanning against bad eraseblocks to build in-memory bad block table (BBT). And it is probably more than 1.23 sec. If you start using on-flash bad block table, this should go away. I never used on-flash BBT, but I know MTD supports this and for example OLPC has on-flash BBT.

May be UBI should print a message at startup.

Matthieu

#17 lore raw
Quoted 1 line · Matthieu CASTET

May be UBI should print a message at startup.

Yes, it will, the change is already in my tree and should go to 2.6.26.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#18 lore raw
In reply to Artem Bityutskiy
Quoted 17 lines · Artem Bityutskiy

Hi,

Quoted 14 lines · Matthieu CASTET
Quoted 13 lines
[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.890000] UBI: attached mtd9 to ubi0

Hamish

Quoted 7 lines
Quoted 1 line

Do you know when the bad block scanning finish and the ubi scan start ?

Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver for scanning against bad eraseblocks to build in-memory bad block table (BBT). And it is probably more than 1.23 sec. If you start using on-flash bad block table, this should go away. I never used on-flash BBT, but I know MTD supports this and for example OLPC has on-flash BBT.

Well I think from past use of "time ubiattach ..." that most of the missing time is in the attach.

I've added a message at start of ubi_attach_mtd_dev(), but I can't test it from home as I left my test hardware at the office switched off tonight.. I'll confirm tomorrow.

Would combining the bad block scan with the UBI scan save time? I guess it would be a bad layering violation however :-)

What sort of speed do you get using
dd if=/dev/mtdblock9 of=/tmp/foo bs=128K count=64 (where mtdblock9 is your raw mtd NAND device). I'm seeing about 6 seconds to read that 8Mb, which is quite long I guess.

Hamish

Signature · 1 line
Hamish Moffatt VK3SB <[email protected]> <[email protected]>
#19 lore raw
Quoted 2 lines · Hamish Moffatt

Well I think from past use of "time ubiattach ..." that most of the missing time is in the attach.

Sure, UBI takes most of the time. Its just if you want to save 1.2+ sec, you may try to play with on-flash BBT.

Quoted 2 lines

Would combining the bad block scan with the UBI scan save time? I guess it would be a bad layering violation however :-)

Yeah, it could be done as custom hack but not for mainstream.

Quoted 4 lines

What sort of speed do you get using
dd if=/dev/mtdblock9 of=/tmp/foo bs=128K count=64 (where mtdblock9 is your raw mtd NAND device). I'm seeing about 6 seconds to read that 8Mb, which is quite long I guess.

I have busybox so stuff like 128K does not work. Here are my results:

# time dd if=/dev/mtd4 of=/dev/null bs=131072 count=64
64+0 records in
64+0 records out
real    0m 0.28s
user    0m 0.00s
sys     0m 0.28s
# time dd if=/dev/mtd4 of=/dev/null bs=131072 count=256
256+0 records in
256+0 records out
real    0m 1.12s
user    0m 0.00s
sys     0m 1.11s

Its OneNAND which has a controller and it is quite quick.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#20 lore raw
Quoted 5 lines · Artem Bityutskiy
Quoted 2 lines · Hamish Moffatt

Well I think from past use of "time ubiattach ..." that most of the missing time is in the attach.

Sure, UBI takes most of the time. Its just if you want to save 1.2+ sec, you may try to play with on-flash BBT.

I'm not sure what this means.. ? Instead of having to scan each block to check the marker, it has a central table? And that table is created once by an initial scan and then added to when UBI declares a block bad? How do I access this feature?

Quoted 5 lines
Quoted 4 lines

What sort of speed do you get using
dd if=/dev/mtdblock9 of=/tmp/foo bs=128K count=64 (where mtdblock9 is your raw mtd NAND device). I'm seeing about 6 seconds to read that 8Mb, which is quite long I guess.

I have busybox so stuff like 128K does not work. Here are my results:

Hmm I have busybox (1.9.x) also, 128K works ok for me.

Quoted 6 lines
# time dd if=/dev/mtd4 of=/dev/null bs=131072 count=64
64+0 records in
64+0 records out
real    0m 0.28s
user    0m 0.00s
sys     0m 0.28s

Very good. It's about 6.1 seconds for me :-(

(Worse on my previous hardware which had compact flash on IDE without a hardware IDE controller: over 11 seconds.)

Hamish

Signature · 1 line
Hamish Moffatt VK3SB <[email protected]> <[email protected]>
#21 lore raw
Quoted 11 lines · Hamish Moffatt
Quoted 5 lines · Artem Bityutskiy
Quoted 2 lines · Hamish Moffatt

Well I think from past use of "time ubiattach ..." that most of the missing time is in the attach.

Sure, UBI takes most of the time. Its just if you want to save 1.2+ sec, you may try to play with on-flash BBT.

I'm not sure what this means.. ? Instead of having to scan each block to check the marker, it has a central table? And that table is created once by an initial scan and then added to when UBI declares a block bad? How do I access this feature?

Yes.

When your NAND driver is initialized, it scans whole flash and builds the BBT. Basically, it reads OOB area of each first (and may be second) page of each eraseblock. The BBT is simply an array with 1 bit per eraseblock.

You may make it save the BBT on the flash media. So next time you boot, it reads the BBT from a pre-defined place (e.g., the last eraseblock) and that's it. It does not scan and does not waste time.

This all is happening in the driver, before UBI starts.

Yes, if UBI marks a block bad, which basically means it calls an MTD function, the NAND infrastructure changes the on-flash BBT.

But I never used this myself, so my knowledge is theoretical. I just know others use this and I saw the code in nand_base.c.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#22 lore raw
Quoted 3 lines · Artem Bityutskiy

You may make it save the BBT on the flash media. So next time you boot, it reads the BBT from a pre-defined place (e.g., the last eraseblock) and that's it. It does not scan and does not waste time.

Thanks for the info. It looks like this would not save me very much time so I don't think I will bother.

[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    1.250000] UBI: attaching mtd9 to ubi-1
[    6.890000] UBI: attached mtd9 to ubi0

I notice in your patch that you read a whole min_io_size block, even though you only need the EC and VID headers (total of 128 bytes each, or 576 bytes as a single read according to my calculation):

+ /*
+ * During scanning we read EC and VID headers at one go in to the
+ * @si->read_buf, and then check EC and VID header. This must be faster
+ * than doing 2 small read operations.
+ */
+ si->read_len = ubi->vid_hdr_offset + UBI_VID_HDR_SIZE;
+ si->read_len = ALIGN(si->read_len, ubi->min_io_size);
+ si->read_buf = kmalloc(si->read_len, GFP_KERNEL);
+ if (!si->read_buf) {
+ err = -ENOMEM;
goto out_si;
+ }

Won't reading 2K bytes be slower than 576 in some cases? If you have soft ECC then you have to read the whole page anyway, but if you have hardware ECC then you have no need to read the whole page into RAM.

Hmm. The software ECC seems to work internally on 256 byte blocks. However it appears that nand_base will always read in a whole page (2K on my flash). It should be ok to read only a 256-byte block as that's all you need for ECC calculation? Not a whole 2K which requires 8 ECC calculations.

Hamish

Signature · 1 line
Hamish Moffatt VK3SB <[email protected]> <[email protected]>
#23 lore raw
Quoted 2 lines · Hamish Moffatt

Thanks for the info. It looks like this would not save me very much time so I don't think I will bother.

Ok.

Quoted 3 lines

I notice in your patch that you read a whole min_io_size block, even though you only need the EC and VID headers (total of 128 bytes each, or 576 bytes as a single read according to my calculation):

Err, 64 bytes each even.

Quoted 3 lines

Won't reading 2K bytes be slower than 576 in some cases? If you have soft ECC then you have to read the whole page anyway, but if you have hardware ECC then you have no need to read the whole page into RAM.

Yes, I guess. Current MTD implementation reads whole page in any case, though.

Quoted 5 lines

Hmm. The software ECC seems to work internally on 256 byte blocks. However it appears that nand_base will always read in a whole page (2K on my flash). It should be ok to read only a 256-byte block as that's all you need for ECC calculation? Not a whole 2K which requires 8 ECC calculations.

However, there was a patch from Alexey which may certainly help you. It was not looked at properly, unfortunately. I'll try to find it in my mailbox and will send to you.

Signature · 2 lines
Best regards,
Artem Bityutskiy (Битюцкий Артём)
#24 lore raw
Quoted 10 lines · Artem Bityutskiy
Quoted 5 lines · Hamish Moffatt

Hmm. The software ECC seems to work internally on 256 byte blocks. However it appears that nand_base will always read in a whole page (2K on my flash). It should be ok to read only a 256-byte block as that's all you need for ECC calculation? Not a whole 2K which requires 8 ECC calculations.

However, there was a patch from Alexey which may certainly help you. It was not looked at properly, unfortunately. I'll try to find it in my mailbox and will send to you.

Thanks. Yes I found the patch in my linux-mtd folder too. It's very nice!

Here's my attach time before:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.900000] UBI: attached mtd9 to ubi0

... and now:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.970000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.060000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.170000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    3.880000] UBI: attached mtd9 to ubi0

Hamish

Signature · 1 line
Hamish Moffatt VK3SB <[email protected]> <[email protected]>
#25 lore raw
In reply to Hamish Moffatt
Quoted 20 lines · Hamish Moffatt
Quoted 19 lines · Artem Bityutskiy

Hi,

Quoted 15 lines · Matthieu CASTET
Quoted 12 lines
[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.890000] UBI: attached mtd9 to ubi0
Quoted 1 line

Hamish

Do you know when the bad block scanning finish and the ubi scan start ?

Quoted 8 lines
Quoted 5 lines

Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver for scanning against bad eraseblocks to build in-memory bad block table (BBT). And it is probably more than 1.23 sec. If you start using on-flash bad block table, this should go away. I never used on-flash BBT, but I know MTD supports this and for example OLPC has on-flash BBT.

Well I think from past use of "time ubiattach ..." that most of the missing time is in the attach.

Here's the evidence:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    1.250000] UBI: attaching mtd9 to ubi-1
[    6.890000] UBI: attached mtd9 to ubi0
[    6.900000] UBI: MTD device name:            "gen_nand.0"
[    6.900000] UBI: MTD device size:            512 MiB
[    6.910000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
[    6.910000] UBI: logical eraseblock size:    129024 bytes
[    6.920000] UBI: number of good PEBs:        4090
[    6.920000] UBI: number of bad PEBs:         6
[    6.930000] UBI: smallest flash I/O unit:    2048
[    6.930000] UBI: VID header offset:          512 (aligned 512)
[    6.940000] UBI: data offset:                2048
[    6.940000] UBI: max. allowed volumes:       128
[    6.950000] UBI: wear-leveling threshold:    4096
[    6.950000] UBI: number of internal volumes: 1
[    6.960000] UBI: number of user volumes:     4
[    6.960000] UBI: available PEBs:             0
[    6.970000] UBI: total number of reserved PEBs: 4090
[    6.970000] UBI: number of PEBs reserved for bad PEB handling: 40
[    6.980000] UBI: max/mean erase counter: 29/1
[    6.980000] UBI: background thread "ubi_bgt0d" started, PID 641

Hamish

Signature · 1 line
Hamish Moffatt VK3SB <[email protected]> <[email protected]>