DOCBoot support for NFTL-based DOC2000

As promised (albeit somewhat delayed), I have just checked into CVS some changes to DOCBoot supporting NFTL-based DOC2000 devices. Full instructions are contained in the updated README file.

Please note that changes to the diskonchip driver were required in order to get this working. I have just checked them in as well. You WILL need to update the diskonchip driver in order to use DOCBoot on NFTL-based DOC2000 devices.

PLEASE PLEASE help me test this. I've given it only the most cursory testing. Let me know if it works for you.

Thanks,
Dan Brown

Thanks for this - it's just what I need!

Unfortunately my 2.6.11.5 kernel patched with mtd/patchkernel.sh, is crashing with the following stack trace (I've just jotted down the symbols)...

EIP is at nand_do_read
...

nand_read+0x2c/0x30
find_media_headers+0x3e/0x150
nftl_scan_bbt+0x8f/0x360
nand_scan+0x466/0x940
doc2000_count_chips+0x4e/0x60
init_nanddoc+0x34b/0xad0

do_initcalls
...

The MTD section of my .config is below (I removed NFTL because I'm only going to be using JFFS2, ok?). Is there anything obviously wrong here?

- Zeri

CONFIG_MTD=y
# CONFIG_MTD_DEBUG is not set CONFIG_MTD_CONCAT=y
CONFIG_MTD_PARTITIONS=y
# CONFIG_MTD_REDBOOT_PARTS is not set
# CONFIG_MTD_CMDLINE_PARTS is not setCONFIG_MTD_CHAR=y CONFIG_MTD_BLOCK=y
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_MTD_CFI is not set
# CONFIG_MTD_JEDECPROBE is not set CONFIG_MTD_MAP_BANK_WIDTH_1=y CONFIG_MTD_MAP_BANK_WIDTH_2=y CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
# CONFIG_MTD_RAM is not set
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set
# CONFIG_MTD_COMPLEX_MAPPINGS is not set
# CONFIG_MTD_PLATRAM is not set
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLKMTD is not set
# CONFIG_MTD_BLOCK2MTD is not set
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set CONFIG_MTD_NAND=y
# CONFIG_MTD_NAND_VERIFY_WRITE is not set CONFIG_MTD_NAND_IDS=y CONFIG_MTD_NAND_DISKONCHIP=y
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set # CONFIG_MTD_NAND_NANDSIM is not set

Hmm. Looks like I broke something when I changed the code to scan the entire device for media headers. Some questions for you:

- What size is your DOC?
- How big is your firmware area? (I'm assuming you resized it using the DOS-based tools)
- Did the diskonchip driver work for you without crashing before my latest changes? (Obviously this would be with a standard-sized firmware area)
- Can you tell me what kind of crash it is? NULL pointer dereference, etc?
- What information (if any) is printed when the DOC is detected, before the crash? I'm interested in information about the number and sizes of NAND chips, number of chips per floor, etc.

I didn't see anything wrong in your config options.

-Dan

64MB (MD2202-D64)

Well, I used dformat on a different chip as suggested, but that was with
the unpatched kernel, so the chip probing failed (but didn't crash).
I'm now using a fresh one - I didn't think I should _have_ to resize the firmware area first, should I? I'll try the patched kernel with a resized-firmware-area chip tomorrow....ZZZ

Yes, I've been using 2.6.11.5 with the diskonchip driver compiled in - see first dmesg section below...

See second dmesg section below. Looks like it's falling over around where the ECC errors were occurring.

Sorry I haven't offered any real help on this!

- Zeri

------------------Unpatched 2.6.11.5------------------------ DiskOnChip found at 0xd0000
DiskOnChip 2000 responds to DWORD access
Detected 2 chips per floor.
NAND device: Manufacturer ID: 0xec, Chip ID:0x75 (Samsung NAND 32MiB 3,3V 8-bit)
2 NAND chips detected
ECC error scanning DOC at 0x18000
Found DiskOnChip ANAND Media Header at 0x18000
ECC error scanning DOC at 0x1c000
Found DiskOnChip ANAND Media Header at 0x1c000

	DataOrgID		= ANAND
	NumEraseUnits		= 4090
	FirstPhysicalEUM	= 6
	Formatted Size		= 66928640
	UnitSizeFactor		= 255

Bad block table at page 193, version 0x55
Bad block table at page 225, version 0x55
Creating 1 MTD partitions on "DiskOnChip 2000 (NFTL Model)":
0x00020000-0x4000000 : " DiskOnChip BDTL partition" Found alias of DOC at 0xd0000 to 0xd2000
Found alias of DOC at 0xd0000 to 0xd4000
Found alias of DOC at 0xd0000 to 0xd6000
Possible DiskOnChip at 0xe4000 failed TOGGLE test, dropping.
...
------------------------------------------------------------

------------------Patched 2.6.11.5------------------------ DiskOnChip found at 0xd0000
DiskOnChip 2000 responds to DWORD access
Detected 2 chips per floor.
NAND device: Manufacturer ID: 0xec, Chip ID:0x75 (Samsung NAND 32MiB 3,3V 8-bit)
2 NAND chips detected

Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c8861dcd
*pde = 00000000
Oops: 0000 [#1]
...
Call Trace:
 [<c015c3bb>] d_lookup+0x1b/0x40
 [<c010daa6>] recalc_task_prio+0x96/0x170
 [<c010dbd1>] activate_task+0x51/0x70
 [<c8861d4c>] nand_read+0x2c/0x30 [nand]
 [<c88581ae>] find_media_headers+0x3e/0x150 [diskonchip]
 [<c885834f>] nftl_scan_bbt+0x8f/0x360 [diskonchip]
 [<c8863cb6>] nand_scan+0x466/0x940 [nand]
 [<c885815e>] doc2000_count_chips+0x4e/0x60 [diskonchip]
 [<c8858e0b>] init_nanddoc+0x34b/0xad0 [diskonchip]
 [<c01273e2>] sys_init_module+0x132/0x1b0
 [<c0102277>] syscall_call+0x7/0xb
---------------------------------------------------------------

No, you're quite right. In fact, it's very useful for me to know that the code crashes even when you don't have a resized firmware area. Don't even bother trying it with a resized firmware tomorrow :)

So is it correct to say that you've just switched from the diskonchip driver which is distributed as part of 2.6.11.5, to the diskonchip driver from the MTD CVS repository?

If so, then any of the changes between the MTD version in 2.6.11.5 and the current CVS are potentially the problem, not just my latest change to diskonchip.c

The version of diskonchip.c in 2.6.11.5 is 1.45. I don't see anything in the changes between then and now (1.50) that should cause the behavior you're seeing, which means either the problem is in a different file or I'm just not seeing it :)

I'll stare at it some more tomorrow. Anyone else (Thomas?) have any ideas where to look?

On the contrary, you're much better at providing the right information than most people.

-Dan

Been far too busy, anyway :)

Exactly. I didn't feel the need to use latest MTD sources for the diskonchip driver until your patch. Looks like the problem is in nand_base.c.

The call stack goes from mtd->read into nand_base.c nand_read() then nand_do_read_ecc() passing NULL to oob_buf and oob_sel. The pointer dereference occurs on oobsel when defining ecc_calc and ecc_code... I (nervously) replaced these with hard values and the probing completed successfully. I think most of the activity lower down does null checks on oobsel or doesn't use these values due to the eccmode.

I have no idea what the fix should be! Maybe move the definitions of ecc_calc and ecc_code below the check and set of oobsel to &mtd->oobinfo?

- Zeri

You're 100% correct. A recent modification to support variable-sized ecc_calc and ecc_code arrays didn't take into account the possibility of NULL oobsel. Fixed in CVS.

Let me know if DOCBoot finally works for you, please!

-Dan

OK... diskonchip probing was fine with a fresh chip. Here's what I've tried...

Built a kernel with MTD etc linked in (temporarily including hard-disk
stuff) and verified booting it from the hard-disk (obviously the diskonchip driver doesn't show the firmware partition, but that's ok).

Used M-Systems dformat (from tffs_5.1.4_DOS_TOOLS.zip) to resize the firmware area

    A:\> DFORMAT /WIN:D000 /BDKL0:2M

Rebooted into a kernel with modules and ran

    # modprobe diskonchip show_firmware_partition=1
    # cat /proc/mtd
dev:    size   erasesize  name
mtd0: 04000000 00004000 "DiskOnChip 2000 (NFTL Model)"
mtd1: 00208000 00004000 " DiskOnChip Firmware / Media Header partition"
mtd2: 03df8000 00004000 " DiskOnChip BDTL partition"

Edited docboot/cmdline to
root=/dev/mtdblock1 rootfstype=jffs2 ro

Uncommented "#define OLD_DOC2K" in doc_bootstub.h

Copied in the bzImage I'd tested booting from hard-disk.

    # make
    # flash_eraseall /dev/mtd1
    # nandwrite -o /dev/mtd1 doc_spl
    # flash_eraseall -j /dev/mtd2

Tested mounting and unmounting /dev/mtdblock2 which seemed OK. (My cmdline above assumes that this will show as /dev/mtdblock1 when the linked-in driver is loaded.) Rebooted and disabled/removed all other boot devices. As the BIOS gets going, I see "Installing DOCBoot." then a "System Configurations" page with

    ...<snip>...
    Verifying DMI Pool Data .........
    Loading kernel... _

where the "_" is a flashing cursor. The keyboard is unresponsive and nothing further happens :(

By the way, I have not yet copied any files to the jffs2 filesystem on /dev/mtdblock1 except a small text file to test the filesystem.

Any ideas? I'd really like to get this going.

Thanks again,

- Zeri

OK, I'm slightly stumped. So, I've just checked into CVS some additional changes to DOCBoot. There is a small (but important) fix to memory size detection, which shouldn't affect you since you're not using initrd (I think).

There is also a new option in doc_bootstub.h: #define DEBUG_BUILD. This will print out some interesting info during the "Installing DOCBoot" phase, and wait for a keypress. Please copy it down and let me know what it says. It will also be more verbose during the actual kernel loading phase; hopefully that will let us pin down exactly where it's hanging.

-Dan

No, you're right - I'm not using initrd.

--------------------------------------------------------------
Award Modular BIOS v4.51PG, An Energy Star Ally
...
HS-4020 VER:1.4

Cyrix MediaGX With MMX-S CPU at 300MHz
Memory Test : 64000K OK + 1536 shared memory

Award Plug and Play BIOS Extension v1.0A
...
Installing DOCBoot.
doc_seg = 0xD000 setup_seg = 0x2000
int15/88 returns 0xF600 initrd_start = 0x000000
top of low mem = 0x0280k handler seg = 0x9FC0
-- Press any key --
--------------------------------------------------------------

...<snip>...
Loading kernel ... page 0x.... <- this seems to start at zero and be going up at roughly 0x200 a second up to 0xFFFF then restarting at zero again over and over.

Just to give you a clearer picture, the bzImage is 1148021 bytes and doc_spl is 1187472 bytes. The nandwrite command wrote doc_spl from 0 up to the erase block at 0x118000 which is well within the 0x208000 firmware area. The board is a Boser HS-4020 v1.4.

Unfortunately, assembler is greek to me!

- Zeri

Hey, I just noticed that 0x11c000 (0x118000 + 4000) is only 1163264 bytes - doesn't that mean doc_spl wasn't fully written to the chip!?

- Zeri

The page number you see there is the offset (in 512-byte pages) on the diskonchip that DOCBoot is currently scanning, trying to find your kernel. DOCBoot looks for a special tag in the OOB area of each page belonging to your kernel image. I can think of two reasons for this process to loop forever:

1) The image didn't get written properly (using nandwrite without the
-o option would certainly have this effect, but you indicated that you did use -o).
2) DOCBoot is having trouble reading your device, perhaps because my
new non-TSOP DOC2k support is not general enough.

No. Each 512-byte page on the flash is associated with 16 bytes of out-of-band (OOB) data. doc_spl contains this data interspersed with the normal data. nandwrite reports the page address, which only reflects the location of the normal data. So, the amount of normal data in doc_spl is

length * 512 / (512 + 16)

In your case that's 1151488, or 0x119200 bytes of normal data. Thus the image should end in the erase block which starts at 0x118000 and ends at 0x11C000, as reported by nandwrite.

Note that a partially-written image would NOT result in an infinite loop in DOCBoot, because (as you saw) it will eventually wrap all the way around to the start of the device if it can't find the number of tagged pages it's expecting. After it wraps around, it will (stupidly) make up the missing pages by reloading tagged pages from the start of the device.

I'm not sure where to go next. Can you please run "mtd_debug info /dev/mtd0" and tell me the result, just as a sanity check?

-Dan

Yes, I definitely used the -o option to nandwrite.

mtd.type = MTD_NANDFLASH
mtd.flags = MTD_CLEAR_BITS | MTD_ERASEABLE | MTD_OOB | MTD_ECC mtd.size = 67108864 (64M)
mtd.erasesize = 16384 (16K)
mtd.oobblock = 512
mtd.oobsize = 16
mtd.ecctype = (unknown ECC type - new MTD API maybe?) regions = 0

This may be useless, but I did a nanddump of the first 0x208000 bytes. 0x00 to 0xff -> data

Then each of the following chunks of data (ie, not chunks of ff) are preceeded by OOB Data (eg,
OOB Data: 36 7f eb 1f 3b aa ff ff ff ff ff ff ff ff ff ff
0x000000200: 0e cd...)

0x200 to 0x2ff -> data
0x400 to 0x43a -> data
0x600 to 0x629 -> data
0x800 to 0x8ff -> data
0xa00 to 0xa75 -> data

Then a large chunk (1144949 bytes)
0xc00 to 0x119075 -> data

Some mysterious OOB Data before 0x119200, though no normal data following it.

Then the rest is blank.

- Zeri

Looks fine.

Sounds right, at first glance. The OOB data actually follows each block (perhaps that's just a terminology issue :) )

-Dan

Ah, I see it! That OOB data is actually wrong, it should be:

?? ?? ?? ?? ?? ?? 55 55 ff ff ff ff ff ff ff ff

Here's the whole sordid story:

1) The only reason DOCBoot has been working for *me* lately is because I've been lazy/stupid.

2) Specifically, although I've been updating my copies of the kernel code to keep pace with CVS changes, I have *failed* to update my mtd/utils subdirectory.

3) Thomas recently made a modification to nandwrite. It forces all OOB bytes not designated as "free" to 0xff. Perfectly sensible.

4) There is apparently a long-standing oddity in diskonchip.c, in which I set the number of ECC bytes to 6, and designate the last 8 bytes as "free" bytes. This leaves two bytes (right after the ECC) unaccounted for.

5) DOCBoot puts magic numbers in those exact two OOB bytes.

6) As a result, recent versions of nandwrite will erase the information DOCBoot needs to identify kernel pages. I've been blissfully unaware of this because I've been using an older version.

7) So, I've changed diskonchip.c to designate all of the last 10 bytes as "free". It seems like the right thing to do, since only the first 6 bytes are used by the hardware itself. BUT ....

8) I have this nagging feeling there was a reason for 4).

9) I think this may break compatibility with any jffs2 filesystems written before the change, because the cleanmarker will be in the wrong place. Hmm... maybe I should back out the change. David? Thomas? Opinions?

10) AAAARRRGGGHHHH!!!!

-Dan

After much discussion and experimentation, a version of diskonchip.c which does not break older jffs2 filesystems has been checked into CVS.
Also, some required changes to nandwrite have been checked in as well.

Zeri, I really think this should work for you now, particularly since I was able to duplicate exactly the problems you were having.

-Dan

Yes! With the diskonchip driver, flash_eraseall and nandwrite and DOCBoot from CVS, my DiskOnChip now boots the Linux kernel.

Thanks for everything!

- Zeri

Very glad to hear it!

-Dan